Best practices for large dataset export

I am asking for best practices based on your experiences for things like:

  1. Frontend vs backend workflows?
  2. Datatype-based intermediate? If so, two data types, Export and ExportRow?
  3. NOTE: I tried the RG-to-CSV approach with a couple of plugins…but that didn’t work due to the known issue around Repeating Groups and “page loaded entirely” falsehood. See: Page and all elements completely loaded - #24 by sam20)

An example export:

  • 200 rows based on two datatypes, based on:
  • 200 rows of “Applications” that each have ~50* child rows of “Responses”
  • The export will have ~55 columns: 5 that are fields & calculations from “Applications” and 50 columns built from the child rows

*NOTE: The number of child rows is dynamic.

Summary: 200 rows * 55 columns

1 Like

If this is an export you are only doing on demand, like the user goes to a page to export, then have you tried this?

You make JSON with searches to build what you want, then this converts it to a CSV file and downloads it.

It might actually have a serverside action available if you need to do this on a recurring basis in the background.

1 Like

@tylerboodman

Thanks for the reminder of this plugin.

I did not understand the documentation (all I could find was the info on the “plugin page”) after a couple reads…so I moved on.

Perhaps I should take another look.

Have you used this plugin for an export at the level of complexity that I mentioned?

Yea should be fine. Here’s a basic example where I’m using it:

This outputs a CSV with columns for ID, Type, Description, etc. And you just fill in whatever data you want in the blue spots. In your case the calculated ones will be some kind of math instead of simple expressions like I have.

Oh, wow! That looks awesome!

I will try that immediately!

I wonder how it will handle my child records that need to be expanded into 50 columns?

1 Like

@tylerboodman

I tried the 1T-CSV plugin, but I don’t think it will work for my case, as there are a dynamic number of child rows that need to be converted to columns. (I just edited my initial posted to add that detail.)

@tylerboodman
After the “Create CSV from JSON” action, how do you download the file?

It should just download right away, so if it didn’t check the dev console it probably errored out due to invalid formatting

I think this is too advanced for CSV format :sweat_smile:

CSV is just like columns and rows sounds like you want some child rows with different columns under like parent rows?

That would be like an interface you make on Bubble or some custom excel spreadsheet with tables