[Updated feature] CSV data export overhaul

Hi everyone!

We just rolled out an overhaul to the CSV data exporting feature, which paid users can see in the Data > App Data panel of the editor.


:dart: Key highlights:

  • No more 20k row limitation (should work on just about any table)
  • The export is now done server-side, and the editor remains fully useable during the export
  • When the export is complete, we will send an email to you with a link to it
  • The result file will appear in your files as a private file only visible to users with editor access
  • You can make several exports at once
  • New file export format options: csv, json, and ndjson (newline-delimited JSON)
  • A lot faster for simpler views!
  • You can see exports in-flight and can cancel any there

:pencil2: Tips and gotchas:

  • Behind the scenes, we export in batches that we stitch together: this means that rows deleted during an in-flight export may still end up in the export if a previous batch exported them already
  • Don’t edit a data view during an export of that view! For the reason stated above, weird things may happen :slight_smile:
  • A view where columns contain data from other tables will be a lot slower to export than a table with only self-contained data. If you want the best possible speed, set all “primary fields” that appear in a given view to be unique ids.

Hope you like it, and be sure to report any problems you encounter with it!

40 Likes

Amazing. Thank you.

Great feature to kick of 2020, cheers.

2 Likes

I love you, right now! :pray:

1 Like

Nice way to start 2020! Any experiments going with exporting nested data? That would be awesome!

Nice one!

Is the user front end data export to CSV going to be updated any time soon?

Thanks a lot and have a good day.

Anyone else having trouble on nearly a big percentage of exports? Out of 22 attempts, 12 have failed. Granted, the bases are pretty large (field wise) and there’s linked fields, but the previous exporting structured was able to handle it…

1 Like

Hey @arcbot, thank you for pointing this out.

Operations that have a lot of linked fields will use a lot of capacity, and as a protection measure we throttle the operation to avoid an app from burning through all the capacity too quickly. This did cause some large exports to fail so I’ve pushed a fix to dial back this throttling a bit, which should address the operation’s reliability on your end.

Don’t hesitate to tell me if this is still broken, however you will indeed experience variable performance depending on your app’s available capacity

2 Likes

Hey @aless, I’m still getting the same issues trying to export the bases with linked fields. What can I do to avoid this errors?

I also tryed to export another table in a JSON and just get 2 of 646 records… thanks in advance.

1 Like

My exports are no longer working. I have a ticket in, but it has derailed my operations for several days. The emailed links never work for me.

1 Like

Hi all,

If you could please file a bug report, our Success team will triage any incoming concerns and pass the bugs along to our engineering team to fix.

Thanks!
Allen

The emailed links don’t work for me either. I filled a bug report yesterday.

@aless Is there also a way to do this via a workflow? Let’s say I want to create a back up of my data on a daily/weekly basis?

4 Likes

I would also like to know this!!!

1 Like

Hey @eddy, this feature isn’t yet tied to workflows, but I’m definitely submitting this feature idea!

4 Likes

Thanks that would be perfect! :slight_smile:

Just rolled out the feature completely, which means we removed the option to use the legacy client-side export.
Keep in mind that the exported file will always be available for you to download from the file manager tab as soon as the progress hits 100% for the exported file and disappears ; you don’t have to open your email.
Best,

2 Likes

Hi @aless, can you clarify if this is now available via workflow? Is that part of the latest release?

thank you,
G

1 Like

It isn’t, but I have submitted the feature request for future iterations :slight_smile:

1 Like

For reasons unknown to me, all dates on your CSV exports are in Vietnamese and I can’t find a way to fix it. P.S., I’m from California (NOT VIETNAM).