Has anyone come up with a good technique for exporting data from large and/or complex queries?
Bubble’s built-in CSV export fucntionality is sorely lacking (unless I’m missing something), as it seems to allow dumping from only a single table (thing) at a time. The same is true of the data API.
And the approach used by @eli’s nifty CSV Creator plugin is, unfortunately, not well suited to large/complex queries either. (That’s no fault of the plugin per se; it’s just that Bubble chokes and barfs an error to the console because of a timeout.)
The only reliable way I’ve found is the Download CSV from RG plugin. The browser’s “Page Unresponsive” dialog appears, but it can be safely ignored, as the full RG is eventually rendered, and then the CSV can be saved, as the data are actually pulled from the DOM (rendered page) to save to a file.
The main problems are:
You have to fiddle with constructing a UI even when there’s no real need for one.
It can take a very long time to render the RG, and the page must be left open to do that - i.e. no way for it to operate in the background. (A query resulting in 740 rows took more than half an hour!)
Ideally, I’d like for the user to click a button and then be able to navigate away from the page while the export is handled in the background. A link or attachment would be emailed to them when the export was complete.
Again, a key feature is to be able to export relational data - not just a single Bubble data type (the contents of a DB table).
I’m wondering if there might be a way using the List:format as text operator along with an API endpoint.
Has anyone developed a workable approach or have any ideas?
I added a server-side action to the CSV Creator last week. Still in Beta while some users are testing it but seems to be working well and is available if you upgrade the plugin to the latest version.
It’s very similar to the client side element but it’s not possible to return a file directly from the server so it simply returns the Base64 text. There’s a second action then ‘Create File from Base64’ that generates the file itself using the data API on your app. Set it up as shown here.
Thanks for the response, @eli. I did notice the SSA code in the plugin but didn’t try it. If you’re saying that users with large data sets and/or complex queries are not experiencing timeouts with the SSA, then that’s encouraging. I will give it a try and report back.
So far so good as long as it’s run in the backend obviously. Running the SSA as a page action will likely have the same time out issues as the client side plugin.
I am using your Json to CSV as a backend workflow but I keep getting an error. When I run the workflow on a small list up to ~300 records, it works fine. But when I run it on, the last list I tried had 437 records, it failed. In the log I am getting this error.
Workflow error - Plugin action Create CSV from JSON (SSA) error: SyntaxError: Unexpected token in JSON at position 147131 at JSON.parse (<anonymous>) at eval (eval at build_function (/var/task/index.js:86:21), <anonymous>:6:18) at /var/task/index.js:349:23 at run_fn (/var/task/u.js:594:18)
It’s been a while since you asked but the reason this would happed is one or more of the values in records 301 to 437 has a character such as a double quote that would break the JSON. I would use the ‘format as JSON safe’ operator for any fields that can contain any breaking characters. Should solve the problem.
