As you may have seen from my previous post, I am currently uploading a lot of data from an external database into Bubble as a migration to a “nocode” tool.
Because the migrated tables are relational (unlike Bubble) it makes it very difficult to build efficient queries so we needed to populate some extra data.
Over the course of this, I have had to raise a number of bugs on this bulk process that Bubble offers. In short, I have just not been able to do what I needed to do in any kind of reasonable timeframe.
“Fetching List” is constantly displayed when trying to run a bulk action
Unable to search in data tab on large amounts of data - you get “an error occurred”
Bulk operations confirmed as “1 per second” no matter what plan. 250,000 rows of data will therefore take around 70 hours to process.
Field “is empty” does not trigger when the field is a reference to another thing, and that thing has been deleted. You now cannot select those rows as they are both “not empty” and don’t have anything in them.
Same 20 rows are being selected when doing items until # 20 with a filter
Cancelling “ALL” workflows does not work unless you pause them first
Yes, there are a lot of rows but nowhere does it say “don’t use the data tab if you have a large table” or “don’t use bulk actions” etc
Bulk data operations in Bubble are not fit for purpose. You simply have no way at all to process operations on large amounts of data.
This NOT mean that Bubble does not handle volumes of data, it does that well. What it does not do is allow you to manipulate that data from inside the tool.
@malcolm and @josh need to be upfront about the limitations here. And what, if anything, they intend to do about this. I raised issues 5 years ago with the data tab (that are still not fixed) and nothing really seems to have changed since then apart from the really good csv uploader.
These are seriously limiting issues with the platform here - which go well beyond “we do appreciate this feedback on the process” from @eve that I have received last week.
As it stands at the moment, I could never recommend to anyone to use Bubble for large data operations, despite what their technical team will tell you. Look at Xano (or similar) instead.
A sad few weeks for me here, I really thought Bubble was better than this
Sadly we can confirm the same issues and this is something that bubble never did well. I would hope this will get more attention as more and more mature products are being built on bubble and these problems become more evident.
If a service line xano offers a faster backend than bubble native, you know you have a problem.
Yeah, we also never recommend using Bubble for large data processing (even medium).
The best way to process it outside the Bubble - and then display data via API/SQL.
As a non technical person, can one of you kindly give a real world example or two of large data processing that Bubble does not do well, so that I can know if it applies to the app I am creating
Migrate some data into Bubble using the (excellnt) CSV importer. Now run something to create Bubble style object relations between the two. So half a Million “book” rows in one Table and 200,000 “author” rows in another. Use a field on one of them to build the Author(s) <> Books relation.
Try to delete 10,000 rows of data
Anything that goes through large amounts of data and updates counts/totals in one go. Try to do it as you go, not rely on “batch” work.
Ok thanks. I have over 450,000 rows of data and a few fields are nested.
In a few months time I will have to query that data to provide some insights for my company.
@NigelG my initial thinking is to have a separate page in my app and create a reporting dashboard type page.
But
Also I’ve been thinking about if it is slow and an absolute pain in the ass I would just download everything in a CSV and use a 3rd party tool like Excel.
Here are the techniques we have used to data process on the Bubble server. Using your examples:
For the book importing, we create empty author and bookthings, where book contains fields raw author as a text and book author as a reference to author. We then add a trigger on any change to raw author that looks up the author and sets the book author to the best match. We then upload all the authors first. Once that is complete we then upload all the books.
For bulk deleting we go old school/low level using a deleted flag and behind the scenes garbage collection. We hide any records that have a deleted flag in the UI/UX and have triggers that run behind the scenes to eventually get around to deleting. This is basically how the on disk delete operations work, except much faster, in classic RDBMSes.
For statistical processing at scale we move off of Bubble and use Map/Reduce techniques on external services like AWS Lambda. All Bubble has to do is respond through it’s native Data API.
Maybe @malcolm can provide an engineering explanation as to why Triggers seem to run much more robustly than scheduled APIs for bulk data processing? I suspect the call stack for Triggers sits DB side and is much larger and more stably supported, and that the call stack for Workflows sits Fibers side and is smaller and more brittle.
It is worth noting that the internet is not nearly as fast as we perceive it.
We still managed to get triggers to fail at scale.
Yes, that is definitely the way to go here. In our case we had 100,000s rows generated in test by API calls and nested recursive workflows. There was duplicate data so we really needed to clear it down regularly.
Wow triggers failing at scale. And they are the better alternative…*?!
That said I elided much of the discipline around our design pattern in triggers:
We moved everything off of cascading scheduled workflows, absolutely everything.
We insulate trigger code within a scheduled workflow, the only thing our triggers do is call exactly one scheduled workflow.
We only pass thing now from the trigger to the scheduled workflow. This appears to commit the database transaction of the trigger, no reversies, no take-backs.
Our triggers are tightly conditioned on “thing before’s some field is not thing now’s some field”, we specifically detect and run against one field change at a time.
When possible we do time delay the scheduled workflow, to space consecutive triggers out. Note that we still have multiple triggers running in parallel, this just introduces a rate limiter into triggers cascading to other triggers.
Any field in a thing is only ever written once by the triggers.
Overall we use triggers in a “calculated field” design pattern. It took months of re-factoring to get to this highly disciplined design pattern. But it was absolutely necessary because we have to interface with a number of archaic legacy systems that do horrible things like send us emails of PDFs of faxes of images of documents.
I say take your data off bubble and harness aws or Firebase for large data. Heck, why not auth too? The Price for this extra capacity in data data storage/retrieval you’ll receive? Dollars a month for most not huge apps (or less).
The initial investment is not cheap but you can get away from the limitations and really let the data flood in and be processed very quickly.
Bumping this as it has been over a year since @NigelG brought this up – It still is a major problem.
I’m in the process right now of running a Bulk operation on 4700 records to update 1 field and this is painful - It still closes out the UI and prevents you from doing anything else. Most of the time, I do recurring schedule api… which is slow itself, however, figured I’d give the Bulk operation a shot again. Mistake. Time to go grab coffee… and perhaps lunch… maybe dinner… and will keep my laptop on overnight.
