Forum Academy Marketplace Showcase Pricing Features

Performance/Best Practice for Creating Things in Bulk

Need help
#1

Hello!

I’ve got my app generally working in Bubble, but I’m having a problem on the initial mass import steps I need to take. What I need to do is make a few GET calls to an external API and then break apart the responses (they’re in JSON) to the GET into individual things. Each step could produce up to 200 things.

The initial GET takes only a second or two from the external API and contains the data for the 200 things. My problem is the breaking up of that result is taking ~1-2 seconds per item to create the things… which obviously isn’t the best user experience when importing a chunk of data.

How I’m currently executing is the following:

  • User initiates the import which passes to a Backend Workflow
  • Backend Workflow has an individual step of Schedule an API Workflow (Create a Thing) on a List. The List is the result of the GET.
  • The workflow “Create a Thing” is populated with the parameters from the GET results, then creates the Thing based upon those parameters.
  • There’s a few subsequent Schedule an API workflow steps based upon incremental parameters so I get all the user’s results

This seems very inefficient and it’s processing very slowly. Is there a better way to do this?

EDIT: Just for clarity, the only GET is for each list of activities, there aren’t any subsequent calls to the external API in the creation of the things.

#2

Not a very helpful answer but in my experience, backend workflows are by nature slooowwww and it’s just a “feature” you have to live with. It’s kind of difficult to comprehend that generally, databases deal with 1000-10000 inserts per second, Bubble does 1-5 inserts per second. So if your problem is bulk loading of data quickly - Bubble will never be a great solution.

The best place I’ve got to with Bubble is to schedule “batches” of say 200 units of processing every hour, and schedule it to run over 12 hours. (because of silent process death with long-running processes in Bubble)

My suggestion would be for those tables that have a lot of records that you need to load and unload quickly - use an external database for those records - @jared.gibb has done this successfully with Firebase.

On the plus side, I’ve not seen a backend workflow impact the runtime performance of the frontend Bubble App.

eg this sort of stuff…
https://forum.bubble.io/t/is-there-a-limit-on-how-many-records-can-be-processed-at-once/174338/8

#3

@lindsay_knowcode , thanks for the reply. That actually does help in the sense that I’m not going to get the performance I need with the approach I’ve got. I’d run across this thread (Forum - Exploiting the database without slowing it down); however it seemed like was going to take some code (obviously not a heavy coder having chosen Bubble) and didn’t seem like it may fully solve the issue.

I’ll have a look into Firebase, thanks!

#4

:raised_hands: :gem: :rocket: :full_moon_with_face: :raised_hands:

#5

@julienallard1 tip is a good one, and if it makes sense in your App to conflate your inserts onto a single String - great! :slight_smile:

By similar logic, you could redesign your database to be very flat - so you had a single insert that had many columns (without the referential tables), that might help work around Bubble’s limitations, but it’s just going to depend on what makes sense in your particular case.

Ultimately - if your problem is a Database problem requiring load and unload speed, volume of data, fast operations and search - then that is probably best solved not with Bubble’s Database :slight_smile:

#6

I guess I need to do some more research on this to determine best approach. Thanks for the feedback!

#7

I’ve managed to put thousands of lines using Google Sheets and AppScript. It isn’t fast (5 per second) but if you can handle failure (timeouts) you can push to 20 or 40 calls a second and retry the ones that fail.

At the moment, with some GoogleSheets trickery, I am able to push large amounts of data automatically unattended.

#8

Flat tables kill data retrieval.
Backend workflows have high overhead. Do more than 1 record per iteration. You’ll find a sweet spot depending on app variables.