I am running some API workflows (triggered by buttons in my admin page) to edit my database and keep having the workflows stop in the middle - the server logs say “Workflow error - Operation timed out – app too busy”
I reached a point where a single action, doing a search for about 300 things and updating 1 number attribute, is timing out.
The CPU usage shows I am not exceeding 50% of the capacity.
Is there a way to understand the issue? Is there a better way to handle bulk database actions?
Edit: when i do the same action by loading the things in a repeating group and applying the change in a page workflow, it takes 5 seconds - there must be something wrong
Facing a similar situation today…Single API call, 300 results. Scheduling an api call to the list of results. Retrieval of the original list is almost immediate, but it will never execute the api on a list action. Schedules it out for ten minutes in the future, and then errors with the Operation timed out - app too busy.
My solution was to break it down into smaller chunks, where each chunk can be completed within a few minutes. I have a series of endpoints that each do the exact same thing, except they retrieve a different set of data to work on. Each endpoint schedules an api call on a list, and then if needed, it schedules another batch. It’s not ideal, but it does work and is pretty easy build out with copy/paste of endpoints.
Here’s what happens in each identical batch:
Creates a new thing (let’s call it a “batch controller”) that represents a batch to be processed. In that control, it has among other things the total number of records to be processed, which drives how many batches will be run.
Triggers a custom endpoint (there is just one of these used by all batches, not one for each batch) and I pass it the batch controller created in step one. It is basically a way to pass multiple pieces of information to a common task. In this endpoint, I schedule an api on a list, which uses information from the batch controller. All the tasks in here must complete within 4-5 minutes (I’ve seen both used in posts here as the limit.)
Schedules the next batch (api endpoint,) if required based on the batch controller. It passes information to the next batch, including incrementing the batch number.
In my case, I have fifty batches setup to account for 5,000 total records of import. Once I had the logic down with two batches, the rest was just copy/paste/rename. The key is to get the batch controller down so that it calculates batch numbers, etc. so you don’t have to modify a bunch of stuff when doing copy/paste.
It should hold me until Bubble prioritizes some serious data manipulation support – I’m sure that will come with time, since they’ve done great in so many other areas.
Hi @mebeingken, I’m trying to duplicate your workflow but I’m having some trouble understanding it completely. I ended up trying to schedule the next api endpoint from the previous one, but since I was running on a list I didn’t realize it would do it every time (so the next api call was scheduled 50 times by the previous one since the previous list batch had 50…it was bad ) Could you go into further detail on how yours is set up?
Glad it is working. This solution though, is a bit obsolete now that a workflow api can schedule itself iteratively giving us the ability to loop through chunks of data without having to create a bunch of hard coded batches.
Here is an example of a looping api endpoint. In this case, it is importing contacts from Google, and creating a contact record in Bubble for each contact received. The use case is that because the user may have thousands of contacts in Google, we want to break it into manageable chunks.
Let's first create an endpoint that will be used to create a single contact record. Our looping endpoint (which we will create next) will call this endpoint and pass it the values needed to create the contact:
Next we create the looping endpoint used to break into chunks (import_contacts.) We pass to this endpoint the values needed, which in this example are a starting point for the import and the maximum number of contacts to retrieve.
Step 3 then schedules the execution of the looping endpoint (itself) but only does so when the total results is greater than what we have previously processed. In this example, the Google api provides us with the total results available for our query (not to be confused with max results which is the number of results we ask for.) So the calculation is our current starting point plus the number of results we get in each batch must be <= to total results. If it is, that means we have more results to fetch, otherwise we have received the last batch and won’t schedule it again. We also increment our start index, which is how you loop.
What if the API won’t give you the information in batches? I tried only capturing the IDs so I can get more information on them one by one. Still, as API sends the whole 150 of them (best case, there are larger numbers returning in that single call as well), I can’t even just save their IDs to the DB.
Just stumbled upon this thread while researching another WF-related topic, and I was wondering if you have any experience with the “bulk create” feature. It seems applicable and that it might provide better performance. Have you tried it?
Yes, I’ve used bulk create api from external systems posting into bubble. It is definitely an option, but I never used it from within bubble. Especially with the recent updates to improve the speed of inserts, bulk should be considered.
I face a similar issue with a client side workflow where I create 42 to 90 items by copying them from template items. It takes for ever and times out.
I don’t know what to do with this, and it’s really frustrating. I also don’t understand whether this is a local limitation (per app user limitation) or global (share resource for all users on the app at the same time). So I’m worried that when my app goes public it’ll be basically unusable.
) Could you go into further detail on how yours is set up?
