Optimizing API Calls with Recursive Workflows in Bubble: Seeking Solutions for Efficiency and Rate Limit Issues
I’m very stuck with this. From my backend, I need to make an API call to an external service to retrieve a list of data that I need to create or update if it doesn’t exist.
The API call returns between 100 and 150 results with each call, and this API has a request limit per hour.
The problem with the recursive workflow is that we cannot send a JSON list that is not a Bubble data type.
So for now, I have to:
Workflow 1: GET API data, then schedule workflow 2 on this list ID.
Workflow 2: Get API Single Data (with the ID) and then make changes or create if it doesn’t exist, schedule workflow 2 minus the first item ID.
As you may have understood, I consume as many rate limit tokens as there are data items, which is very unfortunate because during my first call I obtained the complete list with all the information.
Does anyone have a solution?
This becomes very demanding for my third-party API and moreover, I do not understand why Bubble does not allow passing a JSON format in the parameter in the API workflow, which would also be less costly in WU than making a single API call for each data item.
A) do you really need to schedule recursive workflow and not a Schedule on a list WF? You could pass the needed fields for each item in the JSON array.
B) You can create a DB with a field that will store the API body (the whole result from Get data from API) > send this item to a recursive WF with the current list item # and schedule it recursively +1 until you reach the last item.
Indeed, I had already considered idea B, which involves creating a database to temporarily store JSON responses in a dedicated field. This method seems to be one of the best options because it would also allow me to create different types of fields to store requests based on the API endpoints, and to execute them as items on a to-do list.
Regarding option A), upon reflection, in the case of a recursive workflow, there is a risk of missing certain elements, especially if a field ends up empty, which could lead to a shift in data recording. Moreover, although the workflow on list seems to be a better alternative despite the large number of JSON parameters (about 25), I feel that this function consumes a lot more WU and could overload the server.
I will perform further tests to be sure of the workflow on list and compare it with storing in a database field.
Everything would be so much simpler if Bubble simply allowed this operation directly.
If you are on new WU plans, the schedule on a list is cheaper to run, faster and more reliable than before. Also, you don’t need to create a DB thing and fetch for the data recursively. However, this cannot be used if you could have racing issue.
Well spotted! Indeed, the new workflow system on list seems much more interesting and less demanding than the recursive workflow.
I’ll try it out and see how it goes on my side. As for requests with not many parameters, it should be fine. But when it comes to requests with a large number of parameters (20+ parameters), it might get complicated :-/
