Need help looping over JSON response

My app has a backend workflow that is acting as a webhook. A 3rd party service calls the workflow with data in the format:

{ "body": [{ "id": 12345, "name": "foo" }, { "id": 12345, "name": "bar" }]}

Within the backend workflow that I’ve exposed publicly I’m able to access the returned data as types such as response's body:first item's id. However, I need to loop over each item in body and perform some set of actions.

In the past, I’ve used recursive backend API workflows to handle looping over data, but in this case, I can’t pass a parameter of type response's body to a new API workflow and I can’t schedule a call to the public-facing endpoint in order to call it recursively.

How the heck are you supposed to loop over data like this? The response I’m getting could potentially contain thousands of entries, too, so writing everything to the database as temporary storage seems way too wasteful.

Any help would be greatly appreciated!

You would need to create one more workflow (internal/not publicly exposed) to accomplish this. In that WF create a two key-value pairs, one for dataset you are interested in processing with the Type of array (could be text), and another for iteration type of number.
In the WF actions “do any processing” by using iteration variable as item number, and call the same WF again in the last step by incrementing the iteration number by one. Add a condition so that this step executes when iteration < item array count.

Hope this helps.

Good Luck!

1 Like

@logicleapconsulting 's solution looks good, but is it scalable to thousands of list items?

@chris72 what kind of operations will the app do with the list?

Possibly you want a serverside action such as Toolbox server script, to run the data through a Node.js script and output aggregate data for the Bubble app?

If yes, you’d want to:
API endpoint: set to Automatically detect, and either grab the whole body as one Text, or turn on the “include raw body” option, to pass to the server script.

Or maybe just save the API endpoint’s data in the format it is received, save to database and loop through it on the front end?

Yes. I would make a change though to optimise for WU:

For the Schedule API workflow api-processing, just send the data. No need for iteration parameter.

In the api-processing workflow, create a new thing using data:first item

Schedule the api-processing workflow with data = data:minus item(data:first item) only when data:count > 1 (as when there’s only one data item left, we know that it’s all been processed)

We don’t need to send all of the data for things that have already been processed every time, so we can remove them from the list and forget about them.

1 Like

Thanks for the ideas everyone.

What I’m trying to do is for each entry in the list:

  1. Check to see if there’s an entry in the database that has a matching ID.
  2. If yes, update the record. If no, create a new record.

The issue I’m running into when trying to do recursive looping is how to pass the data through. The source data I want to iterate over is an array of arbitrary objects. I haven’t found a way that I can create a backend API Workflow that accepts a matching data type without either storing it in the database or having it be an external API. If it’s an external API then I can’t schedule it, which means it can’t be recursive.

For now I’m going with “Schedule an API Workflow on a list” as an immediate solution, although I’m not hopeful about how that would handle a list with potentially thousands of items.

Unless I’m missing something…

1 Like

What are the expected counts for:

  • matching entries that need updating
  • matching entries that dont need updating (if any)
  • total entries in database

A very broad estimate, but on average I would expect numbers around:

  • 50 matching entries that need updating
  • 50 entries that need objects created in the database
  • 30,000+ entries in the database for the Items being updated

Don’t be. I rarely trust it with more than 20. I know some will try 100 but certainly not 1000s.

Based on everything I’ve read, then, it seems like there’s really no way to handle a situation like this in vanilla Bubble.

My options for handling larger data sets seem to be one of:

  1. Process the list somewhere else (like AWS Lambda) and use the data API to read/write the entries that I need.
  2. Write entries to the Bubble database in some sort of temp table and then process those entries since Bubble will then “know” about the data type.
  3. Maybe write a plug-in to iterate through the list and directly read/write from the Bubble database (if that’s possible)?

Both of these options seem unnecessarily complex and costly (in terms of WUs).

Have you tried creating multiple lists?

List of ids (12345, 12345) (it’ll be body:each item’s ID or something along those lines)
List of names (foo, bar)

Still reference the first item of each list, and remove the first item from each list when you’re done with it.

When the incoming data is thousands, do you expect most of them not needing updates? I.e. they match existing records with no change?

These could be prefiltered (Lamda / data API) before handing the remaining to the backend workflow.

Edit - if doing the prefilter, may as well do the entire update there as well, unless you want to include other data types, other workflow steps, etc.

1 Like


A little tweak to your option #2: " Write entries to the Bubble database in some sort of temp table and then process those entries since Bubble will then “know” about the data type."

If you only have ~100 entries coming in and 30k records in the DB, why not just create the 100 records as they come in?

Once they are in DB, you can loop thru them at your (really, Bubble’s) leisure, checking if they are duplicates or not (checking for unique slugs can eliminate WF heavy searches).

If a record is a duplicate, you can merge it into the original record or delete the original record (if that works in the data structure).

1 Like

Why have a temporary table and all its overhead and not just put the data directly into the table itself and then clean up the dupes?

I did consider that. I was concerned about being 100% sure that Bubble is maintaining the order of each list so that item 1 in the ID list still lines up with item 1 in the names list, etc.

I would love to put them directly into the table itself, but I’m not sure how to do that without looping over the array of data I get from the API. Also, the fields that are returned by the API don’t match the fields for the Things I’m creating/updating, so I need a step where I translate.

@mishav, doing this all in a lambda function using the Data API is a different path I considered. One of our other developers has experienced rather annoying rate limits and timeouts when working with the Data API - timeouts that shouldn’t be happening according to the Bubble docs - so I wanted to try the Bubble workflow approach first.

We’re expecting that the vast majority of processes will need to work with < 100 entries, so the thousands scenario should be rare. If we start running into issues then I imagine I’ll have to revisit the lambda function approach.

It’s always nice when the best Bubble approach is “don’t do it in Bubble” :joy: .

1 Like

It would - there’s no way for that not to happen. But good that you found another solution!

I think for Bubble to know about the data type, the webhook endpoint needs its parameters to be automatically calculated, and initialized.