Need alternative for recursive workflow on big list

[sociosemp]

@telaholcomb

Each entry is a price that needs to be subtracted from the nth price to come up with the 1 week, 2 week, 4 week change.

Example:
478.80 - 483.60
478.80 - 459.69
478.80 - 451.09

Repeat for each of the 4,000 prices that are loaded into the database. And I need to do this for each new Thing these prices are for.

But the only way I know how to let the workflow know the correct order (which price is item # 1 vs item # 7) is to send the prices as a list sorted by date. Here’s the backend workflow (BEWF) I currently have setup:

This is only for initial upload of the Thing and its prices. It’s 15-20 years of daily prices that’s why it’s such a big list. After initial upload, daily updates as new prices come in are not an issue.

I’m looking for an alternative way to process these initial calculations since a recursive workflow is so WU intensive especially with this many records.

Hey @telaholcomb

I wrote an step by step for help you. I think that can be good for your problem.

Since processing 4,000+ prices in a recursive workflow is too WU-intensive, here’s a more efficient approach:

1. Use Bulk Processing with API Workflows (Instead of Recursion)

• Go to Backend Workflows and create a new API workflow (e.g., "Batch Price Calculation").
• Set it to process a batch of 200 records at a time instead of all at once.
• Use “Schedule API Workflow on a List” to process these smaller chunks sequentially.
• This reduces WU usage compared to fully recursive workflows.

2. Precompute & Store Price Differences

• Instead of calculating differences in real-time, create a new field or datatype to store precomputed values.
• When new data is uploaded, run the calculations once and save them for quick retrieval later.

3. Offload Heavy Calculations to External Tools (Most Efficient for Large Datasets)

If the dataset is too large for Bubble to handle efficiently:

• Export data to Google Sheets, Airtable, or a database (e.g., PostgreSQL, MySQL).
• Use a script (Python, Google Apps Script, or Make.com/Zapier) to process price differences automatically.
• Re-import the processed data via Bubble’s CSV upload or API.

4. Optimize Data Sorting

• Ensure your prices are sorted by date before processing.
• Store a unique Index ID for each price entry to reference previous records more efficiently.

Best Approach for You?

• If Bubble-only: Use batch API workflows with precomputed values.
• If performance is key: Offload calculations to external tools and sync results back into Bubble.