Hi everyone — First off, it’s awesome to see the engagement and feedback around this feature. It’s a testament to the vibrancy of our Bubble community! I’m jumping back in to provide some clarifications and address the questions that have come up about this feature enhancement, which (to answer one of them) was rolled out to all users the last week of September.
Performance and Limitations
The limit of Schedule API Workflow on a list will depend on many factors, including the length of the list being passed to the scheduler, the number and type of parameters required for the workflow, and the values being passed for these parameters. In internal testing, we have successfully run this on lists as large as ~70K things. The amount of time it takes for Schedule API Workflow on a list to finish running will vary significantly based on the number and type of actions in the workflow. Overall, we’ve found that Schedule API Workflow on a list is typically 2–5 times faster than doing the same work with a recursive workflow.
Generally, lists are fetched from either existing data (Do a Search For) or an external API. After the list is fetched, all workflows are scheduled in a process that runs on the client side, and then the workflows run server-side until completion. Previously, this scheduling process was the bottleneck. Workflows were scheduled one at a time, which was very slow. And because all this happened in a single client-side workflow, timeouts would occur after five minutes and no further backend workflows would be scheduled. Now, Schedule API Workflow on a list schedules the backend workflows much more efficiently, so the five-minute timeout is no longer a limiting factor.
You can run multiple Schedule API Workflow on a list actions concurrently, but it’s worth noting that we still have capacity limits in place to protect our infrastructure. If your app runs into capacity limits, you’ll experience slower overall execution as fewer workflows will run at any given point in time. If that happens, those scheduled workflows will remain in a queue and be executed as capacity becomes available.
Observability
This update doesn’t include any changes to how errors or silent failures in workflows at run-time are reported on or dealt with. However, we know this is a major pain point, and it’s something we are going to address over the next 6–12 months. Until then, we recommend building a step in the workflow that logs completion so you can confirm how many have been completed.
Using Recursive Workflows
While we’re excited about what Schedule API Workflow on a list unlocks today, right now it isn’t in a position to replace recursive workflows in all cases. In fact, there are a few scenarios I’d like to highlight where recursive workflows are still the better solution:
- When the workflows are not independent (e.g., your use case requires them to run in a specific order, or results need to be passed forward from one workflow run to the next)
- When you want the process to stop entirely if any workflow has an error
- When you need a way to cancel or adjust the time between workflows actively running on a list
- When the list you wish to run on exceeds the maximum length achievable with Schedule API Workflow on a list
Workload Implications
There are no changes to the amount of WU consumed by Schedule API Workflow on a list with this enhancement to scalability. However, compared to running recursive workflows, there are some potential cost savings because recursive workflows require an action to schedule themselves on the next item. These savings could be meaningful when running very simple workflows on very long lists, but less impactful in other cases.
We value your feedback, and our team is working diligently to address the limitations and improve the ability to manage and utilize large datasets in Bubble. We’ll be releasing updates like this on a rolling basis. Your insights and experiences play an essential role in shaping our platform’s evolution.
If we haven’t addressed your specific question, please reach out via the Support center. Your unique use case may require a deeper look. As always, we hope you’ll keep adding feature requests to our Ideaboard and submit any bugs via our bug report form. Thank you for helping us continue to improve Bubble’s bulk data functionality!