[Feature Enhancement] Schedule 100k Workflows with Schedule API Workflow on a List

[steven.harrington]

Hi everyone,

I’m Steve, a product manager here at Bubble focused on helping you scale your apps. I’m excited to share that the Schedule API Workflow on a List action can now schedule up to 100K workflows, making this a robust, native option for completing bulk operations. This update builds on previously announced improvements to reliability and performance and represents an important step in strengthening Bubble’s capabilities as an enterprise-grade platform where you can scale confidently.

Note: If your app is on a legacy plan, you will need to upgrade your plan to unlock these improvements.

Schedule API workflows with confidence
A few months ago, we announced enhancements that increased the number of workflows that could be scheduled with Schedule API Workflow on a List. But we know we left a critical question unanswered: exactly how many workflows could be scheduled?

This question was difficult to answer because the limit could vary dramatically depending on the memory available at any given time and the parameters of the workflow being scheduled. To eliminate the resulting ambiguity, our engineering team made changes to batch the tasks being scheduled and clear the cache along the way. This allows us to avoid hitting a memory limit and significantly reduces the variability around the number of workflows that can be scheduled. Now we can reliably schedule 100K workflows irrespective of external conditions or the specifics of the workflow parameters.

If your app is on a dedicated server (available on the Enterprise plan), you will be able to schedule at least 100K workflows — and in many cases, multiples of that. There is no explicit limit since the capabilities can vary based on your instance’s configuration. Please reach out to your Technical Success team if you have questions about your unique instance.

Capitalize on performance and reliability at scale — by default
The enhancements we’ve made over the past few months have helped us significantly speed up scheduling and execution of backend workflows for bulk use cases. At the same time, we’ve improved workflow reliability and protected performance for your app’s frontend users during processing.

With this foundation in place, we’ve updated the default (empty state) interval for Schedule API Workflow on a List to schedule workflows at a much higher frequency. If you’re using the default interval, you’ll see a significant difference in how quickly a given set of workflows completes. Existing actions with intervals that have been set manually won’t be modified automatically, but you can update them easily in the Bubble editor.

Here are results from benchmark tests for simple bulk operations using Schedule API Workflow on a List to execute 100K workflows compared to scheduling them recursively:

	Schedule API Workflow on a List	Recursive
Delete 100K things	20–25 min	6–7 hrs
Copy 100K things	60 min	10 hrs
Modify 100K things	75 min	12 hrs
WU for scheduling 100K workflows	~12,000 (~0.12 per workflow)	70,000 (0.70 per workflow)

The results above are approximate and intended to illustrate some simple examples. Results will vary based on the specifics of your app and workflows.

Looking forward
We’re committed to giving you more ways to support your apps as they grow. If you’d like to learn more about our initiatives across the Bubble platform, our new Director of Platform Engineering, Payam, shared more about the progress we’ve made and our plans for the coming months in this post.

Over the coming months, the Scale team is investing further in observability, including new functionality to help you track progress, understand outcomes, and troubleshoot issues for workflows scheduled using Schedule API Workflow on List.

As a reminder, for technical reasons, these improvements are only available on our new (workload-based) pricing or Enterprise plans. If your app is still on a legacy plan you can upgrade to one of our new plans to take advantage of these changes and the rest of our exciting scalability improvements on the roadmap for 2024.

[aj11]

This is awesome! Huge improvements for our use case. We rely heavily on API workflows for many features and this opens a ton of doors for us. Thank you!

[hoke]

What about simple native looping functionality.

[lindsay_knowcode]

So do read this correctly?

… where previously advice would have been to use recursive workflows to process lengthy lists (I have had this recommended by Bubble support)

… now “Schedule … on a List…” is a magnitude faster and cheaper? (and now considered reliable)

Have I read this right?

[aj11]

That’s my understanding.

[boston85719]

@steven.harrington THIS IS FANTASTIC…but, could you guys maybe post here before implementing them into the system? I saw this default setting of interval with the new recommendation to leave blank many hours ago and the up to 100K so was very curious about what those changes were there for (I say this ONLY because I NEED to find something wrong with THIS).

Glad to know it is for a kick-A@# improvement that is going to change how we build apps from now on. Thanks Bubble team!

@lindsay_knowcode

Have I read this right?

Lie your eyes do not

[cmarchan]

Woot woot woot!

@steven.harrington Please elaborate on what is considered as a sorted list:

Thank you!

Hard limits | Bubble Docs

[NexradJosh]

Incredible! This is fantastic news.

[aj11]

50K is still super high, but this is a big caveat for sure. Needs clarification.

[TipLister]

great, this is nice.
do i understand correctly that you have reduced WUs so that the action “create a thing” or “make changes to a thing” is 5x cheaper than it used to be when we bulk schedule?

or is the WU cost you write the cost of 1 scheduled wflow?

this makes a big difference because 50k modify actions are too WU expensive to currently run on bubble and i just do them on xano. i would do them on bubble if they were cheaper, it is just the WU markup is too high.

[eli]

@steven.harrington

If your app is on a dedicated server (available on the Enterprise plan ), you will be able to schedule at least 100K workflows — and in many cases, multiples of that.

Steve, does Bubble handle any type of throttling of these workflows if you begin to max out capacity or is everything scheduled out immediately with no control over when they are run regardless of capacity usage?

Additionally, one of the biggest issues with ‘Schedule on a list’ has been the lack of insight into what actually happened to any individual workflow. Did it error or not run? We simply don’t know.

Has there been any improvement on this front or can we be 100% certain that if we schedule 100k workflows they will run.

[georgecollier]

@cmarchan

Woot woot woot!

@steven.harrington Please elaborate on what is considered as a sorted list:

Sorted lists only return 50,000 items in Bubble. So, using :sorted operator will make a sorted list, or, using the sort by field in a search (among other things).

[tylerboodman]

Yea right now it’s a shotgun blast of 100K workflows… A great improvement would be allowing us to specify a workflow to trigger at the end with # of successful, # of errored (or the list of API IDs), or even let us return data from each workflow and that final triggered one can contain returned data from each.

Either way @steven.harrington great improvement making it more reliable

[akalati]

You speak of “schedule API WF on a list” but it seems from the context of the post that all of this also applies to other list operations, like delete a list; make changes to a list; etc. Can you clarify that please?

[onerbarisoner]

@steven.harrington This update caused scheduled API workflows to not function properly and it needs to be fixed URGENTLY.

I have done tests regarding the issue and I am sharing the steps for you and other forum members to test.

1. Create 2 data types:
   

2. Define the fields for these types as follows:
   Item’s Fields:
   

   Item Group’s Fields:
   

3. Manually create one “Item Group” from the Data tab
   

4. Create an API workflow in the backend and add 2 parameters:
   

5. Add 2 actions to this API workflow:
   

   5.1. Create a new Item
   

   5.2. Make changes to Item Group
   

6. Create a test page and add a test button
   

7. Add a button click workflow and add the “Schedule an API Workflow on a list” action.
   

8. Select the API workflow we created and input the parameters as follows
   

   

   8.1. I split the arbitrary text with commas to create 10 items. Alternatively, a text list can be taken in another way
   

   

   8.2. Since there is only one “Item Group”, I’m selecting the “Item Group” we initially created using “Do a search for” with :first item. You can use different methods as well.
   

Result: In the end, there should be 10 “Items” created and all of these items should be added to the Items List of the “Item Group”.

Actual Outcome: 10 Items were created. However, only some of the items were added to the Items list of the Item Group. It behaves differently each time. Sometimes only 1 item is added, while other times 2-3 items are added to the list. It’s not functioning correctly.

The following video demonstrates this behavior more clearly:

Items:

Item Group with only 2 items:

Finally, considering that many apps rely on scheduled API workflows, I find it unacceptable that such a core function hasn’t been thoroughly tested. I hope it will be resolved soon.

[georgecollier]

Seems like a race condition problem. Now, with scheduled workflows running faster, it’s more of an issue? Speculating though - someone else might be able to elaborate more. I’ve never run into race condition issues on my own apps.

[aj11]

This is not an issue with the release of this feature.

You’re running into a race condition where all the workflows are editing the list at the same time (“add” is misleading because the action it’s still editing the whole list via API). The only reason you did not run into this issue before was that the performance was just slow enough that the workflows ran with enough time between them not to conflict with each other.

To solve this, you either need to enforce an interval long enough to eliminate the race condition (make the workflow on a list intentionally and predictably slower), or you need to set up something that waits until all items are created, then sets the list one time in the end.

I’ve accomplished this before by adding a “last item” parameter, with the logic “If This Item is List of Item’s Last Item”. Then, in the workflow, just add a step to schedule a final workflow or custom event a second later to set the list based on a search. I would have it schedule it out a second (as another workflow or customer event), as opposed to just setting the list in the last workflow, as there still might not have been enough time by the last workflow for all items to appear in a search.

The irony here, is that this new release is literally so good it’s causing issues. The only way to avoid this would have been not to make the improvement. It’s an unfortunate but unavoidable problem that would never have been caught with any amount of prior testing.

[onerbarisoner]

Thank you for the explanation. If that’s the case, I need to take back my words about not being tested enough.

I just wish things like this waiting or locking were automatic in scheduled workflows since all operations are done by bubble. I’ve never run into a race condition before. New fear unclocked

[cmarchan]

Yeap that is the conventional wisdom for sure George

[code-escapee]

@georgecollier

I’ve never run into race condition issues on my own apps.

How have you managed to avoid race condition issues? I want the DAYS of my life spent building queue workarounds in Bubble…