Limitations running workflows on a list

Need help
2

Hey Alex,

Thanks for the clear, well-structured question. Couple things to note upfront:

  • The main limit with big list operations right now is a time limit on workflow run execution – ie, from the evaluation of the event to the last action of the workflow. That time limit is 5 minutes for workflows run via the scheduler, and slightly longer (it’s somewhat variable, but think of it as 5 minutes and you’ll be safe) for workflows run in response to a user action or API call. If a workflow crosses this limit, it’ll end with a “too busy” error that will stop further processing.

  • We have limits on the max length of a list that can be stored as a list-type field in the database; currently it is 10,000 on the main cluster. Attempting to store a list longer than that will result in an error.

  • We have a limit on the number of results that can be returned from a search; currently it is 50,000 on the main cluster. Exceeding this won’t result in an error: you’ll just get 50,000 results instead of 50,001 results. Searches that don’t actually return results, like counting the number of items, will process more than 50,000 (so, you might see higher counts).

So to answer your questions:

  1. The limits described above apply in the following ways: if it takes > 5 minutes to schedule all the workflows, the schedule API workflow action will fail with a “too busy” error. Items scheduled prior to that happening will still run. If it takes more than 5 minutes to load the initial list (ie, a very slow search), nothing will get scheduled. And the max search length and max item list length constrain how long the source list can be.

  2. There are no hard limits on S, but in practice the 5 minute constraint will apply. I’ve never seen workflows with thousands of steps, and it’s possible things will break / you’ll run into weird bugs if you try to do that.

  3. This is the 5 minute limit mentioned above.

  4. Once a workflow is scheduled to run, it is independent from its peers. So if you schedule a list of a 100 to run, that initial scheduling action is a single operation subject to the 5 minute limit, but as soon as the item run gets scheduled, it’s now totally separate from the other 99 items… Bubble doesn’t consider them related or track limits across them.

  5. The interval determines how far apart Bubble attempts to start each workflow. So if the interval is 1 second, Bubble will try to start each workflow 1 second apart. (Note that Bubble doesn’t strictly guarantee exactly when things will run… this is a best effort attempt, though usually if your app has capacity, it’ll run within a second or two of the target start time. We do guarantee it won’t run too early). If interval is left blank, we default to 2 seconds. In light of the new pricing / capacity management, we reduced the minimum interval to 0.2 seconds, so if you type ‘0’, you’ll actually get 0.2. With the new pricing, it is significantly less important for protecting Bubble’s infrastructure. However, it is still useful to help users manage their own capacity consumption; spreading out the scheduled workflows avoids burning all your apps’ capacity on running them. (We’ve taken other measures to help with this; for instance, if your app is low on capacity, we will wait to start scheduled workflows until it recovers, and we also put some limits on the number of workflows any one app can be running in parallel).

  6. No, I don’t think this introduces any changes to the above.

8 Likes