[New Feature] Scheduling API workflows can now be done recursively

Hi all,

We had previously restricted scheduled API workflows from scheduling themselves (or scheduling another workflow that then scheduled itself). This restriction is now removed.

Recursive scheduled workflows can be useful in situations where you are processing an unknown quantity of data that might be quite large. For instance, the following workflow will make sure every Thing in your database has been processed within 7 days:

Processing data this way is generally a better approach for large quantities of data than using the “Schedule API Workflow on a list” action. A brief comparison:

Scheduling on a list Scheduling recursively
Workflows may overlap Workflows run one at a time
No way of knowing when all workflows are done Workflow can check if there is more data to process, and do something else if there isn’t
Limited by the time / capacity it takes to search for the list and schedule all the items (max 5 minutes) Can continue indefinitely
Faster and simpler for short lists (< 50 - 100 items) More reliable for long lists
Can burn through a lot of capacity if workflows run in parallel Capacity usage is limited to one workflow at a time

A couple notes:

  • In order to avoid maxing out your app’s capacity, I recommend starting with a 5 second gap before scheduling the next item in the list. You can then shorten this gap once you see what the capacity impact is.
  • If you make a mistake and create an infinite loop that does not look like it will ever end, you can end it by manually deleting the next scheduled run in the Logs → Scheduler tag. You can also modify or delete the API workflow itself: deleting or changing the workflow won’t remove a scheduled item from the list of scheduled tasks, but when it comes time to run that item, Bubble will use the updated workflow or do nothing if it has been deleted.
  • If things get really bad, you can pause and resume all scheduled workflows for your app using the new “Pause tasks” button:

89 Likes

Thank you! This is huge.

1 Like

great article! Thanks for the new options…

1 Like

@josh thank you for enabling this. Really excited to see it!

As well, I appreciate that you went into detail - especially the tips for mitigating issues and the comparison between scheduling on a list versus scheduling recursively.

4 Likes

Thanks! This is a fantastic development which will help for many use cases.

1 Like

Do I :eyes: LOOPS!?

Nice…Thanks team Bubble.

Thank you so much for this feature and the comparison! :slight_smile:

1 Like

Oh, this is fantastic! :raised_hands:

5 Likes

Great work!

So I get this right, and I am sure there are still a few of us who need to see this in a flow chart :slight_smile: - anyone chime in - can some one please give a real world practical example?

3 Likes

This is the best day of my life!!!

2 Likes

So much awesome :slight_smile:

1 Like

So Thing A can call Thing A. That’s about it. It’s helpful.

2 Likes

This is great stuff, thank you Josh + team!

It is a “Do … While” loop. Where the “While” is able to be dynamic i.e. you don’t need to know the answer before you start.

Example : Update all the users who have not been active for 7 days …

Previously you would have to generate the “list of users” in advance.

Now you can loop the API workflow while there are still users to process. Although it may be that this is still better done by “schedule on a list”.

Example : Create a thing for each month in a year.

Previously you would have to generate a list of 12 numbers or months, and do a workflow on this list.

Now you can “Do … while there are less than 12 months created”.

A more complex use case is one where you are doing some processing within the loop that might be difficult to work out in advance.

So say you want to add up invoice items until the total is > ÂŁ500 and then create something.

Now you can loop while the total is < ÂŁ500.

Obviously this allows you to … “Do … While black = white” so loop forever.

11 Likes

Thanks @NigelG just needed an old school explanation :smiley: Now I know what it is this is awesome!

Thank you very much for that awesome update!

Hi @josh, nice explanation.

Can you clarify, does the scheduled workflow wait for all remaining steps of the current workflow to complete, as well as its scheduled date? Is this only for recursive workflows?

Definitely one of the best updates recently!

1 Like

This is great – thanks!