Hello all,
I am looking for some assistance with how to best handle recurring workflows in bubble. In short, my app allows users to schedule monthly reports.
How does Bubble handle when lots of scheduled events are scheduled for the same time? For example, if all reports trigger at 1am and there are 200 reports, will Bubble intelligently space them out? Or will it try to run all at the same time and possible cause performance issues, or worse, cause workflows to fail entirely?
Does Bubble complete one workflow (and let’s say it takes 10 seconds), then the next, and so forth? Meaning the whole process could take an hour to complete?
Thanks
Andy
It looks like you will need to space them out yourself. This is from the Bubble docs:
How scheduled workflows are queued
The scheduler is a queue, and can be imagined as a funnel. If a funnel is filled with water faster than the rate at which it drains, it will be congested and it will affect its flow. Similarly, if the scheduler is “filled” with a lot of simultaneous or very shortly spaced workflows, the scheduler can be “congested”, leading to delays.
Avoiding delays
To avoid delays when working with scheduled workflows, follow the recommendations below:
-
The more heavy the workload of the workflow, the more server capacity it will use. This is both affected by the number of actions in the workflow, and the complexity of those workflows
-
Database operations that work on a large volume of data (searches (especially with advanced filters) and actions that work on a list of things are particularly taxing
-
A high number of actions, and in particular if they depend on each other (such as using result of step X ) can also slow things down
-
If you are scheduling a big number of workflows (by use of Schedule API Workflow on a list or recursively), it helps to consider spacing them out with some time in-between each cycle. How much time depends on the complexity of the workflow and how much capacity your app has in general
- If the process does not need to finish at the highest possible speed, you can experiment setting a delay time of several seconds to control the capacity.
Overall, server-side workflow are a very powerful feature, but you may find that you need to experiment a bit to learn how to build workflows that don’t overwhelm the system.
Also I know if you schedule a workflow on a list then you can add a variable to space out each action but if users can all separately schedule recurring monthly workflows at a set time, like 1am on the first of every month, then they will all be scheduled for that time and the spacing out will not be handled, leading to a “congestion”. Possibly, for an example, a solution would be to have say a field on the User that is a yes/no if they want a monthly report. Then have one recurring monthly event that does a search for Users where that field is yes and then run a workflow on that list and use the time spacing variable provided by Bubble instead of separately scheduling events.
1 Like
If you have 200 users each schedule 1 report for the same time, then yes they would all be processed at the same time and there would be a chance of a timeout. I’d set it up so that what looks like scheduling to the user is really just making a database change to the relevant datatype (or a new datatype), then each day I’d run a workflow to search for all reports that need to be created and loop through them 1 by 1.
Hi Alex
So, that’s what I have tried to do, but running into issues most likely related to privacy rules. Even checking the boxes to ignore privacy rules on the workflow doesn’t seem to work.
Andy
What makes you think it’s related to privacy rules? I don’t see how that would be the case if you’re ignoring privacy rules.
What do the logs show?
The logs show Action condition failed, which I am pretty sure is because it wasn’t being passed an actual item. At this point I’ve made a bunch of changes, so it’s hard to replicate the same issue.
Ultimately I think trigger a single job that identifies any reports that are needed would be a good solution, but then the question is, what item is used to run the recurring event? I had it set to user and then use my current user (admin) to trigger it.
The workflows I had were like this:
Daily Report Scheduler (recurring daily workflow) calls API “Daily Report Controller”.
Daily Report Controller finds any reports that are to be created this day and then calls “Daily Report Worker” on a list. I am limiting list size to 10 (for testing) and then reiteratively calling the controller again until all items have been processed.
Daily Report Worker creates the report
In my testing it worked fine for any reports under my name. However, no reports under other users were created. This led me to the privacy rules issue. When changing privacy rules to ignore, it actually stopped creating reports altogether.
I could use the actual user to run it, in which case each user would have a recurring job, but then they are also all at the same time, so it doesn’t make sense.
It seems that running the recurring event on the report level is the best way to make it work, but that results in the issue of not having control of spacing them out…
I am going to have to play with this some more
The admin user can trigger it the first time and then you can construct the workflow in a way that each time it runs it will schedule itself for the following day.
Regarding the privacy rules, it’s hard to say without having a look at it. I suggest to doublecheck that privacy rules are being ignored on both actions (front and back end) and also that the parameters necessary for the backend workflow to run are being passed.
