What happens with scheduled (or dependent) workflows when server problems or outages occur?

I’m curious to hear how Bubble developers deal with errors and outages that might affect scheduled workflows - or how you manage dependent processes that would ‘break’ if a particular step fails.

Does this occur in your app?
How do you avoid it, ensure you’re aware when it happens, and restart the sequence?

For example, we have a survey app that automatically sends questionnaires to raters on a regular basis, with steps such as:

  1. Project launched, which leads to an API being scheduled for the launch date
  2. …the API creates a questionnaire for each rater on the launch date
  3. …and schedules another API to send out emails with survey links to those raters
  4. …which schedules another API in a month that creates the next questionnaire for each rater
  5. …and triggers the API to send out emails
  6. …and rinse and repeat.

Although we’ve done our best to develop a robust app using Bubble best practices, our fear is that a ‘break’ in a particular step would derail a whole project (e.g., if the API failed to create a survey for a rater (2), subsequent steps would not be scheduled and the project would stop).

Thanks so much!

-Chad

2 Likes

Mechanisms to protect and complete workflows must be put in place. This problem happens with all sorts of servers that crash. Keep a history of all scheduled workflows and ensure that they are executed to the end.

It is important to stop your operations when you receive a technical problem email from Bubble (prevention).

@eve A good idea will be for Bubble to do it automatically on activated request.

6 Likes

Thanks, @JohnMark , it’s helpful knowing that others are in the same boat. I may have missed it in the Bubble docs and posts, but I haven’t seen much about how one might manage or be informed of these workflow breaks - even though I’m sure many people share this concern.

We’ve done a few things that we believe would help, such as the following, but I’d be interested in hearing any other solutions from experienced Bubble developers:

• Use recursive workflows to process any lists, as opposed to ‘schedule API on a list’, which seems to cause errors for many users. And we add at least a 10-second interval before rescheduling the same WF.
• Storing all ID numbers for scheduled APIs in a relevant table (e.g., a scheduled survey’s API ID is stored in that survey’s table, so we can see when it’s missing). This allows us to quickly check that an expected API has actually been scheduled.
I’m assuming this is what you mean by ‘keep a history of all scheduled workflows’, although I’d love to hear what others do to ensure all WFs are executed (or a way to be notified when this happens).
• Created a ‘SuperUser’ type (i.e., us) that reveals a dashboard with a list of potential errors. For example, during a stress test, we noticed that the ‘end date’ for an ‘active’ survey was earlier than the current date (i.e., a current survey should always end or close in the future). This occurred because a WF broke and subsequent surveys weren’t scheduled. So, we added a repeating group to this SuperUser group that searches for surveys with this problem, and emails us if count > 0.

I also came across your post about hiring someone who was able to code for recursive WFs, which seemed to improve performance and reliability. It’s nice to know the option is available:

Schedule API Workflow on List - failing for lists of 100+ items?

Thanks again,

-Chad

2 Likes

It is absolutely necessary to install an emergency button in your admin in order to warn the user to come back later.

Great points :+1::+1:

4 Likes