Delayed API Workflows [Outage]

Bugs
1

I have already filed a report with the team but there appears to be about a 5-15 minute delay on when backend scheduled API workflows are supposed to be ran versus when they actually are executed.

We were affected by the live outage last week of the randomizing data objects so we might be grouped into this one as well.

We noticed the onset around 1:32PM EST today affecting live version. We had not pushed any updates that would have caused this.

Nothing on the Bubble status page yet so want to see who else is experiencing.

1 Like
2

We experienced a 2-hour!!! delay without a corresponding spike in WU usage.

Using the following script, run via the browser’s console, we canceled specific scheduled workflows:

function cancelActionForRow(rowElement) {
    // Get the elements within the specified row
    const cancelBtn = rowElement.querySelector('.cancel-btn');
    const apiEvent = rowElement.querySelector('.api-event');

    // Check if the api-event is either "send_email"
    if (apiEvent.textContent === 'send_email') {
        // Click the Cancel button
        cancelBtn.click();
    }
}

tableRows = document.querySelectorAll('.scheduled-row');


tableRows.forEach(function(row) {
    cancelActionForRow(row);
});

terrible experience
the app wasn’t working at all

1 Like
3

@josh

4

This is happening again for us. Workflows are now being delayed from 5-10 minutes. Please give an update here. This is another outage that has no report on the status page.

Almost like clockwork on the timing as well. around 10am PST

3 Likes
5

Don’t forget to submit a bug report at Bug Report | Bubble. That’s the best way for the team to work out where issues are happening and fix them.

1 Like
6

thanks George. Yeah, best thing to do is submit a bug report, that’s how things escalate to the team in case of emergencies. I’ll go check internally on this

7

the best way to waste your time

1 Like
8

Yeah it’s frustrating that often reports have to be submitted for them to know there’s an issue, but that’s much like your own Bubble app, where you’ll only notice when people report things… it takes like 3 minutes, it’s worth reporting if you find an issue… then, if it’s still not fixed, definitely feel free to complain :wink:

6 Likes
9

Great enthusiasm

10

I don’t know what your situation is atm, but Bubble support couldn’t help us with our bug report.

Regarding the Bubble Scheduler
I’ve analyzed the issue.

They indeed enhanced the Scheduler, but didn’t test it well as I see.

It works well if the scheduled workflow contains synchronous actions only.
If the scheduled workflow contains such actions as scheduling (asynchronous) other workflows - when we have a problem.

So, the update ([Feature Enhancement] Schedule 100k Workflows with Schedule API Workflow on a List) is good for simple scheduled workflows. But the update downgraded the speed and performance for scheduled workflows that schedule other ones.

The issue is that when you schedule an API workflow on a list (even if the list contains 1000 items) and leave the interval field empty, the system tries to process the list as quickly as possible and execute all actions within the workflow. If there are steps inside the workflow, such as scheduling another API workflow with a scheduled time of the, for instance, current date/time, the system cannot properly assess how to schedule these actions correctly. At the time of scheduling the workflow (Schedule on a List > Schedule API Workflow), the system cannot account for the queue that is being created simultaneously.

What confuses me:

image
image1280Ă—648 103 KB

Honestly, I think the examples attached on the photo are not actual.

Such apps as Hiring Marketplace have tons of scheduled complex workflows (synchronous, asynchronous, etc.).

Our client’s app’s monthly WU usage is kinda 11M.
We got a delay of a couple of hours while the WU usage for this short period was 350K.

Since Bubble support wasn’t able to assist, I’ve figured out a quick solution.

Instead of just Current date/time:

image
image802Ă—230 18.7 KB

image
image900Ă—608 21.7 KB

image
image914Ă—886 38.5 KB

The idea is adding a dynamic delay. In my case, I’m generating a random number from 1 until 9.
It doesn’t throttle the Scheduler since each workflow (Schedule on a List > Schedule API Workflow) schedules at different time.

@nic3
please let me know if that helps

11

Granted I haven’t experienced the issue but this seems unlikely - the scheduler is unaware of the contents of a workflow, only that it exists and when it was scheduled, so I don’t see how specific actions could cause a workflow to start running late.

12

This topic was automatically closed after 14 days. New replies are no longer allowed.