Backend Workflow Malfunction

Hi Everyone!

I have an ERP on bubble and some workflows are suddently malfunctioning today even without any changes in it in the last month.

Is anyone experiencing the same thing?

In my case, the user schedule a workflow to run on a list and it works only for the first items.

No recent changes and it worked fine for over a year.

1 Like

Also experiencing issues all of a sudden with a loop backend workflow

Is there something going on? @fede.bubble

I’ll ask the team. I don’t see anything in our status page about this.

1 Like

The first database entry with this problem was at 12:14pm (Sao Paulo, Brazil)

Had to change the logic cause it was making a mess here

Are you dealing with list fields?

Yep also got an issue with backend workflows being scheduled on a list of data. Very strange behaviour!

What caused the issue for me:

Scheduled a regular workflow1 via an API call backend, with the variables “USER” and “PRODUCTS” (list). I then scheduled another regular workflow2 with a parameter of “PRODUCTS” as a list. Part of workflow2 worked, but then when I had to schedule a workflow on the list of products, something broke. I think it is a combination of who is scheduling the workflow (API client vs a current user (front-end)). Idk

MY FIX:

Running the workflow on a list of Product outside of workflow2 and in the first workflow1.

1 Like

But I can attest that this was working perfectly fine before today.

Fala joĂŁo!

I’ve been running through some issues on backend workflows too, sometimes the data shown on the front end is not the correct one, once I refresh the page, the data updates to the correct one.

From what I can see, Bubble changed something during their Tuesday maintenance window, I’ve noticed weird things happening since then.

  • A looped backend WF stopped working after the first run. After trying to see what the issues is, I figured out that bubble is passing the rest of the list, after the first run, as an empty list for some reason. It just seems like when you use list 1:minus item: list 1: first item the entire list gets emptied (at least in this case)

  • A looped backend wf in another app went nuts with rescheduling itself on a loop (even though I had a condition to stop this from happening) - condition was ignored.

Both workflows worked perfectly fine without any issues for months since they’ve been created.

Something weird happen today as well. I deployed a backend workflow using a hotfix branch and it worked well. I then pushed another regular update live, and the workflow I created w the hotfix branch was magically gone. I recreated the backend workflow again, pushed live, and now the original backend workflow showed up magically again. Now I have two of the same backend workflows… Wasted time.

Well, it’s clear to me that something is going on with recursive workflows.

The problem ia that I have a Lot of these. @fede.bubble , please, can you help us with this?

Its making a complete mess with the data.

I guarantee you that there was no change on the workflows and they simply stopped working properly.

I’ll ask internally but best way to escalate this is via bug report because the team can review your app along with your report and escalate accordingly

I wonder if this is not what led to our 1.5 Million WU overage based on a workflow condition that did not execute properly and caused a race condition. Still waiting for Bubble to give me feedback.

On a side note, the discussion with my CEO about this and the implied costs basically ended up in me admitting we’ve given Bubble a blank cheque with their (fair to Bubble) WU model. You can imagine how the rest of that conversation went…

Edit: Forgot to add, our issues started around the 25 / 26th and the pawpaw properly hit the fan 1 March. As it is I cannot take a chance to run any more scheduled tasks. Bubble automation down the drain.

1 Like

VĂ­deo:

This is whats happening for me

What do the logs show?

We are also experiencing issues with backend workflows. Ones that itinerate over lists of things are seemingly partially executing.

I’m not seeing anything relevant in logs yet, but we haven’t had to do a lot of troubleshooting on workflows historically.

yep bubble.io backend workflows are dead to me. Aweful debugging on API’s and so many mysterious issues when performing on list schedules. Time for complete redevelopment to self-hosted n8n…

Hey everyone,

Has there been any resolution to this issue? I’m experiencing the same unusual behavior with my backend workflows, and it’s causing major disruptions to our operations.

This is really concerning, as my app handles time-sensitive processes, and I have no way to resolve the problem on my end without Bubble addressing it. If anyone has found a workaround or has received an update from Bubble, I’d really appreciate any insights.

Thanks!

No feedback from the support, so far.

I’ve been working with bubble for over 18 months. This is really disappointing and the first time that I doubt the reliability of the plattform and considered changing it. As we are talking here, our databases are being messed. I just realized that this impacted my operation significantly.

1 Like

Thats pretty much what our problem was as well. The condition was to say if the list is longer than 20 then reschedule. We have proof in the logs the list was less than 20 yet it decided to reschedule and when this happens it starts with a massive list from the beginning.

I also got a message back from the Bubble team regarding recursion limits. The short of it is that you have to build very short, WU inefficient workflows in order for recursion protection to be of any use whatsoever. (More dev days wasted undoing an optimization exercise that resulted in significant savings and more dev days needed to migrate the workflow to SQL to mitigate the cost and risk :frowning: )

The problem I have with this whole WU thing is that I’m moving away from the Bubble DB more and more into SQL. I have probably close to 200 SQL storedprocs that I now need to call / use to manage WU. Any lowcode time gains now gets allocated to writing SQL.

The path forward is becoming painfully obvious. I guess if I wanted to see a silver lining in this I could call the effort I’m forced to do a “phased migration”. :slight_smile:

Yes, I’m not impressed with the generic messages received from support.

BWT, I just tried checking if there’s a number I can call on their website so I can sort this out with a human. Couldn’t find one or am I blind?