It finally happened, I missed something and used 7 million workload units in 24 hrs!

I’ve literally been having nightmares about this happening ever since the bubble pricing change.

I have many apps that use iterations and recursive workflows to process data. I have been super careful, triple checking and testing like mad before pushing to live environment - watching and checking the workload logs like a hawk.

But I made a mistake. I knew this day would come eventually. It’s impossible to get everything right forever, I’m actually surprised that I made it this long without making a mistake.

So I pushed the app live last night, the plan had only just renewed so I had plenty of workload units and I was managing my usage of them super carefully. Today I wake up to “you’ve used 100% of your workload units”. No “usage spiked” email alerts. I start freaking out expecting the worst… millions of workloads and YEP 7 million workload units used in less than 12 hours! A big fat auto charge bill on the credit card of $1000.

I’ve reached out to bubble support but I’m not holding my breath for a credit. They’ve been very clear and strict that they do not give credits after the first credit (which was for a measly 100k units).

I’m posting this as a warning but also as a plea to bubble. Please add some sort of safety guards into this new pricing model! There needs to be something that catches a mistake like this and stops it - puts a hold on it or something.

I’ve done everything I can humanly do on my end to test and test and test - but something slipped through the cracks and it’s got a big fat price tag on it. (Normally the entire month usage for this app is 400k workload units)

21 Likes

LOL

If it wasn’t so expensive this would be funny. I just now got the “workload usage spiked email”

I literally slept for 8 hrs, woke up, checked emails, wrote an angry forum post, turned off the scheduled workflows and started to debug the issue - BEFORE the workload usage spiked email was even sent notifying me of a problem!

Bubble you’ve got to seriously fix these issues.

9 Likes

Do you have the checkbox that prevents overage charges enabled?

2 Likes

Personally I want to be able to define an app’s soft and hard WU limits. Notify me if use more than X WU and stop billing at Y WU. I can live without much else but this is at least something to prevent situations like yours.

11 Likes

Ouch! :astonished:

What do you mean by “after the first credit”? Are you saying this is the second time it’s happened?

I was under the impression they were very understanding of such mistakes.

I guess one consolation is that with the recent (and coming) improvements to bulk data operations, there will be less of a need for recursive workflows.

1 Like

i made it so that i have field on my dev user that will +1 whenever a backend workflow completes, so i can see if im getting runaway workflows.

1 Like

Agreed. It’s indefensible to roll out the whole WU model without basic controls. And custom notifications (slack/WA/sms)

@mitchbaylis I’m pretty sure you’ll get a credit. Especially if “usage spiked” email alerts weren’t sent. Is the WU count semi valid? Either way, if you don’t get a credit, please let us know…

Does anyone know if status monitoring is tied to the WU email alerts as they’ve been spottier than usual today:

1 Like

I had overages enabled as last month we went 50k over our plan limits.

I’d left the overages enabled on (forgot to turn it off).

My first credit was right after I upgraded, I did a bulk update which triggered a bunch of things (before I cleaned up my app logic). It used a mere 100k but I had another event that used another 100k about a week later when I did another bulk update as I was cleaning up the logic and they refused that request stating “you only get 1 credit per user”.

I’m all for billing for workload units - it’s certainly made me a better bubble builder, but it’s the lack of controls and alerts that are the issue. The high pricing of the usage is less of a concern, I’m happy to pay for a premium for the ease of low code/bubble but I’m not happy to tiptoe through development and then get whacked with high costs when I make an inevitable mistake.

The whole pricing model seems to be setup to allow devs to fail and make costly mistakes. We will see what bubble comes back with.

During the cleanup phase I created a data thing stating the workflow name and some notes for each workflow so I could get a clear history of what was being triggered (before bubble added the workflow unit analytics - now you can just click on the high usage culprits). I still use that diy logging system when I debug workflow calculations as I find it’s so much clearer than the built in workflow history tool and I can log only what I care about rather than sifting through pages of meaningless logs.

4 Likes

I was able to find the issue.

I had a recursive workflow that relied on data existing. The data had been created correctly and the recursive workflow had ran correctly on the initialization (which is how I tested it). However the front end user had asked for the ability to delete data and I hadn’t accounted for that in the logic when I had built and tested it originally. So when they clicked a button which reran my creation logic the workflow ran but my search filters were no longer returning values and so it created data into infinity and the creation of the data triggered a bunch of other api workflows (only get triggered on creation).

So the system created 17k records in 12 hours and each record had a a string of workflows that triggered from it’s creation (no recursive issue here, just a multiplier effect from the 17k x expected usage)

For anyone interested, I’ve restructured the workflows to remove the infinity possibility so it’s impossible for me to get caught out in the future. Instead of using recursive logic to create multiple things that are the same up to a desired quantity, I now use a series of creation steps within the 1 workflow and I’ve added enough steps to account for the maximum desired quantity (10). This way if someone triggers the workflow it will never recurse upon itself.

Old logic:
search database for things create if count is less than desired count > rerun api workflow if count less than desired count

New logic:
search database for things create if count is less than desired count
search database for things create if count is less than desired count
search database for things create if count is less than desired count
search database for things create if count is less than desired count
search database for things create if count is less than desired count
etc

finish workflow after a fixed number of checks

This new method will use more workload units but at least I’ll never run into the nightmare 7 million units issue again.

I’ve actually been toying around with the idea of having a datatype specifically to track recursive workflows. Dunno if it will help you but here’s the logic.

When i run a recursive workflow it will create a new “Recursion Tracker” datatype. Fields include the expected/max number of loops and another number field to verify that there is a minimum number of expected datatype. I can add other fields to use as verification to start or stop something.

The logic is that it’s cheaper to send one Recursion Tracker datatype as a parameter for a WF to read and write from instead of having to do a search everytime.

1 Like

scary…

1 Like

I want @tylerboodman and @jared.gibb to fight it out over Supabase and Xano (but not on this thread because Mike will move it :joy:)

5 Likes

I’m about to nuke the entire supabase/xano discussion. You all can yell at me in DM, but please don’t divert threads… thanks.

8 Likes

Please

If anyone wants to debate this shit, I’m on Twitter. Find me there!

@Codelessconsult

6 Likes

Where’s discourses ideaboard to upvote the feature to extract a reply into its own topic with context? :wink:

1 Like

It is always a delight to read threads where drastic solutions like externalizing critical application tiers (such as the backend) are considered without quantified problem definition and forward thinking in terms of solution architecture, data architecture, change impacts, and maintenance costs, especially at nowadays’ developer rate.

To a man who only has a hammer , everything he encounters begins to look like a nail.

3 Likes

You mean this post or Bubble’s arbitrary WU pricing model :wink:

I :heart: this quote. There should be a law requiring all no-code apps to have this disclaimer in the same font style as the surgeon general’s warning on cigarettes.

1 Like

So today I had a boiler plate reply from bubble support. I doubt they even read my emails detailing what happened and how I had resolved it as their only acknowledgement was “thanks for the details”.

The boilerplate reply was the expected one - NO CREDIT WILL BE PROVIDED because it is a one time courtesy. Further, the courtesy only applies to the account owner - so basically 1 credit per app not one per user.

I’ve pushed back with another reply but I’m not holding my breathe here.

I accept responsibility for making a mistake and I’m happy to pay for 100k units or so (my daily max). BUT I fully expect the 7 million units to be refunded back because that is on bubble for not having a single safety feature in place to stop the auto charge billing into infinity. I mean, what if I went away for the weekend? Monday morning I’d come back to a charge of $5k.

What’s the most concerning part though is now that I’ve run over my plan limit I have to keep the auto charge on - which means that if I have something go wrong again I’ll be on the hook for yet another $1k or even more if I don’t catch it and rely on the bubble email alert which caught it after I did.

The whole pricing model just feels like a greedy cash grab. I am now actively looking for a replacement for bubble. I was a big supporter even through the new pricing change, but now they’ve really kicked me in the guts.

10 Likes

I… don’t know what to say.

:heart:

1 Like

If you left your lights on and came back a year later, wouldn’t you expect to pay a bill?

And yes. This situation is super shitty. They have the overage stop feature and they’ve graced you once. How many times should they do it for everybody in that case? I’m not defending them necessarily, cause I love the underdog small guy, but seems like they gave you a fair chance the first time around and you made similar mistakes again.

That said, it’s supes expensive and could probably be brought down a factor and still turn a profit.

This actually reminds me of my student debt situation. Why didn’t any stop me when I was taking more money than needed or before I dug myself into a pay back forever situation. There was no warning valve on that tank of misery and costliness.

4 Likes