With the current setup, it seems like the choices are:
a) Budget for overages and expect they will happen, even if you watch it like a hawk. So the actual cost of the app isn’t the normal monthly price, but some multiple of that.
b) Click the option so your app shuts down if you hit the limit, then buy some extra capacity to get it running again.
Unexpected overages may seem like a “bug” to users, but it may be a “feature” to Bubble. Based on the user posts on the pricing change, users seemed to prefer the old way of buying capacity, but Bubble said no.
Until this is not an issue anymore I think I will make a credit card just to use with Bubble and set a safe limit to it. Recommend everyone to do the same
When you accidentally leave your lights on for a weekend you don’t get a power bill that is 10x your normal bill… You get a 1-2% bump - a realistic expression of the cost to supply the power.
And to further exaggerate the point I left the “lights on” for less than 12 hours. If I’d left the lights on for a weekend I’d be looking at $5k auto charged - that’d be 37x my normal monthly charge for 2 days of usage. A house has fuses which blow if you go over the expected amp draw (expected usage per circuit).
As it stands $1k is about 7x my normal bill.
I take responsibility for the mistake. I’m posting it here as a warning to other users. Bubble is a no code platform which means a lot of things are assumed to be taken care of by bubble so we as builders do not have to think about all the little details.
Not having any fail safes is akin to running all the circuits in your house without any fuses and having to run out to the meter box to flick the main breaker anytime you suspect there is a problem. By which time the circuits may have already overheated, caught fire and the house is burning down…
We had a similar situation yesterday, where a last-minute update to fix something from the backend was pushed at night.
The next day, we had a webhook that triggers 5-8 other backend endpoints that schedule another 2 or 3 backend workflows that manipulate a lot of data. That specific webhook should have run once and it that started to schedule itself while triggering everything in each run.
I detected it pretty fast because our app started to hit the 100% capacity (we still in the old pricing), and everything started to work slow + I have some health statuses that started to indicate unhealthy system. When I looked at scheduled workflows, I had +1000 workflows scheduled already.
Time to debug, I didn’t want to just stop the workflow but find where it looped, so spent around 1:30h finding it as that specific endpoint is pretty complex with tons of data, conditions, and other schedules stuff.
Normally, our daily usage is 400k-500k/WU/day, but yesterday we did 2M because of this…
This is my plan for now too. One of my apps will be expecting a 20x increase in usage in the coming weeks after securing more contracts. I’ve made my calculations and I should not be going beyond Tier 2 monthly but nonetheless I think this is the safest bet for me so I can allow some overages.
I feel sorry for you on this. Bubble clearly hasn’t got all the safety guardrails and such in place and not having any flexibility due to this seems a lack of care towards users. This is why I wouldn’t advise anyone who has the legacy plans to upgrade until the last possible second and that is only if enough of these issues are sorted out. I don’t have confidence in these things being resolved by then either so keeping tabs on other solutions is also wise.
When I first had an issue with recursive workflows associated with data not being there it was because I had a setup to go through all of my database, 100 entries at a time…so my recursive setup required me to schedule the backend workflow until the number of entries (ie: my list parameter) was greater than or equal to 2, then I had a subsequent action to schedule the backend workflow when the number of entries was 1 and that was performing a search to find the next 100 entries and populating my list parameter with the values of the new search.
The problem I encountered, was that despite there not being any new entries found in the search, it still scheduled the backend workflow so went into an infinite loop doing the search for the next 100 entries.
My fix.
I put a terminate workflow action at the beginning of the backend workflow with a condition if my list parameter count is 0.
Based on the explanation of your New Logic with the multiple searches, I believe you could probably just implement the same as I had with the terminate workflow at the beginning of the backend workflow and save a lot of WUs.
I considered leaving the recursion but felt it was too high risk considering this app is very complex with the logic.
I decided to trade slightly higher workload usage for absolute assurance that the recursion event would never repeat again. I’d prefer to pay for 50k extra workload units every month and not have the risk of being charged an infinite amount if the filters fail and go into a loop. If bubble add some controls I may revisit the logic but for now I was able to set a maximum number of things to create (10) and then daisy chained the creation modules rather than recursing.
To save on workload usage I could do the search once at the start and then use the number value as the filter in the create modules ie search for things minus expected thing quantity - if result is great than 1,2,3,4, etc then create. I may add that, but the search is not that heavy, the workload usage issue was in the other api workflows that were triggered from the creation event.
No worries. In development it is a to each his own approach. If later in time you decide you don’t want to spend an unnecessary 50K WUs a month (and more than that if app scales) you could revisit this thread and find the solution.
I feel like the reply you’ve got from Bubble support is not aligned with their C-level executives values?
Our goal is to win because you win, not because of accidents. We understand that in the process of exploring and developing on Bubble, accidents happen. If you make a mistake, please reach out to our team at support@bubble.io. We’ll deal with these instances on a case-by-case basis.
[source]
Until there is no option to hard limit at least your apps WU usage (not even speaking about setting limits for separate workflows, for example) AND get real-time notifications (email and sms) about your WU usage exceeding the threshold - devs/business should not be punished for mistakes at all.
That analogy doesn’t hold up at all.
How is 12 hours like a year?In what scenario do electricity companies charge you an artificially inflated price and not the actual cost of electricity?
The better metaphor is that you just spent many hours and lots of $ installing central air conditioning in your house. The day it’s installed, you receive a notice from the electric company that, for this specific AC model only (which has custom vents that aren’t compatible with any other model), prices are going to be 10x higher. Not only that but if ur AC is on for 12 hours straight you may, if you don’t check on the meter to see there’s a warning, be charged an extra $1,000 fine.
IF you could set hard and soft overages, provide proper notifications (customizable and that actually worked), and partition certain WFs then you would have a case.
similar mistake again? as in the mistake of being on Bubble? I’m sure the issue was a different one than the original one, and considering how fast things are shipped and how uncertain the changes are (can ANYONE tell me how many items a API workflow can do without timing out?!) there should be some leeway.
Finally, if that’s how it’s gonna be, let it be give and take on both sides. Every time Bubble causes major issues with a website solely because of shipping something without proper QA, let them compensate app owners for lost business due to their negligence / cost cutting. Doesn’t even have to be in cash, Bubble credits will do!
I’ve tried to find out anything about “first mistake/chance/opportunity”. But it either I was looking for something wrong or there is no such thing stated publicly. Bubble doesn’t know about loops (thing they are “are considering for the future” as I remember from BubbleCon), but wanna punish devs for workarounds they proposed to use?
That’s not definitely a win-win. That’s a huge L for Bubble despite of getting quick and easy $1000 from @mitchbaylis…
As a conclusion from my side - what is Bubble really missing is DevRel. Where is the Developer Advocate inside Bubble team that can firmly represent the interests of developers and be taken seriously by Bubble internally? I understand that $1000 is $1000, but is it worth it?
For what it’s worth, I made sure this thread is on the radar of Bubble’s community director.
I feel bad for OP, I really do. But, since it’s out there, this is at least the third time this has happened to OP, and Bubble issued a credit the first time. It’s also unfortunate that OP was nice enough to warn others to be very careful with the overages feature but then forgot to turn it off for their own app.
Sadly, I’m sure this kind of thing is going to continue to happen until better safeguards are put in place (the soft and hard WU limits is a great idea, by the way). Hell, my dumb ass recently pulled a bonehead move that is going to cost me a few hundred bucks, and I’m just going to eat it because the mistake was somewhat easily avoidable, and I put that on me. $1,000 is a different story, of course, but I don’t want to believe the community thinks Bubble is greedily and/or happily trying to take a grand from OP just because they can. Call me naïve (or any of the other colorful things I have been called out here), but that simply isn’t true.
I take responsibility for the mistake, it is on me to check things. Where I take issue is that the cost of the mistake is infinite - the only safe guard would be my credit card limit. But even then bubble may still “keep the lights on” and then hold the app hostage until the bill is paid.
It’s only a matter of time before a live app with thousands of end users pushes a release with an iteration bug in it and blows their account usage up like I did to mine - but on a much grander scale.
I’ve had several iteration mistakes in make.com and they have a reasonable system in place to prevent users from costly mistakes.
they email you when you run out of credits and you can turn on auto recharge
auto recharge is a predetermined value not an infinite open value
they email you EACH TIME you auto recharge so your inbox basically gets spammed if there is an iteration issue
they auto turn off scenarios that have large spikes in usage and leave the other scenarios on (each time you run out of credits)
you can buy additional packages of credits manually at any time so you don’t need to leave autocharge turned on and open ended
they have internal rate limits established and a max execution time per scenario (40 minutes)
APIs that you connect to have built in rate limits which will cause a time out and turn the scenario OFF in make.com
My main point that I’m raising with this thread is that there are absolutely no built in safety features or fallbacks for the auto recharge and the new pricing model and that it’s extremely risky for anyone building in bubble. One slight oversight by anyone in development could end up costing thousands in a few hours/days unless someone is constantly monitoring the usage reports.
For instance in my case I had already thoroughly tested the iteration workflows and they were working correctly - they were not the fault. But I added a function for the user to delete something in the front end and that thing was linked to the iteration workflow (which I built over a year ago) and by allowing the user to delete something I broke the iteration logic and caused the infinite loop. I tested the delete function, but I didn’t retest the iteration logic and when the user clicked the button (in a completely different part of the app) to recheck the iteration the infinite loop was created. Talk about a butterfly effect!
Hi Mitch - Laura here from Bubble (sorry for the delay in responding - I usually look for threads like this but i missed it!). I really appreciate you taking the time to explain your frustrations here (I agree it’s frustrating and we can do better here). We do have a warning that gets sent out if you have spiking workload (see here: Notifications for spiking workload unit consumption) . But, as you noted, it came too late. Our team is looking into this right now as obviously a warning that comes after you’ve already fixed the issue isn’t useful. It is supposed to go out real time both in the editor and via email. That doesn’t solve the issue of “going away for the weekend” but it should help mitigate. There are other ideas we’re considering (being able to set an overage cap, burst protection, warnings on recursive workflows without stop conditions… lots of things we could do here, but I’m not sure yet what the right solution is). I’m also weighing investment here against other areas we could work on to make things better (better visibility into and visualizations of workload consumption for example). Feel free to drop me a DM to share any thoughts you have - always happy to connect with our customers to learn more.
