Warning to Newbies from a Newbie (1million WU explosion from API WF scheduled in the past)

Sharing my experience in hopes of sparing another Newbie the anguish.

Some background:
Tail end of first app underpinning a soon-to-open physical location. While testing our API WFs, i entered test cases with shortened reschedule API times (+hours:1 instead of +days:7). i’d read about how recursive API WFs looping through lists without sufficient end-conditions would run up massive WU, but since we don’t process any lists in backend WFs i wasn’t worried. What i didn’t know: the hazards of scheduled dates in the past because Bubble’s internal logic wasn’t described anywhere (i requested an update to the Manual and they’ve added the sentence: Note that if you schedule an API workflow in the past, it will trigger immediately.).

To make a week-long story short: Since WFs scheduled to run in the past are immediately triggered and dates are treated as date 12:00am, if an API WF is rescheduled in the backend with date +hours:1, then after 01:00 the WF will have a scheduled date in the past and will continuously reschedule itself (NOT every hour). The one field which i thought would update every hour ended up using 1 million WU in a couple hours…

So i applied for a WU waiver and now our site has been down for a week (honestly didn’t think the WU waiver review process would take this long); and our opening is postponed indefinitely because Bubble has been unable to tell me how long the ‘process’ will take (although i’ve just been informed that the request has --finally-- reached the ‘highest level of required approval’ so maybe sometime next week our site will be back up - i’ve worked in orgs of 30k-50k which felt leaner than this, sigh)…

Lessons Learned:

  • When rescheduling API WFs in the backend make sure none of the scheduled dates can Ever Ever Ever Be In The Past. Especially when testing using shortened times (<1day) and date (use Current date/time instead) !!!
  • If you’re not sure, ask. Surprisingly helpful experts on the forum, although some are more snarky than others : )
  • If you do make a mistake and run up a WU tab, you can write Support to request a one-time exceptional WU waiver. Maybe get used to the phrase ‘Workload requests must be handled and approved by many individuals across multiple levels of the company…Thank you for your patience.’
  • If you decide to buy extra WU (Workload Tier), pause/cancel your WU-intensive Scheduled WF, otherwise …
  • Am a bit torn on this last one. Am generally quite optimistic and used to think Bubble was The Best Thing Ever… but this past week of being stonewalled by The Big Bubble
    B-u-r-e-a-u-c-r-a-c-y and being repeatedly told how ‘extremely important’ my app is but that there are many individuals at multiple levels involved in a ‘thorough approval’ of my request…
    the doublespeak…has just changed something fundamental for me, like when you sense for the very first time that a relationship r-e-a-l-l-y isn’t healthy and you somehow know down deep that it’s not–it can’t be–a long-term thing…
    Anyway, it’s all Live & Learn
    Good luck on your journey, wherever it may lead

PS Submitted to Tier 2 Support (i was told to enter a suggestion on the Ideaboard, which I did…)

Let me try again:

  • Bubble has intransparent internal logic which immediately triggers WFs scheduled to run in the past (where is this logic described?)
  • Bubble also has logic that treats dates as date 12:00am

So if someone schedules a single API WF to run on a single date with a single task of updating a single field and then reschedule itself for date +hours:1 (instead of Current date/time +hours:1), then at 01:01 your scheduler immediately spins into a never-ending Twilight Zone loop of rescheduling itself, with no indication in the Scheduler that lots of workflows are running because it’s just one workflow running up a million WU in a couple of hours

My suggestion:
Either put guardrails around this or at the very least DESCRIBE IT CLEARLY IN THE MANUAL WITH WARNINGS ABOUT THE POTENTIAL COST OF MAKING THIS VERY SIMPLE MISTAKE!!!

i understand that more WU means more Bubble revenue but it would be the decent thing to do to be transparent to help the little guys avoid paying thousands of dollars for a simple little mistake

But maybe asking companies to do the decent thing is a stretch these days

4 Likes

Did you reach out to Bubble support to get the charges reversed? Bubble did state publicly on the forum in the past that they will be working with users who have these experiences to remove the charges.

reached out to support continually for a week and are waiting on Bubble to give us WU credits so our site will come back within our Growth + WU Tier plan (when a site exceeds this, Bubble takes it down and routes to a ‘this site has exceeded it’s plan’ page with a button to buy more WU…)

Have you tried enabling overages? That’s why your site is down unless Bubble has another secret shut off mechanism…

went the WU Tier route instead and wanted to follow this ‘process’ through to avoid any more surprises. we could of course keep paying more and more and hope that Bubble credits us everything in the end, but it’s all been a bit of a once-burned-twice-shy experience…

1 Like

@kunigunda Thanks for sharing. I think this is a very rare situation to encounter, but nevertheless has a very high impact to users like you. Even though there is an option to contact Bubble support, you still had to wait 1 week(and counting) for Bubble , delaying your opening launch.

I agree with you that guardrails need to be put up for scheduled workflows that end up in the past. Or to flag out workflows that end up running every minute(or periods less than an hour).

Jordan
@nocodejordan