My first run-away loop -- Almost 200k units in a span of 30 minutes.. My story

[w.fly]

Well, it finally happened… TL:Dr; Be careful with “Schedule API Workflow” and running a large list…

---

Some context: We have a basic ticket portal. It has around 10,000 tickets. I had to go update the description for these 10,000 tickets.

Now-a-days, we all know that “Schedule API Workflow” is the way to go to loop.

So, I took those 10,000 tickets, put it in a list in the Schedule API Workflow.. And for each ticket, update the description, then remove that ticket from the list.. Rinse and repeat.

Well, a powerful powerful lesson was learned… That 10,000 tickets I placed as a list was costing 200+ WU units EACH LOOP… Even though I was only “Make Change to” on 1 ticket each time.. Yikes.

So, here’s the best part… For Schedule API on a Workflow, it averts the Scheduler. Traditionally, if I had a run-away, I’d just go here.. And cancel.

No dice. The workflows were running quick - And didn’t hit the Scheduler..

So I had to hit Pause Tasks… Which finally shut it down.

---

So when the dust settled, I learned some powerful lessons.

1. Be wary of passing in large lists to recursively run on.. even if you’re just doing one change to one item on a list at a time. That large list you pass into the API workflow apparently sucks a TON of Workflow Units every time you run that Schedule API Workflow.
2. Schedule API Workflow on a List is no longer a black sheep to me.. Because atleast you can shut it down if it runs foul.
3. Once the development credits run out, your production credits get hit. Thank GOODNESS I had “Disable Overages”.
4. I restructured the loop.. I placed a database field for bSync.. Each loop didn’t pass in all of the tickets, instead, it did a fresh query for tickets that were bSync = no. I went from 200+ WU units per run… To ~2 WU. Same job.. Just different method.

At the end of the day, let’s be real.. It cost me $30. Does it suck? Yeah. But $30 is a pretty cheap lesson. And as much as I’m not a fan of the WU concept, I can’t complain because frankly, my software dev company I own’s payroll is $35k per month.. If an employee messes up, I’m out $200-$300. I do save money using Bubble compared to traditional development.

---

Hope y’all enjoyed and hope y’all have a better day than me!

[Jici]

@w.fly

Schedule API Workflow on a List is no longer a black sheep to me

This will also save you a lot of WU and run faster VS loop (but you could have race condition and this may be your case from what I read)

[w.fly]

Yeah, back in ye’ old days, I used Schedule API Workflow on a List primarily.. It then went through a spell of having data inconsistency.. However, from some initial testing, it’s pretty solid now.

It’s quite a bit easier to setup too.

[akamarski]

Why not create a BE wf and run a bulk action for a custom view from the database view itself? Would have costed like 15k

[w.fly]

Didn’t consider that – Sounds like it is another solid option.
@akamarski

[sudsy]

Hey Riker , bummer about the incident, but glad you escaped relatively unscathed and gleaned some useful insights.

@w.fly

For Schedule API on a Workflow, it averts the Scheduler.

They’re definitely being scheduled; it’s just that they’re executing as soon as they get scheduled. It would be mostly chance to see one in the list of scheduled WFs. If you repeatedly clicked the “Show” button like a monkey, you’d likely catch one every now and then. (Yes, I’ve done it. And yes, I’m a monkey.)

@w.fly

That 10,000 tickets I placed as a list was costing 200+ WU units EACH LOOP

While searches themselves are 0.3 WU, each thing returned from the DB costs 0.015 WU. For 10K items, that’s 150 WU just for retrieval in the first iteration. And it sounds like the list was decremented by one with each iteration; but with 10K iterations, that’s a lot of WU. Add to that 0.000003 for each character retrieved, 0.5 for each thing modified, and the overhead of repeated scheduling, and it adds up.

@Jici

This [SAWOL] will also save you a lot of WU and run faster VS loop

@Jici is spot on, especially given the recent enhancements introduced just over a year ago.

In fact, I recently had to do a data update similar to yours. I needed to modify two fields in each of exactly 10K items, one of which was a reference to an image file that was uploaded at the same time. I used SAWOL, and the total WU cost was somewhere around 15K (which is what I expected having first tested on just 10 items and extrapolated). And if memory serves, it took less than 20 minutes (to update 10K records and upload 10K image files to Bubble storage).

TL;DR  The SAWOL improvements Bubble has made seem quite robust and are very performant. Generally speaking, they should be the go-to, as long as you’re mindful of the considerations noted in the docs for parallel vs sequential processing.

Or in perhaps more familiar terminology… With the new and improved SAWOL, it’s like Bubble replaced the dilithium crystals and rerouted the plasma conduits. It does warp 8.7 now.

[mghatiya]

@akamarski

Why not create a BE wf and run a bulk action for a custom view from the database view itself?

What is this? Can you shed more light on how it works?

[akamarski]

When I want to change data in bulk, I make a BE WF that does whatever I want to do and make it accept the data type I want to modify. In the workflows itself i change the data. This BE WF has no button trigger.

Publish to live.

Create a view of the datatype you want to modify and add filters id necessary. Switch to Live mode.

Go to the view, select any row and do “Bulk”, then select the BE WF. It asks whether to run the workflow for all items in view or the selected one and confirm.

[mghatiya]

Ah right right. I use it quite often myself.

I had not got that you were talking about this one.