Hey everyone! Hoping the community can help me rule out a major issue I found. A bug got reported by a user on a live app I have that runs a workflow on a list.
The lists usually contain between 100 and 500 items.
The API changes about a data field on each item in the list. For example, let’s say the API runs on a list of Fruit in the user’s account.
It goes through each Fruit, and changes “is tropical fruit” from “no” to “yes” (yes/no data type).
After the user reported the bug, I looked into the DB, and found that about 30-40% of the items in the list were unchanged. Since the rest worked perfectly, it doesn’t seem like it’s a bug, it’s like the API just didn’t affect certain items.
This was another breakdown by a Bubble user, who essentially said workflows on lists are more unreliable (but faster), while recursive workflows work 100% of the time (but are slower). Incomplete Schedule Workflow on a List
Also read that @romanmg recommended someone do a recursive workflow to fix this problem.
I’m willing to upgrade to a higher paid plan (I’m on personal legacy right now) but before I do that, I wanted to see if you all think it’s a core issue with the API workflow on list. If I need to switch to recursive workflows, I’ll take the time to do that.
I have a similar workflow that checks for changes in the status of an API list of events, then both updates existing listings and imports new listings. My list is about 3200 records growing by about 100 per/day. I ran into a similar problem. My fix was to increase the length of time of my interval to around 20 seconds. The time of day seems to affect the performance. If I run this at 4 pm in the afternoon my capacity is at 100% and I seem to miss updates, I just ran it and now and had no problems.
One of the things I do make sure nothing is missed is I created an admin area that lists a log of all changes by date, time and record updated. This way I can see if there are any problems. I also made my workflow recursive so if it does miss a few records, it will get it the next time it runs.
My 3200 records take about an hour to run at 20 seconds per.
Thanks so much for your help. To clarify, do you still run “schedule workflow on list”, or have you transitioned everything over to “recursive”?
Just wondering if your suggestion was to stay with “workflow on list” but add some time between, or whether I should switch my flow entirely to recursive, AND add some time between calls.
Good to know it’s a bigger issue. I figured it wasn’t a bug, since it worked perfectly for 50+% of records.
How do you get away with “schedule api workflow” on a list of things? My flow is a one-time one (happens once, on scheduled date, when the user chooses).
For example, let’s say each user has a list of “Houses”. They want to press a button and update the “Price” (field) for each “House” (thing) in the list. Do you have to do a “search for” and always do the :first item that hasn’t been updated?
Now, to get it to be “recursive” (in other words, I want it to move to the next item in the list), I have to remove the first item in the list and reschedule. I do that as follows.
I call the same endpoint - “Create New Loans”, I set it for 1 second in the future (honestly no reason, just like adding that second - In your case, if there are 100+ houses to run, you could probably not add that second.). For the list to process, it is “list to process MINUS the first item” since this loan was just created.
Note that there is an ONLY WHEN - You don’t want this to loop if there isn’t anything else to process - So if when I remove the first item in the list and it is 0, do NOT call “Create New Loans” again. (P.S. If you do want to double check that you don’t run an ever looping function, use the “Scheduler Tab” in Logs. Remember to click “Show”.)
Note that the list I am providing the Backend Workflow is a repeating group that has all of my fake loans in it. This can very easily be a “Do a Search For Loans that Need Processing”.
I’m using ‘workflow on list’ for most things running on daily basis, recursive for my checksums, which are more manual process I do in the admin area I built.
Thank you so much! There should be a way to “buy someone a coffee” on this forum.
I followed your steps, and was able to basically migrate my flow FROM “run workflow on list” to this “recursive workflow”. It worked perfectly.
More than that, I learned about “minus last item” which I didn’t know was even a thing. Thanks again for taking the time to share so thoroughly. You saved me hours! Also, I’m going to migrate all my other “workflow on lists” to this recursive flow. Seems much more sound.
Super. I have now started pre-empting that the list will not run completely - miss random items in the middle and so I usually run the workflow several times of the residual list till it completes.
@josh would the schedule API call on a list become more reliable on an upgraded bubble plan? Currently on personal plan.
Have you checked your capacity logs? This will tell you if this is a capacity issue (which it almost certainly is). How long is your second interval between repeating workflow steps? Whenever I have run into capacity issues, increasing to 2 sec interval fixes the issue.
Thank you for such a detailed walk through! I am going to try to set this up for our daily updates. Question…
If I’m not wanting to create a new thing (house/loan), how would I specify the path for each list when I do a look up against my API. For instance (sticking with the house example), if I needed to pull the latest estimate for each house via an API how would I link this “(path) symbol” to the address for each house so my app cycles through each address for the estimate first then makes changes to the list by adding that new estimate to the list?
Late update for anyone still going through this. I finally paid for a consult with CoPilot (a bubble dev part of this community) and learned a ton. Here are some takeaways, hopefully I’m remembering them right:
If you use workflow on list, it CAN be effective, but you need to allot enough time (for me, this was like 10 seconds) in between each workflow. This wasn’t sustainable. If the app gets too “busy” and can’t catch up, you’ll just miss items and never know which ones you missed.
So I moved to recursive, or at least, what some people call recursive. This is: “do this action until this condition is met”. The problem with this is - each action is a link in a chain. If the flow works for 100 items, and then on item 101 there’s something wrong (i.e. it goes to send an email, but the email address isn’t formatted properly), it “breaks” the chain and won’t work. So it will have done 100 items, break on 101, and skip the remaining. This happened to me a few times.
Although I DID get it working with recursive… I felt like it was really unsustainable that if any process had an issue it would break the chain. So I paid for some custom code fixes from CoPilot. This basically outsourced all the heavy lifting that Bubble isn’t great at, to Javascript.
The result was it ended up being WAY faster (literally went from 30 minutes to under 5 seconds for 1,000 items). It’s also more reliable, because it’s processing each record individually. So if it can’t send 1 email or change 1 thing, it doesn’t break the chain for all the rest.
Sorry for the halfbaked writeup, but I wanted to get SOMETHING out before I forgot. This was the worst experience I’ve ever had on Bubble. It took a month of me trying to do it myself with lots of bugs and complaints from a customer, and in the end I was really happy with CoPilot. Totally unsolicited recommendation. I try to do almost everything myself, but they helped me learn a ton about building in a better more sustainable way, and I found there are some things code is just better for.
TL;DR - if you keep asking questions, my answer will be: I’m not sure why yours is breaking. This was a nightmare.
For me, workflow on a list didn’t work, it just never processed more than 60-80% of the records. It seemed like the app just hit capacity, couldn’t do a few, and would move on to the next. I couldn’t get it to work.
Recursive was unsustainable because if your flow has trouble processing 1 record, it will cease running the recursive workflow on all others that remain.
Custom code was faster, more reliable, and doesn’t break the chain. I don’t know JS, so I hired someone.
@jeremiah - I just came across your very helpful post, about 30 seconds after creating a similar one myself.
We’re launching an app that has a number of dependent APIs and recursive workflows (e.g., to send an email to each person on a long list), and having a process ‘break’ and stop has been a fear of ours - particularly as we wouldn’t be aware of a stoppage. Our stress testing has worked out well so far, but it’s nice to know there’s an alternative in having someone code these critical steps, should an issue occur.
How in the world did you get any kind of response from @Copilot? That’s the most amazing thing. I need the same functionality. How much did they charge you for the final solution?
Perhaps someone will have some better feedback.