I only have a personal plan and run an API workflow on a list of 100-5000 things. I tried a few things:
I ran 200 at once and it worked well.
I broke it up to 20, 20, 20, 20 (200 total) … and ran them 60 seconds apart and some of them failed.
So I created new endpoints (copy and paste) and ran a 100 of things on each within 30 seconds of each other on 6 different endpoints. 1 failed and 5 scheduled but then I ran into capacity issues. So I boosted the app with extra capacity last night. The app kept running really slow and at 100% capacity, most workflows kept failing. So I canceled everything in my scheduled workflows (refreshed the page a few times to make sure all are canceled), shutdown the computer and went to sleep. This morning I found out the app kept running workflows and remained at 100% capacity for next 2.5 hours. Some workflows failed others worked. The extra boost got the app down to 80% capacity.
I have seen some solutions on other forums and I will try to implement them. What concerns me is the unpredictability of a workflow failing / passing. Any way to predict this? Or what’s the best practice?
Second question is, how come the workflows kept on running? I have specifically focused on clearing the whole list before going to sleep. No other user could have triggered these workflows either.
Are there any tips, suggestions, videos, help articles on this topic that I have missed? Thank you.
@Martin If your app maxes server capacity usage over an extended period of time, your app will be rate limited which can lead to slowness or workflows failing. The best approach is to keep an eye on your logs. Separately, ‘operations timed out - app too busy’ can mean 2 things - either you are maxing capacity or your workflow ran beyond the 5 minutes limit. To prevent timeout, break large workflows into smaller batches or use custom events.
Regarding workflow scheduling exceeding 5 minutes. Is there a way to predict how long something will take to schedule? And if the app is slower, will the scheduling take longer time?
@Martin There is no workflow complete prediction tool but you can run tests and check your server logs to get a ballpark. If your app is rate-limited, everything from runmode to workflows can be slower. However, as long as you keep an eye on your usage charts and optimize workflows, you should not run into issues.
Hi @neerja, how would a custom event cause the workflow to use less capacity or take less time? I thought they are run within the workflow that calls them, so the workflow still takes as long.
@mishav Custom events can help if you have an original workflow with many steps. Triggering a custom event whose workflow has conditions ensures that only necessary actions are run. If the custom event workflow condition is not met, the actions within will not run. So it’s not that custom events are stopping the clock but that you are only running what is necessary.
Separately, if you have an action with a condition within the original workflow and that action contains a lot of searches, they will still be evaluated which can be data-intensive.
These are suggestions so @Martin should do what works for that app’s situation.
