Backend workflows - infinite loops & scheduling

calum.rr.scott · 2025-02-12T17:58:22.281Z

Hi everyone, being driven slowly insane by this hoping you can help.

I need to send data to GPT and run a process, I need to poll the process to see if it has finished and if not then wait 10s and poll again. When the process is complete I can fetch some data and move on.

I’m trying to do this using a loop in a backend workflow but I keep getting huge spikes in scheduled flows, or no data even when including some loop control elements.

Screenshotted the workflows:

Initiates a new chat thread via API by posting a message to the thread, then I run the thread. I create a “Loop control” record with the “thread_id” and “run_id” to store whether the run is still processing or not. Invoke the second workflow
Checks whether the run is still processing, if so then it should schedule itself to check again in 10s. I increase the retry count by 1 and only reschedule if count < 10. I also only reschedule if “loop control” says run is still processing. If run is complete then loop control sets “is_running” to NO and the loop can end and I fetch the messages from the thread.

For some reason without the loop control record and just using the retry_count I schedule 1000 workflows even if only 10 complete. Either I’m getting a load of workflows shceduling or right now no data back. What am I doing wrong?

Screenshot 2025-02-12 at 17.48.24716×587 42 KB

Screenshot 2025-02-12 at 17.48.58715×840 66.1 KB

romanmg · 2025-02-12T18:42:05.213Z

What’s your config for step 6 in your first screenshot when (to run the 2nd workflow)? I wonder if you’re sending empty values.

Some troubleshooting tips:

Go to your logs > scheduler, and see if you have multiple runs going right now from your testing. Cancel them so you can start fresh.
Check things out in the server logs. It tells you whether conditions pass or not with the actual values being used.
Restrict your recursive logic so it only runs up to 3 times rather than 10. It’ll just make your troubleshooting in the server logs a little easier.
Do you get data when you just run it once without any recursive logic? You could schedule the data fetching after a few minutes (when you know for sure it should be done) just to at least confirm the data retrieval part.

Gaby
Coaching No Code Apps

calum.rr.scott · 2025-02-12T19:48:59.700Z

Thanks for the quick response! I stripped out all the recursive elements and data comes through ok for one workflow and when re-run does not populate data for other workflows even though data seems to be fetched ok.

I’ll follow your advice and do some more testing to solve the data issue first then come back when ready to get the looping back!

calum.rr.scott · 2025-02-14T07:43:54.379Z

Hi @romanmg so I stripped everything back to basics and found I was not pulling data correctly through the workflows. I solved this and decided to store the thread_id, run_id and run_status on my flow record. Then I reference this value in the loop in the workflow as well as a retry counter. This has solved the issues and now the loop works great! Thanks for your input

romanmg · 2025-02-14T14:53:46.824Z

Fantastic – nice work!