"App Too Busy" While Running a Calculation in a List

Need help
1

I have a list of data and need to run a calculation on each item, then create a new thing with the calculated data. Currently, I’m encountering performance warnings and App too busy warning.

  1. When a user clicks a button, I schedule an API workflow on a list (contains unique Agents, approximately 900 items). However, I filter out only those Agents meeting specific conditions, reducing the list to 150 items.
  2. Then, it triggers another API workflow in the backend to again schedule an API workflow on a list (contains list of Payments, around 70,000 items). Once more, I filter this list with conditions, reducing it to approximately 7,000 items.
  3. Finally, another API workflow is triggered to create new entities. I expect 7,000 items (already calculated) to be created in my database, but an error occurs, resulting in only around 1,800 items being created.

My question is, how do I improve the performance and prevent my App to run into any performance error.

2

Are all filters in back-end? Generally App to busy errors indicate a poorly constructed query and/or a inefficient DB schema.

3

How fast do you need to process the items in the list?

You can use recursive workflows.

If you set a one second interval and did one list item at a time, it would take about 5 hours to get through everything.

The good part of recursive workflows is that you know when they are done, as you can use the last step to do whatever post processing. E.g.: last step of recursive workflow 1 can kick off recursive workflow 2.

They can still be interrupted, but they are more reliable for large lists. I use them all the time. Slow AF but flexible and reliable AF too.

@petter goes into detail with it.

You might want to check out his Ultimate Guide to Performance book.

4

He doesnt need recursive for 7k things and it will cost him a arm and a leg.

5

sorry for the late reply, first I filtered the data in front end then passed that data to my backend workflow

When button is click I run an API WF on a list

image
image859×788 57 KB

Do a search for agent contains 900 items then added a constraint to filter only data I need (150 items)

Then I’m passing a list of selectives (70,000 items) with a constraint (7,000 items)

image
image860×786 63.9 KB

In the backend wf

I scheduled another API WF in a list (I’m running a list that I filtered in the frontend so this is now 7,000 items)

image
image420×783 28.9 KB

Then this will trigger API WF that will create items in my database and perform calculations

image
image424×784 29.4 KB

6

Haven’t tried recursive wf yet, but I heard that it will cost a lot of WU.

7

@yamada.shoji99

7k iterations x 0.70 WUs/iteration for scheduling = 4,900 WUs

Yep, more expensive than I thought with the new cost model.

8

150 * 7000 = 1.05 million scheduled workflows. I’d be busy too. First, add intervals to the 150 (your agents). I’d give 5 seconds or so if it’s just a daily thing that’s not time sensitive.

And upgrade to the WU plan…

9

Hello, I didn’t know that it scheduled millions of workflows, because I thought it runs 150 workflows (agents) and then processes a single list of 7000 items.

10

No, you said yourself that in the backend workflow
you run Schedule API workflow on a list of 7000 items. That means scheduling 7,000 API workflows (one for each item)

11

That’s crazy, I need to make changes in my workflow right now. Do you know what’s the best way to deal with this? Using intervals is not what I want because I need the result as fast as possible.

12

I don’t know your use case, so you’ll need to share more.

13

I did this last week and saw 5-10x speed improvements on Schedule API Workflow on a list, with the default (blank) interval set. Definitely can recommend for that improvement alone!

1 Like
14

This topic was automatically closed after 70 days. New replies are no longer allowed.