Workflow Runs usage "timed out"

boston85719 · 2020-03-21T17:29:35.087Z

I am hoping somebody might be able to explain what is happening generally with my workflow runs.

I am not familiar with the analytics located in the logs section so really have not much insight into their meanings. I do have some idea that my workflow runs indicate the number of workflows perform over a period of time.

In this image I am assuming the information in the red square for the point in time of the red circle indicates that 55 workflows were run during the 15 minute interval beginning at 22:00

chart3850×479 39.7 KB

And in this image the information shows that 26 workflows were run in the 15 minute interval of 1:30

chart1

And in this image the information shows that 14 workflows were run in the 15 minute interval of 16:45

chart2728×445 30 KB

I have a few questions about this as I have been allowing a recursive backend workflow run with no other activity on the app for the purposes of processing some data. The backend workflow is set up as so

Screen Shot 2020-03-22 at 12.38.36 AM

Since I have a 15 second delay for the triggering of the workflow shouldn’t I expect to have about 60 workflows run every 15 minutes (15 minutes * 60 seconds = 900 seconds / 15 second intervals = 60 workflows triggered)?
What factors might contribute to the lower number of workflows per 15 minute interval such as 26 or in one case only 14?
What would be a reason that it seems like during this period of time

chart 4

it seems like either none or only 14 workflows were triggered during the period of time starting at 1:30 and ending at 16:45. The reason it seems like there may have been none triggered is that I am unable to put a marker anywhere along the straight downward sloping line to see a number of workflows for any 15 minute interval except at the beginning (1:30) and the ending (16:45).

During this period of time there would have been no activity whatsoever on the app. The only thing that I can think of is that this is the period of time that I didn’t have my preview page open which has the button to trigger the backend workflow. I couldn’t imagine how that could actually play a role as backend workflows shouldn’t have anything to do with it.

The only reason I think it may have something to do with it is that when I closed the page late last night I had processed 2332 entries after first triggering the backend workflow and when I reopened the page to check the count I had only process 2494, a number that was quite surprising as I closed the page at around 1:30AM my time and reopened it around 4:45PM my time.

boston85719 · 2020-03-22T08:30:14.850Z

So I did a little test last night when I closed the preview page that has the button used to trigger the backend workflow (which I have not pressed in over 48 hours) and the count of the number of data entries that have been processed with the backend workflow.

When I closed the page at 1:05AM I had a total of 3389 items processed.

When I opened the page at 1:05PM I had a total of 3564 items processed.

So in a 12 hour period I only had 175 items processed for a rate of about 0.5 per minute.

chart1.1

chart2.1851×454 43.2 KB

Then after I opened the page and checked 30 minutes later I had 3618 items processed for a rate of 1.8 items per minute.

chart3.1847×447 41.7 KB

What I don’t understand is why I am not seeing the rate expected based on a 15 second interval between triggering the recursive backend workflow (expected rate would be 4 items per minute).

And I still don’t understand why it is that when the preview page is not open I get such a drastically reduced rate represented by the flat downward trend line

chart4.1

Anybody have any ideas why this might be occuring?

@eve

SerPounce · 2020-04-11T17:54:22.341Z

@boston85719, did you ever get an answer or fix for this issue?
I believe I have something similar going on.

My situation is as follows:
When a user signs up to my app they can select 2 teams from a variety of sports. I then have a BEWF create their custom list of games based on their selections. If the user selects 2 teams in leagues with just a few games (less than 30) then the BEWF works fine. But if the user selects teams with 100’s of games, then the BEWF doesn’t complete.

boston85719 · 2020-04-11T19:11:21.259Z

No, unfortunately I didn’t get an answer on this. Not really sure who from bubble I should tag to get a response or if I should send an email to support.

johnny · 2021-03-17T18:53:51.613Z

Hey @boston85719,

If you don’t mind me asking, what action was your recursive workflow running? Was it trying to make changes to a type with tons of records or something?

boston85719 · 2021-03-17T19:48:43.989Z

it was making changes to a single entry of which there were around 5K to process…it was a recursive setup, so one item at a time before moving onto the next…had to process about 7 pieces of data for the purposes of adding a new data type entry to match the processed data, being a date.

johnny · 2021-03-17T20:03:28.792Z

Ah interesting, I’m chatting with @eve via email and I personally was using recursive workflows on over 6K users to send Drip Campaigns, and

“Essentially, you’re looking to set up a workflow that runs through a very specific set of data, with narrow constraints. The workflow is expected to run one item at a time through a list, until no more objects match its run conditions and it stops. It’s very important to have a specified list in mind with a specific end condition, otherwise you wind up with an infinite loop that eats up your capacity.”

Have you reached out to see what they thought of your issue?

NigelG · 2021-03-18T08:28:58.077Z

This is a real worry for me too.

In the past I have seen slowdowns due to workflows tripping over each other when scheduling “on a list” at an interval.

I have also seen the dreaded “workflow consistency” issue that disappeared from the roadmap where you schedule 1000 things to be created and added to a list and 1000 get created but the list is 999 (or less).

But if this is single-threaded, that should not happen, right?

The lack of predictability is a real worry here.

@allenyang this is what I was mentioning recently.

@peterj might be the one to ask, as there are a few of us pushing at this boundary now.

allenyang · 2021-03-18T12:49:22.956Z

We believe we stamped out one source of “workflow inconsistency” a few weeks ago, so if you’ve seen it more recently than that, please file it as a bug with as much detail as you have. Thanks!

NigelG · 2021-03-18T12:53:42.637Z

allenyang:

We believe we stamped out one source of “workflow inconsistency”

great news. Will set up some testing … I know how to recreate it

w.fly · 2021-03-18T14:01:10.841Z

My experience the past few days, Schedule API Workflow on List has actually been really reliable - I ran it on a few hundred database items and for the first time I can ever remember, it had 100% data reliability. Granted, it was a very straightforward API Workflow, literally just updated a record and added a record… I wonder if I’d still get 100% data reliability on a chunky API workflow like I used to do in the past. So far though, haven’t run into anything major wrong.

boston85719 · 2021-03-18T14:13:03.516Z

I never did reach out via an email, as far as I can recall. I just tagged Eve on the post linked and never got a reply.

My issue was not running on a list, I was running a recursive workflow with a specified list in mind and a specific end condition, so never did get an explanation on why the slow down. It was very inconsistent too.

Luckily I have not needed to do this recently. The last time I did it, what should have taken 6 or 7 hours took me 3 days.

NigelG · 2021-03-24T10:01:50.832Z

w.fly:

I ran it on a few hundred database items and for the first time I can ever remember, it had 100% data reliability.

That is great news. I wonder what the “limit” on lists like this is. We are up at the 10,000 level so think recursive is the way to go.

Matthew McGowan:

My issue was not running on a list, I was running a recursive workflow with a specified list in mind and a specific end condition, so never did get an explanation on why the slow down. It was very inconsistent too.

Have not experienced the “slow down” yet. What we did get was failures on Hobby plans.

See the other thread. I am really hoping that this is some sort of Bubble hard limit (so at least it is purposeful) rather than something squirrely.