Backend Workflows--Behaving abnormally

Hi there Bubblers, I have come across a strange issue today. To give some context, I built an app earlier this year its a double elimination tournament organizing app. It was working for creating huge tournaments but now its breaking even for small ones (P.S. no workflows were updated).

Here is the structure, user creates a tournament, assigns different divisions to it, it can be any number (I tested several times with up to 40 of these before and app worked fine).

So each division has matches created, according to the double elimination logic.

The workflows are setup like I divide the divisions for a tournament into two parts so for Tournament T , having say 4 divisions, I save it as 2-2 (for odd the latter gets one more)

say the divisions are D-1, D-2, D-3 and D-4, each having 5 participants in them, (meaning each division will have 15 matches [1 final, 1 secondary final, 7 in winner bracket and 6 in loser bracket])

a total of just 60 matches for T.

So I run 2 "workflow on a list", the list is divided divisions, meaning one runs for D-1 and D-4 and the other for D-2 and D-3.

I that workflow, I call a workflow which takes a single division and single division and passes that to another workflow which is responsible for the creation of matches for that specific division. Since the creation of matches for each bracket (winner, loser and final) is independent of each other I run workflows for them simultaneously, it saves time.

The matches are created in “recursive manner”, meaning the workflow calls it self, yes I did remember to put a condition to break recursive flow.

there is a delay of 30 sec for “workflow on a list”, to ensure no overwriting.
there is a delay of 3 seconds when the recursive workflow calls itself.

there are similar workflows like that of creation of matches for the propagation of matches , to ensure the double elimination format. It is just making changes to the already created matches, and yes it has been made sure that these are called only after all the matches for a tournament are created.

here is the main issue, I am tracking in percentages how much of creation and propagation has been done for each tournament. So at some point it gets stuck at 88% creation 0% propagation, or even 99.8% creation and 0% propagation. Also 100% creation and 89% propagation.

Please note again it was working just fine for all the use cases before, we even organized a few live tournaments on it and it never broke.

Now, we have a tournament in less than 12 hrs and it is not working

Here is what I found in my server logs

The app still works fine 2/3 times but this is too risky for a live event for us.

@emmanuel
All help is appreciated.

Thanks in advance

Do you reach max capacity?

This topic was automatically closed after 70 days. New replies are no longer allowed.