I found a way to run API Workflows sequentially

code-escapee · 2024-11-17T16:46:28.699Z

Hi it seems the process you described would be helpful for race conditions but for sequential API workflows wouldn’t using custom events suffice?

antolin.lacaton · 2024-11-17T19:38:58.664Z

You’re absolutely right that Custom Events can be helpful for creating sequential workflows in some cases, particularly when everything operates within a single workflow. However, the “Filler Bucket” method is specifically designed for scenarios where backend workflows must operate across multiple steps and datasets, especially when dealing with:

Large lists or external API calls: Custom Events don’t support iterating through large lists of items sequentially without triggering everything at once. My method ensures that each item is fully processed before the next starts, avoiding race conditions or partial updates.
Cross-trigger dependencies: Custom Events are limited to the scope of a single workflow and can’t easily handle scenarios where workflows depend on triggers or data changes from other workflows.
Dynamic lists and statuses: My method allows dynamic tracking of each step’s progress through the BucketFiller and BucketManager system, which is crucial when processing lists where the length or complexity varies.

In essence, while Custom Events are great for linear workflows within a single execution, the “Filler Bucket” method is better suited for complex, multi-step workflows with asynchronous dependencies, large data operations, or when strict sequential processing is required across multiple backend workflows.

This is just my opinion, but in my case and my uses (calling microsoft azure API, sending sms/mail to the customer, creating/modifying data, etc…) this method has proved to be the most effective.

code-escapee · 2024-11-17T20:25:35.496Z

Ok but your list can easily be dealt with custom events just nested within a API WF so that it can handle huges lists 9r be triggered from DB triggers

antolin.lacaton · 2024-11-17T20:43:17.677Z

I completely understand your point, but in my case, the number of steps I need to execute varies depending on specific conditions, such as users, dates, or other factors.

To give you an example, my application is a cost-tracking software for restaurant owners. Users import invoices, create recipes from the items in those invoices, and for each new invoice, the system highlights cost changes and their impact on recipe margins. This involves managing data for invoices, suppliers, items, unique items (grouped lists of items), ingredients, and recipes.

The complexity increases as recipes can also serve as ingredients (sub-recipes for other recipes), requiring additional looping. Not all recipes are sellable, so margins are calculated only for the sellable ones, adding another layer of logic.

Additionally, since cost variations are tied to the invoice date, I need to check if a cost is already recorded for the same date or a later date to avoid duplicates or errors. This requires dynamic checks during each step to ensure data accuracy and integrity.

Given these complexities, I must ensure that one step fully completes before the next begins. I’ve tried implementing this with Custom Events nested within workflows, but I couldn’t achieve the flexibility or control I needed.

That said, I’m no expert, and I might be missing something. If you could share an example of how this logic could work with Custom Events, I’d genuinely appreciate it, it might simplify my workflows significantly !!

boston85719 · 2024-11-18T04:40:32.044Z

This method costs way too much WUs, is too cumbersome to setup and maintain. Would be better to use custom events and bulk updates via API connector. I highly recommend asking ChatGPT to give you a better approach than this one it gave you.

betteredbritain · 2024-11-18T06:44:15.616Z

Not sure if its just me but this post we are all responding looks ai generated, so I would take it with a pinch of salt. WU on this flow would be off the charts and the flow would take at least a couple of minutes to complete depending of course on the number of items on the list.

antolin.lacaton · 2024-11-18T09:14:41.690Z

boston85719:

This method costs way too much WUs, is too cumbersome to setup and maintain. Would be better to use custom events and bulk updates via API connector. I highly recommend asking ChatGPT to give you a better approach than this one it gave you.

You’re not the first to tell me, but I haven’t managed to do what I wanted with custom eventts. Do you have any concrete use cases for multi-step logic that work for you?

antolin.lacaton · 2024-11-18T09:18:12.301Z

betteredbritain:

Not sure if its just me but this post we are all responding looks ai generated, so I would take it with a pinch of salt. WU on this flow would be off the charts and the flow would take at least a couple of minutes to complete depending of course on the number of items on the list.

hahaha it’s true that I used GPT to clean up the method so that it could be understood by as many people as possible, since I find it quite complex to explain!

In terms of WU, I don’t see much difference compared to my old system where I used to play with the “Current date time +second” before launching the next step, but I was convinced that it was more reliable in the long term.

Do you have any other ideas for achieving the same result without consuming too much WU?

antolin.lacaton · 2024-11-18T09:26:54.521Z

I don’t know if it makes any difference, but my system has to work entirely on the backend so that my user doesn’t have to be logged in for the logic to be triggered!

antolin.lacaton · 2024-11-18T09:30:10.904Z

I don’t know if it makes any difference, but my system has to work entirely on the backend so that my user doesn’t have to be logged in for the logic to be triggered!

That’s one of the problems I had with custom events, I couldn’t store the results in a custom state since the user isn’t logged in.
What’s more, the next step takes into account the entire list from the previous step before launching.

Example: If I update a recipe’s ingredient list, I need to take all the updated ingredients to update the recipe (to add up all the costs and define the new margin).

I’m sure this isn’t the most efficient way, but if you’ve got something better, I’d love to know it!!

georgecollier · 2024-11-18T09:49:34.387Z

antolin.lacaton:

Example: If I update a recipe’s ingredient list, I need to take all the updated ingredients to update the recipe (to add up all the costs and define the new margin).

So just have a trigger when an ingredient is changed, ane Make changes to the Recipe with Do a search for Ingredients:each item’s cost:sum.

Don’t overengineer it!

antolin.lacaton · 2024-11-18T09:57:17.421Z

georgecollier:

So just have a trigger when an ingredient is changed, ane Make changes to the Recipe with Do a search for Ingredients:each item’s cost:sum.

Don’t overengineer it!

I’ve already tried to make it simpler, but bubble goes too fast and some results are overwritten by other operations, making the whole thing unreliable!

My software allows you to follow the evolution of cost variations over time, so each ingredient contains a list of “ingredient history” with a price associated with a date. The same applies to recipes.

An imported invoice can modify 15 ingredients in a single recipe just as it can modify only one (or 0 in some case), and I find it odd to update the recipe 15 times (for each ingredient)… What’s more, the recipe itself can become an ingredient in another recipe, so the complexity multiplies with each step…

As someone with a lot of experience, have you ever seen software like this? That are a little complex in their logic? Your opinion could help me a lot!

georgecollier · 2024-11-18T09:59:16.117Z

antolin.lacaton:

some results are overwritten by other operations, making the whole thing unreliable!

That’s race conditions, as a result of using a List on a data type. Just use a search.

antolin.lacaton:

As someone with a lot of experience, have you ever seen software like this? That are a little complex in their logic?

Yes, and overengineering something like this tends to cause more problems than it solved.

antolin.lacaton · 2024-11-18T10:01:02.224Z

georgecollier:

That’s race conditions, as a result of using a List on a data type. Just use a search.

So according to you, for each step I should just set a trigger that detects a change, and once done I automatically launch the next step?

And how do I know that the previous step is finished and that I can launch the next one?

code-escapee · 2024-11-18T13:41:04.256Z

By putting the next step in a seperate custom event…

antolin.lacaton · 2024-11-18T14:36:31.447Z

Do you have a simple, concrete example so that I can better understand what you’re trying to tell me?

georgecollier · 2024-11-18T18:28:53.843Z

Step 2 will always run after Step 1

CleanShot 2024-11-18 at 18.28.26@2x1560×510 22 KB

jagdish_bajaj · 2024-11-18T20:32:53.754Z

@antolin.lacaton I don’t think this level of complexity is warranted for the use case you describe.

A combination of database triggers and BEWF supported with custom events should suffice. In addition, instead of updating data instantly, I would recommend using recurring events to run daily or weekly routines to loop over data that is not time-critical.

antolin.lacaton · 2024-11-19T11:21:08.486Z

georgecollier:

Step 2 will always run after Step 1

CleanShot 2024-11-18 at 18.28.26@2x1560×510 22 KB

And when I have to wait for a list of items to be modified/created, what do I do?

antolin.lacaton · 2024-11-19T11:24:48.969Z

jagdish_bajaj:

@antolin.lacaton I don’t think this level of complexity is warranted for the use case you describe.

A combination of database triggers and BEWF supported with custom events should suffice. In addition, instead of updating data instantly, I would recommend using recurring events to run daily or weekly routines to loop over data that is not time-critical.

The first version of my method wasn’t as complex, but I had a lot of data that was overwritten due to the fact that bubble was going too fast and triggering steps in parallel!

Recurring events wouldn’t be efficient enough, because that’s precisely my value proposition for this software: to analyze data quickly and reliably for my users.

If you can think of any other way I could avoid such a complex process, I’d love to hear from you!