I found a way to run API Workflows sequentially

antolin.lacaton · 2024-11-17T15:36:42.071Z

Hello everyone,

A few days ago, I discovered how to trigger backend workflows in a sequential manner (even though Bubble doesn’t support this natively). I’m sharing my solution with you because I would have loved to have a guide like this when I encountered this problem several months ago.

The Limitations of Workflows in Bubble

Here are some key limitations you should be aware of when working with workflows in Bubble:

API Workflows triggered on a list run in parallel, not sequentially.
Bubble prioritizes speed and efficiency, not strict linearity between backend workflow steps.
Step dependencies (using “Result of step…”) become ineffective for complex systems.
A Trigger Database Event cannot directly trigger another Trigger Database Event.

For more details, you can refer to discussions on the Bubble forum that explain these limitations better than I ever could.

My Solution: The “Filler Bucket” Method

The “Filler Bucket” method allows you to:

Execute backend workflows sequentially.
Ensure that a step only starts once the previous one is fully completed.
Reliably track each step while maintaining optimal performance and reasonable WU usage.

How to Implement the “Filler Bucket” Method

To apply this method, you should be familiar with the following concepts:

Trigger Database Events.
API Workflows (Backend Workflows).
Schedule API Workflow and Schedule API Workflow on a List.
Option Sets and Data Types.

Step 1: Break Down Your Logic

Divide your objective into simple and clear steps. Break down the overall logic into specific actions.

Example:

Step 1: Process a list of users to analyze them.
Step 2: Generate a report for each user based on the analysis.
Step 3: Send an email to the user with the results.

Each step must focus on a well-defined action and must not start until the previous one is fully completed.

Step 2: Build Your API Workflows

Create a separate API Workflow for each step. These workflows will perform actions on the ListOfItemStep’X’, which contains the elements being processed at each step.

Step 3: Create Statuses Using Option Sets

Set up an Option Set to track the progress of each step:

Start.
Step1.
Step2.
Step3.
Final.

These statuses will monitor the process in your database, particularly in the BucketManager (explained below). You’ll need one status for each step in your logic.

Step 4: Configure the Data Types

Add two key data types:

1. BucketManager

The BucketManager tracks the overall process and manages associated data. It includes:

ListOfItemStep1: The list of items to process in Step 1.
BucketStep1: The list of BucketFillers associated with Step 1.
ListOfItemStep2: The list of items to process in Step 2 (defined after Step 1 is complete).
BucketStep2: The list of BucketFillers associated with Step 2.
Statut: The global status (Option Set, e.g., Step1, Step2) to track the current step.

You will need as many BucketStep'X' and ListOfItemStep'X' fields as you have steps.

2. BucketFiller

BucketFillers are created for each item in the ListOfItemStepX and track progress at the item level.

Statut: The specific status of this item (Option Set, corresponding to Step’X’).
BucketManager: A reference to the BucketManager parent.

Step 5: Create the FillingLoop (Recursive API Workflow)

The FillingLoop is a backend workflow that associates BucketFillers with their BucketManager.

Input Parameters:

BucketFiller (type BucketFiller).

Actions in the FillingLoop:

Associate the BucketFiller with the BucketManager:

Action: “Make changes to a thing”.
Thing to change: BucketFiller's BucketManager.
Modification: Add This BucketFiller to BucketStepX.
Condition:
- This BucketFiller’s Statut = StepX.
- This BucketFiller is not already in BucketManager's BucketStepX.

At the end, relaunch the FillingLoop:

Action: “Schedule API Workflow”.
API Workflow: FillingLoop.
Parameter: BucketFiller = This BucketFiller.
Scheduled date: Current date/time + 5 seconds.

Add a safeguard before the loop with a “Terminate this workflow when” condition to avoid infinite loops if no elements remain.

Step 6: Add Actions to Your API Workflows

Each API Workflow created in this tutorial Step 2 must:

Create a BucketFiller for each item being processed.
Associate the BucketFiller with the BucketManager and set the status corresponding to the current step (e.g., Step1).

If triggered on a list of 15 items, this will create 15 BucketFillers with the Step'X' status and the corresponding BucketManager.

Step 7: Set Up a Trigger Database Event to Launch the FillingLoop

A Trigger Database Event detects the creation of each BucketFiller to trigger the FillingLoop.

Condition:

When:
- BucketFiller now isn’t empty.
- BucketFiller before change is empty.

Action:

Schedule API Workflow: Launch the FillingLoop with the BucketFiller as the parameter.

Step 8: Link Steps Using Trigger Database Events

A Trigger Database detects when all items in a step are completed before starting the next step.

Example: Transition Step 1 → Step 2

Condition:

When BucketManager's ListOfItemStep1:count is BucketManager's BucketStep1:count.
And BucketManager’s Statut = Step1.

Actions:

Update the Status: Move to Step2.
Update ListOfItemStep2: Define the elements for Step 2.
Schedule the Next API Workflow: Trigger the workflow for the next list using “Result of step one ListOfItemStep2”.

Step 9: Clean Up BucketManager and BucketFiller

The final Trigger Database Event, under the condition:

When BucketManager's ListOfItemFinal:count is BucketManager's BucketFinal:count.
And BucketManager’s Statut = Final.

Launches a cleanup workflow to delete all BucketManager and BucketFiller records and prepare for the next operation.

Conclusion

The “Filler Bucket” method allows you to trigger API Workflows sequentially in Bubble, bypassing its native limitations. This approach ensures each step is completed before moving to the next while maintaining performance and flexibility.

FillerBucket Method1920×1080 262 KB

If you have any questions, feel free to ask or share your experience with implementing this method!

PS: I also have a way to apply this in the frontend, but I needed this backend version to ensure the process runs independently of the user being on the platform.

code-escapee · 2024-11-17T16:46:28.699Z

Hi it seems the process you described would be helpful for race conditions but for sequential API workflows wouldn’t using custom events suffice?

antolin.lacaton · 2024-11-17T19:38:58.664Z

You’re absolutely right that Custom Events can be helpful for creating sequential workflows in some cases, particularly when everything operates within a single workflow. However, the “Filler Bucket” method is specifically designed for scenarios where backend workflows must operate across multiple steps and datasets, especially when dealing with:

Large lists or external API calls: Custom Events don’t support iterating through large lists of items sequentially without triggering everything at once. My method ensures that each item is fully processed before the next starts, avoiding race conditions or partial updates.
Cross-trigger dependencies: Custom Events are limited to the scope of a single workflow and can’t easily handle scenarios where workflows depend on triggers or data changes from other workflows.
Dynamic lists and statuses: My method allows dynamic tracking of each step’s progress through the BucketFiller and BucketManager system, which is crucial when processing lists where the length or complexity varies.

In essence, while Custom Events are great for linear workflows within a single execution, the “Filler Bucket” method is better suited for complex, multi-step workflows with asynchronous dependencies, large data operations, or when strict sequential processing is required across multiple backend workflows.

This is just my opinion, but in my case and my uses (calling microsoft azure API, sending sms/mail to the customer, creating/modifying data, etc…) this method has proved to be the most effective.

code-escapee · 2024-11-17T20:25:35.496Z

Ok but your list can easily be dealt with custom events just nested within a API WF so that it can handle huges lists 9r be triggered from DB triggers

antolin.lacaton · 2024-11-17T20:43:17.677Z

I completely understand your point, but in my case, the number of steps I need to execute varies depending on specific conditions, such as users, dates, or other factors.

To give you an example, my application is a cost-tracking software for restaurant owners. Users import invoices, create recipes from the items in those invoices, and for each new invoice, the system highlights cost changes and their impact on recipe margins. This involves managing data for invoices, suppliers, items, unique items (grouped lists of items), ingredients, and recipes.

The complexity increases as recipes can also serve as ingredients (sub-recipes for other recipes), requiring additional looping. Not all recipes are sellable, so margins are calculated only for the sellable ones, adding another layer of logic.

Additionally, since cost variations are tied to the invoice date, I need to check if a cost is already recorded for the same date or a later date to avoid duplicates or errors. This requires dynamic checks during each step to ensure data accuracy and integrity.

Given these complexities, I must ensure that one step fully completes before the next begins. I’ve tried implementing this with Custom Events nested within workflows, but I couldn’t achieve the flexibility or control I needed.

That said, I’m no expert, and I might be missing something. If you could share an example of how this logic could work with Custom Events, I’d genuinely appreciate it, it might simplify my workflows significantly !!

boston85719 · 2024-11-18T04:40:32.044Z

This method costs way too much WUs, is too cumbersome to setup and maintain. Would be better to use custom events and bulk updates via API connector. I highly recommend asking ChatGPT to give you a better approach than this one it gave you.

betteredbritain · 2024-11-18T06:44:15.616Z

Not sure if its just me but this post we are all responding looks ai generated, so I would take it with a pinch of salt. WU on this flow would be off the charts and the flow would take at least a couple of minutes to complete depending of course on the number of items on the list.

antolin.lacaton · 2024-11-18T09:14:41.690Z

boston85719:

This method costs way too much WUs, is too cumbersome to setup and maintain. Would be better to use custom events and bulk updates via API connector. I highly recommend asking ChatGPT to give you a better approach than this one it gave you.

You’re not the first to tell me, but I haven’t managed to do what I wanted with custom eventts. Do you have any concrete use cases for multi-step logic that work for you?

antolin.lacaton · 2024-11-18T09:18:12.301Z

betteredbritain:

Not sure if its just me but this post we are all responding looks ai generated, so I would take it with a pinch of salt. WU on this flow would be off the charts and the flow would take at least a couple of minutes to complete depending of course on the number of items on the list.

hahaha it’s true that I used GPT to clean up the method so that it could be understood by as many people as possible, since I find it quite complex to explain!

In terms of WU, I don’t see much difference compared to my old system where I used to play with the “Current date time +second” before launching the next step, but I was convinced that it was more reliable in the long term.

Do you have any other ideas for achieving the same result without consuming too much WU?

antolin.lacaton · 2024-11-18T09:26:54.521Z

I don’t know if it makes any difference, but my system has to work entirely on the backend so that my user doesn’t have to be logged in for the logic to be triggered!

antolin.lacaton · 2024-11-18T09:30:10.904Z

I don’t know if it makes any difference, but my system has to work entirely on the backend so that my user doesn’t have to be logged in for the logic to be triggered!

That’s one of the problems I had with custom events, I couldn’t store the results in a custom state since the user isn’t logged in.
What’s more, the next step takes into account the entire list from the previous step before launching.

Example: If I update a recipe’s ingredient list, I need to take all the updated ingredients to update the recipe (to add up all the costs and define the new margin).

I’m sure this isn’t the most efficient way, but if you’ve got something better, I’d love to know it!!

georgecollier · 2024-11-18T09:49:34.387Z

antolin.lacaton:

Example: If I update a recipe’s ingredient list, I need to take all the updated ingredients to update the recipe (to add up all the costs and define the new margin).

So just have a trigger when an ingredient is changed, ane Make changes to the Recipe with Do a search for Ingredients:each item’s cost:sum.

Don’t overengineer it!

antolin.lacaton · 2024-11-18T09:57:17.421Z

georgecollier:

So just have a trigger when an ingredient is changed, ane Make changes to the Recipe with Do a search for Ingredients:each item’s cost:sum.

Don’t overengineer it!

I’ve already tried to make it simpler, but bubble goes too fast and some results are overwritten by other operations, making the whole thing unreliable!

My software allows you to follow the evolution of cost variations over time, so each ingredient contains a list of “ingredient history” with a price associated with a date. The same applies to recipes.

An imported invoice can modify 15 ingredients in a single recipe just as it can modify only one (or 0 in some case), and I find it odd to update the recipe 15 times (for each ingredient)… What’s more, the recipe itself can become an ingredient in another recipe, so the complexity multiplies with each step…

As someone with a lot of experience, have you ever seen software like this? That are a little complex in their logic? Your opinion could help me a lot!

georgecollier · 2024-11-18T09:59:16.117Z

antolin.lacaton:

some results are overwritten by other operations, making the whole thing unreliable!

That’s race conditions, as a result of using a List on a data type. Just use a search.

antolin.lacaton:

As someone with a lot of experience, have you ever seen software like this? That are a little complex in their logic?

Yes, and overengineering something like this tends to cause more problems than it solved.

antolin.lacaton · 2024-11-18T10:01:02.224Z

georgecollier:

That’s race conditions, as a result of using a List on a data type. Just use a search.

So according to you, for each step I should just set a trigger that detects a change, and once done I automatically launch the next step?

And how do I know that the previous step is finished and that I can launch the next one?

code-escapee · 2024-11-18T13:41:04.256Z

By putting the next step in a seperate custom event…

antolin.lacaton · 2024-11-18T14:36:31.447Z

Do you have a simple, concrete example so that I can better understand what you’re trying to tell me?

georgecollier · 2024-11-18T18:28:53.843Z

Step 2 will always run after Step 1

CleanShot 2024-11-18 at 18.28.26@2x1560×510 22 KB

jagdish_bajaj · 2024-11-18T20:32:53.754Z

@antolin.lacaton I don’t think this level of complexity is warranted for the use case you describe.

A combination of database triggers and BEWF supported with custom events should suffice. In addition, instead of updating data instantly, I would recommend using recurring events to run daily or weekly routines to loop over data that is not time-critical.

antolin.lacaton · 2024-11-19T11:21:08.486Z

georgecollier:

Step 2 will always run after Step 1

CleanShot 2024-11-18 at 18.28.26@2x1560×510 22 KB

And when I have to wait for a list of items to be modified/created, what do I do?