Bubble Reliability on Backend Workflows

morgan · 2022-05-22T02:30:43.750Z

I am in the process of testing them however

jared.gibb · 2022-05-22T02:47:50.628Z

The most reliable thing I’ve done is schedule a workflow. Pass that workflow a list. Do some action on the first item in that list. Remove the first item from said list. And then schedule that same workflow sending the remainder of that list of items. Do you know some real recursive action. I am not a fan of using bubbles action on list of items. It doesn’t seem to work well enough for me and it maxes out my server too often for my comfort. I find that doing a recursive action and adding a pause of a half second to a second before it makes the next call is the most reliable thing for me.

The quick and dirty is that bubble is not really great for handling large amounts of items to be processed quickly. Sure I can handle large amounts of items, but to handle them quickly is another issue.

morgan · 2022-05-22T02:57:40.610Z

Yes Agreed. How about using the mysql connector so it queries a mysql database and updated the rows in the database. I have used it before and it runs fast as it’s not relying on bubble’s antiquated and slow database but not sure if that would work better?

bubble.trouble · 2022-05-22T13:15:10.509Z

Personal opinion follows:

I am a huge fan of what Bubble has created. However, at this point in time, I would not trust a Bubble workflow to complete a large processing task that is of critical importance like account billing.

We have a fairly simple scheduled workflow that was running against a lot of records that would randomly fail on us inexplicably. If we kicked off the workflow manually, it completes perfectly every time.

In my opinion, part of the larger issue is that you have no way of knowing when these failures happen, the workflow just dies. What you generally see recommended is to have a second workflow, checking to see if the first workflow is still running and if it’s not then to start it again. With a non-critical task, this is probably acceptable, but for our businesses billing is a critical process that must be reliable. We handle this with a traditional APIs and/or Azure Functions / AWS Lambda / etc.

One point of clarity, if you have 5,000 customers and they all are billed at various random points throughout the week (typically based on what day/time they signed up), one account at a time, this is no problem for Bubble and we haven’t seen any issues in our apps. However, if you are trying to loop over 5,000 accounts and bill them all at the same time, one right after the other, this is the type of processing that I wouldn’t trust with a Bubble workflow.

cmarchan · 2022-05-22T19:58:57.014Z

@bubble.trouble

On each pass in the backend, create logic that records successes and failures. Also create logic to record completion of 5000 passes to get 5000 s/f’s.

You can also create notifications or emails.

morgan · 2022-05-23T01:14:51.034Z

Thanks for your note. It reflects my experience and sentiment precisely. In our case we will not be able to trust bubble to handle the workflow. It is such a shame that the bubble backend is unreliable. I’m pretty sure it runs on node.js which is a toy compared to something like Go(lang) that we will build our own backend on.

TipLister · 2022-05-23T01:22:53.230Z

our app runs a workflow every minute, which creates 1k entries per day, since 6 months. not related to payments but without it the app would not work.
I have a flow that runs to check the 1k entries for errors and alert me if a certain number isnt reached.
I have a flow that acts as a backup workflow if that workflow fails if bubble is down.

Errors of about 30 rows come up once in 3 months. But can be found and fixed. Mostly related to capacity it seems. Often it helps spacing out the workflow into A runs B which runs C which runs D, instead of everything in A and B.

If to proceed is your decision

redvivi · 2022-05-23T08:40:58.216Z

bubble.trouble:

I would not trust a Bubble workflow to complete a large processing task that is of critical importance like account billing.

morgan:

It reflects my experience and sentiment precisely. In our case we will not be able to trust bubble to handle the workflow.

I have implemented a scheduled recursive workflow to process thousand’s of records daily to call a specific API, each of the records having an associated completion state, plus the target API is handling success/failure response codes to then update each row’s status accordingly.
So I would say it is entirely dependent on the robustness of your implementation.

ben.nes · 2022-05-23T09:02:29.277Z

Just my 2 cents but when I do this with a little of calculation for analytics it will timeout or stop randomly

redvivi · 2022-05-23T13:04:32.714Z

Sounds odd, you may have a lingering issue.
Either too many workflows are being executed in // vs. the available capacity, or a data issue.

cmarchan · 2022-05-23T17:11:40.975Z

Agreeing 100% with @redvivi !

bubble43 · 2022-05-23T21:08:06.411Z

I just moved my processes over to a MySQL database.

I was doing a nightly process that would make several hundred API calls and then store that in a database. When using the Bubble DB, I had issues with duplicate records being created but the main issue was the app hitting capacity and timing out.

I recently rewrote my process to send an executable string to a MySQL database and that kicks off a Stored Procedure that inserts it into the table. I am also using a Stored Procedure to pass one record to Bubble, mark it as processed in DB, then do XYZ to it.

Granted, my process does not involve payments… if it did I would want to make sure I had a good logging table.

bubble43 · 2022-05-23T21:12:37.546Z

What plan are you on? Above Personal?

redvivi · 2022-05-23T21:26:49.991Z

It was implemented on a Professional plan.

ri_scc_94 · 2022-08-04T22:02:51.237Z

Hi, could you specify how are your backup workflows?

I would like to know how this backup workflow works (if it’s the same as the other or not)
I also would like to know how your flow alert you if the certain number isn’t reach.

By the way, I’ve watched your video on youtube " Bubble Tutorial: Create a List of Things", I guess that’s how you create the entries, right?

TipLister · 2022-08-04T22:29:09.061Z

yes, thats how i create things.

i have a backend workflow that starts the one that creates things, it records every time its last run time and saves it

the backup workflow restarts the above, if the above has not run in the last 30 minutes, using the “last run time” field.

ri_scc_94 · 2022-08-04T22:38:30.576Z

Cool, but I didn’t understand how it records the “last run time”?
Like,
API workflow → create thing → Change database (last time run) → schedule API current time?

nico.dicagno · 2022-09-26T13:36:38.000Z

jared.gibb:

The most reliable thing I’ve done is schedule a workflow. Pass that workflow a list. Do some action on the first item in that list. Remove the first item from said list. And then schedule that same workflow sending the remainder of that list of items.

For how big of a list have you used this method for? Does this scale?

ferhatgulbahar · 2022-10-25T05:07:41.846Z

My experience so far:

Backend workflows are not reliable. I’ve build many “robust” backend workflows that have run good for months, then suddenly it would stop/skip records.
Dont forget that the calls are asynchronous in Bubble, so you dont really have steps in your workflow - which is the main issue most of the time. You can use custom workflows or result of previous step type of tricks, but it gets slow and messy.

My suggestion would be to have a cloud function that would call your app data API, get all records and loop through.

redvivi · 2022-11-01T20:51:06.597Z

nico.dicagno:

Does this scale?

It does. If you hit a max items in the list, break down your list in multiple ones and start one recursive workflow for each.