Update about Bubble downtime on May 8 (and previous days)

Everytime I ask this question, it’s like I’m blacklisted or something. Nobody ever wants to answer, especially from Bubble, and any developer that does answer sorta misdirects.

But this is something I periodically lose sleep over.

Especially since I learned that exporting your app isn’t exporting the entire app and all of it’s configurations. I’m not entirely sure what is exported, but I’ve been told its not the complete app, meaning periodically exporting your app to have backups won’t actually work the way one might think it does.

4 Likes

Hi @josh , I’m sure you will put all your resources to address the reliability issues but please take into consideration another related topic: allow us to manage the situation when Bubble goes down.

Let me explain. I built a loyalty app where Shoppers check-in at a Merchant (my client). The check-in triggers a backend WF that lasts a few milliseconds. If such a WF fails (or lasts too much), I end up having an inconsistent transaction that blocks my app. As already explained to support, I had several failures belonging to the following reasons:

  • reached capacity without any warning to me. For this reason I upgraded my plan.
  • Time-out of a plugin
  • Bubble going down
  • Mysterious reasons likely due to latency problems of Bubble.

The problem is that on the backend there is no way to trigger an event when a WF fails to complete. Therefore I learn of a problem just when my clients complain. This is crazy. I should be able to get notified when my app fails so I can take appropriate actions before receiving complaints. Please introduce also on the backend an action like “when an error occurs”. Thx.

11 Likes

I’ve always used the “Offline Alert Pro” plugin on all my pages, and whenever Bubble is in trouble, all my customers receive an initial message that the connection is difficult… before a crash. So they know they have to wait. Not very good for the image, but at least not too many complaints. :bulb:

Capture d’écran, le 2024-05-09 à 05.23.50

3 Likes

May I ask how many WUs did the DDoS attacks cost? :sweat_smile:

I hope the current challenges of stability and reliability are resolved and agree with the general sentiment shared by others. Dedicated plans should be more protected, code should be deployed earlier in the day and an internal focus should always be on mitigating these issues from the outset.

If a malicious actor deleted our apps and/or data, I assume the response is ‘it’s gone’ and that is unsettling. However I am reassured because I trust our partnership. Your reactions to problems show this is a managed existential risk to each business, inc. bubble directly.

1 Like

Why I had to post this today

@gianluca1 if you have not yet, please upvote the idea on the ideaboard. It has only 16 votes as of now, because obviously nobody is really trolling the idea board, but of course we all know this feature is a MUST have, and not just a ‘good idea’.

Originally posted this 10 months ago, still not implemented.

EDIT @gianluca1 I see that you did vote for it and commented. Excellent. Now we just need everybody else to as well.

3 Likes

Hi, me too I use a couple services that alert me when my app is down but they don’t solve the need to know that Backend WF X failed for users A, B and C. It’s the info I need to fix the inconsistencies occurred to their accounts.

2 Likes

5 posts were split to a new topic: Getting some API requests errors

A post was split to a new topic: Having issues using Google APIs

Our trust is being tested in FAR TOO MANY WAYS for FAR TOO LONG:

  • Reliability & security
  • Pricing switcharoos (what’s to say Bubble won’t pull another shenanigan like the WU rollout in the future?)
  • Saying you will be adding certain features to the roadmap (i.e., ability to specify which fields to send in a search) and then dropping the issue like it’s hot and not responding to those who are asking about it

Very serious issues for me and many others. :confounded:

6 Likes

hey all, some housekeeping notes: please keep your replies relevant to the announcement.
Reply here for things like sharing your concerns and opinions, adding to the conversation around the recent outages, reactions to Josh’s words above, etc.

Do not reply here if you are just trying to troubleshoot issues or ask about this or that not working. That’s what the Need Help category is for.

Thanks! and keep your thoughts coming

4 Likes

Very good option, thanks for sharing, I wonder if it notify for all events that I could alarm more than just inform to clients? Or is measured and only notify in precise oportunities?

Thanks @josh - I think most of us understand that there will be intermittent DDoS attacks. Personally, I appreciate having a full team to focus on defense and remediation when that happens, so thank you for that.

It seems like one positive to come out of this all is that it’s highlighting an important question many of us face in our own businesses: do we need to do more things or do we need to do the things we already do better?

I’ve flagged this question and my preferred path (slow down on new features and do what you’re already doing better please) for five straight weeks in your support@ inbox when something in my app was broken by a new feature release. I doubled down on it when I was told it may take longer than usual to fix the behavior because Bubble was thin on support staff a couple weeks ago.

What I see when I read this string is that a subset of Bubble developers, many of whom have built real businesses that rely on you guys, value consistency, stability and reliability far more than we value the recent feature releases that have undermined those things.

It has to be tough to make roadmap decisions at Bubble right now. In 12-18 months (if not sooner) it will be possible to build and launch web apps with natural language prompts. In 18-24 months those apps will look and perform better than almost all apps on Bubble today.

I’m sure you guys have a lot of work to do to position Bubble to benefit from that shift. I would love to see that happen. That said, until you have the resources to reliably add new features without breaking things, a lot of us are asking you to slow down a little so we don’t have to keep apologizing to our clients for things Bubble has broken and can’t fix quickly.

4 Likes

This is yet another reason why Bubble should seriously consider making self-hosting with a licensing fee an option. Let us install on our own servers and manage our own risk, instead of being a part of a growing hacker target on Bubble’s main cluster.

9 Likes

In my opinion, this is an obvious no-brainer. However, seeing as Bubble has made several profit seeking moves recently, this would probably not happen because it means less profit margins. Even though it’s an obvious benefit to everyone …

2 Likes

Id try either one of these workarounds:

  1. Trigger your backend WF through the API connector, it should send you an error code if something fails/times out.
  2. Break the WF into smaller chunks, maybe 2 different workflows? If WF1 is successful, update a value in your database that it was, and trigger WF2, if its not successful, then retry or handle the error.

I know, extra steps, extra WU, but it seems like a critical feature in your app.

I appreciate this update.

:metal:

2 Likes

I’ve been advocating for self-hosting for the past couple of years, but the response has always been that it’s not a priority. However, I believe that sooner or later, Bubble will have to go down that path to enhance performance and isolate outages.

If you look at most frameworks, they focus on building and improving their core, while leaving cloud services to developers or third parties. Bubble could benefit from a similar approach.

8 Likes

@josh thanks for the update. I am sure you guys understand our frustration and I have faith your team is working around the clock to resolve it, but just know that we are feeling the exact same pain from our end users.

YOU get to control these outages, we don’t.

The story I am telling myself is: bubbles exec team spent more resources chasing shinny new features (such as AI) vs investing resources towards uptime and network security.

If I was in your shoes, I wouldn’t ship a single new feature until we are measuring bubbles uptime in the number of 9’s after 99.9. At this point its a perception issue.

3 Likes

Thank you @bek, option #2 is something I’d rather avoid because that WF is run hundreds of times every day (hopefully thousands when my clients increase) and it already accounts for 15% of my total WU consumption. What more, there is no guarantee that the same problem does not occur while running chunk 2.

Option #1 is super interesting. I’m using the API connector for several 3rd party services. I don’t know how to use it for calling my own app but I’ll go and find how to do that. Hope it does not increase too much the WU.

In any case it would be much better to have a Bubble built-in feature :slight_smile:

1 Like

100% on the last item.

Regional hosting was promised on the roadmap.

I have asked multiple times and get no reply.

In fact I actually got booted from the forum for a couple of days for even asking.

Ironically, it could be likely that it would have mitigated some of the outages

1 Like