Update: Postmortem on April 24 incident

While I appreciate all the Bubble team does to keep things running, and I’m sure it’s a monumental task, it’s time to seriously consider a more WordPress-like approach to updates.

If Bubble would create a way for us to pay for a license and install on our own servers, we could manage the update process better for our own apps and ensure that issues with compatibility/instability/etc. are testing in a staging environment before we push to our live apps.

Pushing updates to live apps without giving us the ability to test them first has never been good practice, but the frequency of issues is getting worse, and it’s pretty concerning. I’d love to be able to utilize Bubble for the long-term, not plan for a hand-coded rebuild, but in order for long-term use of the Bubble platform, we really need the ability to manage updates ourselves and self-host.

3 Likes

Would be great if every code change made by bubble is considered as a version ‘point’ update.

And then allow the user to upgrade to that newer version, or roll back, at his own accord…

5 Likes

I have to disagree with this point. Bubble has made a clear shift away from promoting itself as the go-to for MVP solutions, and has clearly indicated a path towards large, enterprise applications being on Bubble.

I love Bubble, truly. It’s why I made it my job to develop on it. However, lately, they’ve been letting a lot of people down. I may not share the dissatisfaction others here have very publicly shared, but I can totally see where they are coming from. If I still ran on the shared cluster, I’d probably be right here with them being just as upset.

5 Likes

@jared.gibb These burner accounts are out to get you brother.

8 Likes

I agree with @jared.gibb.

Additionally this is one post mortem, Bubble has been out a bunch of times over the last few weeks and we only get a single post mortem.

We’ve had weeks of radio silence. What are we to tell our customers, that bubble has gone down and we have received no response for 2 weeks. When our customer asked why this happens, do we tell them that bubble uses non dedicated accounts as their testing ground to deploy risky updates upon.

It seams that it would be easy to avoid the downtime with less risky updates and more quality control.

5 Likes

You guys should take it private and remove your posts it detracts from the topic.

Where is Mike?

1 Like

Thanks for the post mortem and working on improving uptime.

Seems even tho things are not fully reliable, we are heading in the right direction.

Any update into splitting the shared cluster into smaller clusters so reliability improves?

5 Likes

all code changes deployed are risky.

“but it works on my box”

3 Likes

Thank you @josh.

On our dedicated system, just our editors were down, which is ok as long as our live site is up and running / unaffected, which it was. Dedicated is the way to go to weather these storms.

4 Likes

12 Days later, bubble is down again.

4 Likes

unbelievable, just unbelievable again

3 Likes

Bubble need to stop working on every single initiative and work on nothing but stabilizing their platform! There is nothing more important than that. Some new bell or whistle alone isn’t going to attract or retain anyone. But having a platform that nobody can depend on will both drive away current customers and that reputation will keep new ones away. I don’t get why they don’t understand that.

10 Likes

Honestly, this would probably slow their efforts down even more just to play devil’s advocate. Getting all the engineers, especially the new ones without the institutional knowledge that the veterans bring, up to speed on current code base, and improvement plans, would likely add a lot more time to this timeline.

I don’t think throwing more resources is the solution.

But, I do agree with the idea that yes, a solution has to be found, and quickly. This is getting ridiculous.

3 Likes

Down again, wow

Hey Bubble, we need answers. If you plan to do this periodically, we deserve to be aware of it. Our work and our projects depend on it.

2 Likes

I’m not sure I understand. Are there individual apps that are so workload heavy they are brining the whole site down?