Bubble down 22nd March - DDoS and other Issues

[johnny]

Everything’s working fine on my end. I recommend filing a bug report at bubble.io/bug_report, so they can take a look into your particular case.

[Jici]

Same for me. without debug_mode it doesn’t load

[brian.good500]

same here… debug_mode=true works, but not otherwise.

[carry]

I’m in the same boat - no apps load w/out debug.

[carry]

My apps are up now!

[brian.good500]

We’re back up now too

[josh]

@johnny

Hey @josh,

I know it’s still pretty early, and y’all are still working to mitigate the issues fully, but could you provide us with an update/detailed postmortem about what happened here today?

• What exactly happened? It seems like it was just a large-scale DDoS attack?
• Does Bubble have any systems in place to mitigate these large-scale DDoS attacks that can take down the main Bubble cluster?
• Are there any plans to make changes internally to prevent this from happening in the future?
• What exactly was disrupted during the attack? It seems like it was the main Bubble cluster and the forum (maybe)? Or was the forum unrelated to what happened with the main Bubble cluster?

Yes, we plan to share a postmortem. This afternoon we’re working on making sure the situation is stable and putting together an emergency response plan in case the issue re-surfaces; we’re going to do a formal postmortem tomorrow, and we’ll likely be able to share updates publicly later this week (there may be some follow-ups coming out of the postmortem tomorrow that we want to track down first).

For people still seeing issues:

• We saw a few problems with apps with non-standard CloudFlare configuration or CloudFlare not turned on, but we believe those are fixed
• Try doing a hard refresh of your browser / clearing caches
• If you are still seeing issues, please file a report at Support | Bubble with details about your app so we can investigate

[mac2]

Thanks, Josh. What a fun few weeks you guys have had.

[johnny]

Thanks, Josh!

[gaimed]

Thanks!

[arttnov]

Hello josh!
I’m relatively new user of your great platform! But yesterday’s case worried me
How often these situations with Bubble happen?
Is there a way to make my/others apps work even if Bubble host is down? For example, host app files on my own server?

[keith]

Subscribe to https://status.bubble.io/ and see, eh, @arttnov? Also: Welcome. Now you’re a web developer, do the web dev things.

[sxtn.digital]

Not working for me today, seems to be problem to fetch option sets aswell…

[johnny]

Have you filed a bug report?

[gilles]

That’s correct. With a. Dedicated plan, your live site is still operational. You sit back and watch.

[gilles]

Great job to you and the team to mitigate this ddos attack. Glad we’re integrated with cloud flare to dodge attacks like this.

[mikeloc]

3/22/2023 Incident Postmortem

Hi all, I wanted to follow up on the extended downtime we had on Wednesday with our postmortem and reflections. We take uptime extremely seriously, because we know for everyone who depends on us as the primary tech stack for their business, downtime can be extremely disruptive and costly, and can undermine the trust you have built with your own users. Speaking for our team, we all feel personally responsible when something like this happens, and want to make sure we are constantly learning and …