Bubble down 22nd March - DDoS and other Issues

Well…this has been fun :blush:

6 Likes

Still down for me

image

1 Like

Haha I woke up to like 30 emails from Bubble and New Relic :sweat_smile: Seems like an exciting morning at Bubble :sweat_smile:

1 Like

Yep, same here.

Okay so I cant preview my development version. But my live site is running. So that’s good.

You can subscribe for updates into https://status.bubble.io

However if you want to monitor your app independently, you can use any monitoring bit like uptimerobot

Actually, for me editor load normally and version test. Live doesn’t load.

Hey @josh,

I know it’s still pretty early, and y’all are still working to mitigate the issues fully, but could you provide us with an update/detailed postmortem about what happened here today?

  • What exactly happened? It seems like it was just a large-scale DDoS attack?
  • Does Bubble have any systems in place to mitigate these large-scale DDoS attacks that can take down the main Bubble cluster?
  • Are there any plans to make changes internally to prevent this from happening in the future?
  • What exactly was disrupted during the attack? It seems like it was the main Bubble cluster and the forum (maybe)? Or was the forum unrelated to what happened with the main Bubble cluster?

Thanks! As always-- great work!

10 Likes

There are certain plugins available that can alert users when their device goes offline. I was wondering if anyone who has used these plugins has observed whether they notify the user when they are unable to establish a connection even though they are online.

I use New Relic, there’s Uptimemontior, etc.

I also asked @emmanuel a while ago if he could ask Atlassian to enable their status page so other status pages can add it as a third-party component. So, if you’re using Atlassian Statuspage Bubble’s component status will reflect on your status page.

Instatus also supports this.

Ref: Making Bubble Statuspage Components Available as Third-Party Component on all Atlassian Statuspages

1 Like

Status says all system are operational… Thats not the case for me. We’re still down!

As an additional point of detail for Team Bubble if they’re reading this: the “debug_mode=true” versions of both test and live are loading, but all of our pages still aren’t loading without that.

Everything’s working fine on my end. I recommend filing a bug report at bubble.io/bug_report, so they can take a look into your particular case.

1 Like

Same for me. without debug_mode it doesn’t load

same here… debug_mode=true works, but not otherwise.

I’m in the same boat - no apps load w/out debug.

My apps are up now!

We’re back up now too

Yes, we plan to share a postmortem. This afternoon we’re working on making sure the situation is stable and putting together an emergency response plan in case the issue re-surfaces; we’re going to do a formal postmortem tomorrow, and we’ll likely be able to share updates publicly later this week (there may be some follow-ups coming out of the postmortem tomorrow that we want to track down first).

For people still seeing issues:

  • We saw a few problems with apps with non-standard CloudFlare configuration or CloudFlare not turned on, but we believe those are fixed
  • Try doing a hard refresh of your browser / clearing caches
  • If you are still seeing issues, please file a report at https://bubble.io/support/report with details about your app so we can investigate
22 Likes