Bubble down 22nd March - DDoS and other Issues

Bugs
145

Well…this has been fun :blush:

6 Likes
146

Still down for me

147

image

1 Like
148

Haha I woke up to like 30 emails from Bubble and New Relic :sweat_smile: Seems like an exciting morning at Bubble :sweat_smile:

1 Like
149

Yep, same here.

150

Okay so I cant preview my development version. But my live site is running. So that’s good.

151

You can subscribe for updates into https://status.bubble.io

However if you want to monitor your app independently, you can use any monitoring bit like uptimerobot

152

Actually, for me editor load normally and version test. Live doesn’t load.

153

Hey @josh,

I know it’s still pretty early, and y’all are still working to mitigate the issues fully, but could you provide us with an update/detailed postmortem about what happened here today?

  • What exactly happened? It seems like it was just a large-scale DDoS attack?
  • Does Bubble have any systems in place to mitigate these large-scale DDoS attacks that can take down the main Bubble cluster?
  • Are there any plans to make changes internally to prevent this from happening in the future?
  • What exactly was disrupted during the attack? It seems like it was the main Bubble cluster and the forum (maybe)? Or was the forum unrelated to what happened with the main Bubble cluster?

Thanks! As always-- great work!

10 Likes
154

There are certain plugins available that can alert users when their device goes offline. I was wondering if anyone who has used these plugins has observed whether they notify the user when they are unable to establish a connection even though they are online.

155

I use New Relic, there’s Uptimemontior, etc.

I also asked @emmanuel a while ago if he could ask Atlassian to enable their status page so other status pages can add it as a third-party component. So, if you’re using Atlassian Statuspage Bubble’s component status will reflect on your status page.

Instatus also supports this.

Ref: Making Bubble Statuspage Components Available as Third-Party Component on all Atlassian Statuspages

1 Like
156

Status says all system are operational… Thats not the case for me. We’re still down!

157

As an additional point of detail for Team Bubble if they’re reading this: the “debug_mode=true” versions of both test and live are loading, but all of our pages still aren’t loading without that.

158

Everything’s working fine on my end. I recommend filing a bug report at bubble.io/bug_report, so they can take a look into your particular case.

1 Like
159

Same for me. without debug_mode it doesn’t load

160

same here… debug_mode=true works, but not otherwise.

161

I’m in the same boat - no apps load w/out debug.

162

My apps are up now!

163

We’re back up now too

164

Yes, we plan to share a postmortem. This afternoon we’re working on making sure the situation is stable and putting together an emergency response plan in case the issue re-surfaces; we’re going to do a formal postmortem tomorrow, and we’ll likely be able to share updates publicly later this week (there may be some follow-ups coming out of the postmortem tomorrow that we want to track down first).

For people still seeing issues:

  • We saw a few problems with apps with non-standard CloudFlare configuration or CloudFlare not turned on, but we believe those are fixed
  • Try doing a hard refresh of your browser / clearing caches
  • If you are still seeing issues, please file a report at Support | Bubble with details about your app so we can investigate
22 Likes