Bubble Down Aug 6 - Mega Thread

[fede.bubble]

Hello everyone,

Here’s some info from the team

• What are the timings of the data loss
  • We restored data from a snapshot starting at [August 06, 2024, 02:05:59 UTC-4:00] which finished at time [August 06, 2024, 05:21:00 UTC-4:00], so between those 2 times, the data was lost (that is US Eastern timezone)
• Is it safe to continue working in the editor?
  • Yes, but there might be degraded performance due to the lazy loading restore that is still happening
• ETA when restoration will be complete?
  • Currently we don’t have an exact time, but we will keep this thread updated with progress

Also including here Payam’s update overnight

hey folks,

this will be my last update for a little bit, the next shift will get picked up while we get some rest.

• Only data after 2AM est would have been lost, we do not expect changes on branches prior to that have been lost. If you’re having trouble there, please file a bug report.
• We’re still not positive about the infinite refresh - right now we’d request trying a cache clear / incognito, in addition to making a minor/invisible change and redeploying again. If this persists, please file a bug report. We’re handing this part off to next set of engineers.
• We think some of the issues for example with merge are related to the performance degradation post fix, but the performance is steadily improving.

Otherwise, all databases should be fixed now and are not at risk of recurrence.

Thanks again for your patience and my apologies once again.

Best,
Payam

I’ll be merging every topic about last night’s outage (NYC time) into this mega-thread to keep everything organized.

[aqqura]

Yes, but Bubble main office doesn’t work in this time. I’m worried what we will wait during 6-7 hours

[chad5]

Looks like the shared cluster is also down

[chad5]

Both the editor and my site

[godwinalugbin004]

I just checked one of my apps too and this is what I got…

[bkerryk]

I’ve got this too on a free plan site, but I checked another live site and its still up.

All editors are down though from what I can tell.

[stuart8]

Also experiencing the same message as @godwinalugbin004 on my site, can’t access the backend or dev but my live site is fine for now…

PS. not on dedicated.

[georgecollier]

My live sites are all fine - must only be affecting some apps. You should all submit bug reports @ Bug Report | Bubble

Granted, I just tested on mobile for my own website and two client websites - is my website (linked in profile) down for you guys because it’s okay for me?

[gainly.world]

Yep having same issue here. No apps are showing on site and if you open from past history gets this

[godwinalugbin004]

The screenshot I shared above is from a paid plan. My other apps are working though…

[stuart4]

yea, im getting the same (agency plan)

Oh no!

We’re having some temporary technical difficulties right now. We’re working on the problem as fast as we can!

In the meantime, try refreshing the page or hitting “back” on your browser just in case things are working again…

We’re really sorry!

If this message does not go away within five minutes, please report the following code to the team:

[chad5]

@fede.bubble there is nothing on the Bubble Status Page, maybe they still don’t know

[animisha45]

I think this is only on the editor end. The apps are working fine But not able to load the editor.

[Eugene_West]

Editor is down, dev preview is down

[michal.friedrich]

Same issue with the editor, but app seems ok.

[boston85719]

But…

Oh Man…Just after the fireside chat with @payam.azadi discussing the hard work they have been putting in…now everything is down, even the status page isn’t working

[discovry]

The editor is down for us but the live app seems to be working fine.

[stuart8]

silver lining, maybe the WU counter is down for live apps

[chad5]

Same here, dev editor and dev runtime is down, live site is up