Bubble Down Aug 6 - Mega Thread

fede.bubble · 2024-08-06T14:13:00.251Z

Hello everyone,

Here’s some info from the team

What are the timings of the data loss
- We restored data from a snapshot starting at [August 06, 2024, 02:05:59 UTC-4:00] which finished at time [August 06, 2024, 05:21:00 UTC-4:00], so between those 2 times, the data was lost (that is US Eastern timezone)
Is it safe to continue working in the editor?
- Yes, but there might be degraded performance due to the lazy loading restore that is still happening
ETA when restoration will be complete?
- Currently we don’t have an exact time, but we will keep this thread updated with progress

Also including here Payam’s update overnight

hey folks,

this will be my last update for a little bit, the next shift will get picked up while we get some rest.

Only data after 2AM est would have been lost, we do not expect changes on branches prior to that have been lost. If you’re having trouble there, please file a bug report.

We’re still not positive about the infinite refresh - right now we’d request trying a cache clear / incognito, in addition to making a minor/invisible change and redeploying again. If this persists, please file a bug report. We’re handing this part off to next set of engineers.

We think some of the issues for example with merge are related to the performance degradation post fix, but the performance is steadily improving.

Otherwise, all databases should be fixed now and are not at risk of recurrence.

Thanks again for your patience and my apologies once again.

Best,
Payam

I’ll be merging every topic about last night’s outage (NYC time) into this mega-thread to keep everything organized.

fede.bubble · 2024-08-06T14:13:06.953Z

aqqura · 2024-08-06T07:20:32.657Z

Yes, but Bubble main office doesn’t work in this time. I’m worried what we will wait during 6-7 hours

chad5 · 2024-08-06T07:28:45.519Z

Looks like the shared cluster is also down

chad5 · 2024-08-06T07:31:48.845Z

Both the editor and my site

godwinalugbin004 · 2024-08-06T07:33:12.417Z

I just checked one of my apps too and this is what I got…

Screenshot 2024-08-06 at 08.32.223359×1851 185 KB

bkerryk · 2024-08-06T07:34:56.919Z

I’ve got this too on a free plan site, but I checked another live site and its still up.

All editors are down though from what I can tell.

stuart8 · 2024-08-06T07:34:59.842Z

Also experiencing the same message as @godwinalugbin004 on my site, can’t access the backend or dev but my live site is fine for now…

PS. not on dedicated.

georgecollier · 2024-08-06T07:36:20.805Z

My live sites are all fine - must only be affecting some apps. You should all submit bug reports @ Bug Report | Bubble

Granted, I just tested on mobile for my own website and two client websites - is my website (linked in profile) down for you guys because it’s okay for me?

gainly.world · 2024-08-06T07:36:36.102Z

Yep having same issue here. No apps are showing on site and if you open from past history gets this

Screenshot 2024-08-06 at 08.35.29

godwinalugbin004 · 2024-08-06T07:36:39.696Z

The screenshot I shared above is from a paid plan. My other apps are working though…

stuart4 · 2024-08-06T07:36:45.530Z

yea, im getting the same (agency plan)

Oh no!

We’re having some temporary technical difficulties right now. We’re working on the problem as fast as we can!

In the meantime, try refreshing the page or hitting “back” on your browser just in case things are working again…

We’re really sorry!

If this message does not go away within five minutes, please report the following code to the team:

chad5 · 2024-08-06T07:36:53.732Z

@fede.bubble there is nothing on the Bubble Status Page, maybe they still don’t know

animisha45 · 2024-08-06T07:37:41.476Z

I think this is only on the editor end. The apps are working fine But not able to load the editor.

Eugene_West · 2024-08-06T07:38:05.579Z

Editor is down, dev preview is down

michal.friedrich · 2024-08-06T07:38:09.244Z

Same issue with the editor, but app seems ok.

boston85719 · 2024-08-06T07:39:43.026Z

But…

Screen Shot 2024-08-06 at 2.38.46 PM1154×718 51.7 KB

Oh Man…Just after the fireside chat with @payam.azadi discussing the hard work they have been putting in…now everything is down, even the status page isn’t working

discovry · 2024-08-06T07:39:46.720Z

The editor is down for us but the live app seems to be working fine.

stuart8 · 2024-08-06T07:41:22.756Z

silver lining, maybe the WU counter is down for live apps

chad5 · 2024-08-06T07:41:34.046Z

Same here, dev editor and dev runtime is down, live site is up