Incident postmortem from 11/29

@josh I think you and your Bubble team are doing a great job, and being very forthcoming. I would suggest that you - if you have not already done so - incorporate additional QA effort in your business process design, a key component of your Enterprise Architecture (EA) as your company grows. Too often we technical folk put the effort into our DB and application architectures but totally ignore our internal EA. (A lesson from a past life)

You should be able to sit on the beach sipping your favorite cocktail without having to worry about your user community, or the most basic tenet of an incident response process.

Respectfully, John

3 Likes

@josh thanks for this update. Just to the above, I’m still suffering the following 4 issues (first 2 since this morning, latter 2 maybe present this morning but I didn’t notice until last hour).

  1. Autosave keeps timing out (every few mins) - actually not sure if this is still present but was a problem throughout this morning UK time
  2. Workflows can’t be edited / can’t create new steps (still an issue)
  3. Plug-ins have disappeared from design tab and plug-in tab
  4. Privacy rule tab is blank.

So this doesn’t seem to be fixed yet for all users (I can see on twitter others are still raising a combination of these for example). Thanks for your help and sharing the info above.

1 Like

Hi Anthony - Can you private message me a link to your app’s editor? I’ll look into this for you - the issue has resolved for most users, but there may be something about your app’s settings that is preventing the patched code from reaching your app specifically.

1 Like

+1 for the Bubble holiday day. just added it to my calendar :sweat_smile:

4 Likes

Thanks! I have just done a quick check before closing down for the evening. It looks like the fixes have come through in the last hour (at least points 2-4 from my post above seem now to be fixed - didn’t have a chance to check 1)). I did submit a separate bug report following my post above so hopefully if there are any issues outstanding, those will be addressed through that.

2 Likes

Sounds good, Antony! Someone from our team will definitely follow up on your bug report.

2 Likes

Hey! I don’t know if it’s only for me, but banner with “We just updated this page. Please refresh the page to get the latest version. You will not be able to use the app until you refresh.” is always active. I’ve tried using different browsers, different devices, but the problem persists and I cannot do a thing. Basically this is related to today’s outage.

3 Likes

I’m seeing that as well. It doesn’t go away, but I’m still able to use it.

2 Likes

yep we also see the banner “We just updated this page. Please refresh the page to get the latest version. You will not be able to use the app until you refresh.”

2 Likes

Seems like the issue resolved for me @tirlikas.m @jof It was happening to me too a little while ago

1 Like

Check out this incident on the statuspage: Bubble Status - "Please refresh this page" banner appearing on bubble.io homepage

Looks like it was just resolved.

2 Likes

Why does the “We have just updated this page” banner not go away when refreshed on bubble’s all apps page? Note this is not in one of our apps, it on bubble’s home/login/all apps pages. Please assist. We can’t get to any of our apps because we need to refresh without the notification going away or allowing us to continue.

1 Like

Hi @udentity.co,

File a bug report for the Bubble Bug Investigations Team to take a look at: bubble.io/bug_report

1 Like

I’m having 30 second load times with both my site and this forum (but not amazon or netflix) right now

1 Like

Same problem here

1 Like

Our mobile application is not responding any action. We are facing this issue from last 24hours after outage of bubble in morning.
If we try to nevigate from screen to another screen or try to perform any action application is stuck.

Here is loop video of problem we are facing:

@uzair.farhan… have you submitted a bug report?

Yes.
Submitted bug report.

1 Like

A big problem, in production, occurred when a backendworkflow recurring every X seconds stopped working, out of nowhere, and did not go up when the bubble returned.

What to do when this happens? Do I lose the parameters passed to the backendworkflow?

This can be very serious.

I await an alternative for verification and greater security in this regard.

1 Like

I’m just going to keep repeating this again and again so that Bubble & Co. don’t have to. And understandably, especially in circumstances like this one, it could be received poorly if it came from them. So here it is:

If you have an app that is making money and these kind of issues are hugely detrimental to you or your customers… Please, please please, I beg you, move to a dedicated instance. Either your app makes money or it doesn’t. If it’s making you money then you need to move to a dedicated instance. If it’s not making you money, then these issues can’t be “very serious” they can only be “man that really stinks”.

I understand from a non-technical background this might not make any sense. However, I implore you to go try and spin up a server with all of the necessary dependencies and have someone write your app code. Once they’ve written it, try making updates without bugs and try keeping your server from crashing. It’s not the easiest thing to do and it’s definitely not cheap.

  • Bubble makes no guarantees that your app won’t experience bugs from time to time (go read the ToS)
  • Bubble is working on a solution to this problem for the non-dedicated cluster, but it’s not going to be available anytime soon.
  • Bubble currently has a solution - move to a dedicated instance

For further reading:

9 Likes