[Scheduled Maintenance] Downtime Feb 25 from 11:30-11:40PM ET

As in there’s still time to start the propagation. Not immediately, unless that’s what you want. Gonna have to plan it out based on your comfort level.

Cloudflare has a redirect rule for managed domains that doesn’t need propagation (I think). GoDaddy on the other hand does need propagation so what I can do is redirect my GoDaddy domains to a Cloudflare one that acts as a bridge. So I can take advantage of Cloudflare’s page rules for redirection.

Though that’s too much of a hassle for me… I’m just gonna put up a banner tomorrow on my apps for scheduled downtime and pray nothing goes wrong.

Right, it is quite hassle for ten minutes downtime. And as of now I also plan to put the banner of same kind. Though of course not everyone will visit the app in the period banner is up. So we do need something elegant at the time of downtime. Issue with putting up a banner in advance is that it sounds silly on us that we are doing planned maintenance in our peak time.

What error? “Host unreachable”? “Server timeout”? “Bubble website could not be reached”? “App name could not be reached”? Server down? Blank page? Please be more clear.

1 Like

A popup starting now would work well:

‘We will be performing a very short maintenance this Sunday, February 24th from 11:30 PM until 11:40 PM. We apologize for the brief interruption. Thank you for your understanding.’

Sending emails to your users would work. If you use SMS or push notifications…as well as posting on your social media.

I would imagine regardless of what 10 minutes they picked it could be a busy time for someone.

10 minutes isn’t that long.

It’s good they notified everyone beforehand about the outage.

It’ll just be a browser error, saying nothing happened. No mention of Bubble. It’s a default browser response, like when you load a site without being connected to the internet.

3 Likes

One down, one to go

Scheduled maintenance last night was a success. I’ve been told down time was less than 2 minutes.

NEXT UP

Last night’s updates will allow the second round of updates to occur tomorrow Tuesday Feb 25 at 11:30pm US Eastern Time.
We expect downtime to be a brief blip just like the first one

3 Likes

@fede.bubble the good news is that we didn’t notice any outage at all during the first maintenance window, however there were 4 reported incidents less than 12 hours later.

For my team across several apps this presented as an issue with the database as any CRUD actions basically ground to a halt and while initial page loads were normal, any dynamic data did not load. Similarly the database was unreachable in the editor or took an extremely long time to load.

I haven’t seen any communications on what happened here and the status page doesn’t really give anything away, but I can’t help but assume that this was connected with the maintenance performed?

Is this now completely resolved, and will the second scheduled maintenance proceed as planned? The more we know the better prepared we can be / can set expectations with clients and so forth.

Thanks.

1 Like

I checked with the team @josh24 and preliminary analysis says last night’s scheduled maintenance was not connected to the service degradation we experience today.
I want to complement that by saying that a full postmortem will take a few days and then we’ll know for sure and the team plans to update our status page to share more about what happened.

2 Likes

@fede.bubble thank you! Appreciate you coming back to me so quickly.

I did have some outstanding concerns from clients this morning as they do see the alerts posted in slack when the status page is updated just as I do, so this has helped me keep them updated.

Looking forward to seeing the postmortem when it’s posted.

1 Like

Is this why my send email workflow is not working?
I realised it wasn’t working 2 hours ago, and it’s still not working now (Feb 25th 12:20 PM Bangkok time)
I’ve set up 2 test workflows on 2 apps, none of them worked.

Everything is down - Test version preview and Live apps. Development screen is opening. Live version will get opened after 8 minutes of loading

hey @it15
Our integration with sendgrid is currently suffering an outage (sendgrid disabled our account) so the ready-to-use emailing workflow won’t work.
Best way to get around this is to use your own API key with sendgrid or any other email provider

What about backend workflows, will they run during this downtime?

If backend workflows are down as well during this time, you are basically telling us that any webhook or scheduled backend workflow will be ignored during these 10 mins. How can we rely on this ?!
You don’t have way to automatically “try again” if a workflow fails

the engineering team told me everything will be down for those couple of minutes. (we set a 10 minute window to be safe)

1 Like

It’s always wise to ensure you construct your webhook endpoints in such a way that when the service that’s calling it attempts to retry, it’s handled correctly.

2 Likes

But we don’t control the service sending webhooks, if they don’t have a retry function. Then the webhook is only received once by the backend and if the system is down there’s no way “to retry”. Unless I’m missing a functionality in bubble.

Almost all will

We can’t rely on “almost”. Bubble should be 100% reliable, not “almost” all the time.

And i can tell you that many services don’t have a retry function. Of course they should but they don’t.

1 Like

Check your API service, most have some kind of retry system assuming Bubble’s backend didn’t receive the webhook and just throw it into a black hole.

Postmark for example:

2 Likes