🚧 Upcoming Database Maintenance (March 6–7)

Announcements
1

Planned database maintenance: Friday, Mar 6 – Saturday, Mar 7

Hi everyone,

If you don’t know me, I’m Fede, the community manager here at Bubble. I want to give you a heads-up about some planned maintenance that will temporarily limit certain editor actions next week.

What’s happening

We’re upgrading our Postgres databases. AWS is ending support for the version we currently run, so this upgrade is necessary to keep your apps running reliably on supported infrastructure. It’s a routine but important change, and we want to be upfront about what to expect.

When the maintenance takes place

The maintenance window is overnight Friday, March 6 into Saturday, March 7, from 8 PM ET to 2 PM ET (approximately 18 hours). We recognize that’s quite a long window and plan to work hard to be done faster than that, but please plan for 18 hours just in case.

What will be affected

During the maintenance window, schema changes will be blocked. That means you won’t be able to:

  • Add or modify data types or fields in the editor

  • (Edit as of 3/6) Deploy apps, either through scheduled security dashboard deployments or through the editor

  • Deploy previously changed types

  • Copy, delete, or transfer apps

  • Create apps from a template

If you try to do any of these things during the maintenance window, you’ll see an in-editor notification letting you know that the action is temporarily unavailable.

What will NOT be affected

Your live apps will continue to work normally. All standard data operations — creating, reading, updating, and deleting records — will keep running as usual. Your end-users shouldn’t notice anything.

In short: If you’re not making structural changes to your database you likely won’t be affected at all.

What you should do

If you have schema changes or app transfers planned for next Friday night or early Saturday morning, try to get them done before 8 PM ET on Friday, March 6, or wait until after 2 PM ET on Saturday, March 7. Beyond that, no action is needed on your end.

We know Bubblers build at all times of day every day of the week, so we appreciate your patience as we keep things running smoothly behind the scenes. If you have any questions, feel free to drop them below.

— Fede and the Bubble team

32 Likes
[Reminder] Scheduled database maintenance begins tonight at 8 PM ET
3

Appreciate the heads up

6 Likes
6

I was just informed that the reason our app is completely down is because your update temporarily removed all of our data from our app.

It would seem that your advice of “Your live apps will continue to work normally. All standard data operations — creating, reading, updating, and deleting records — will keep running as usual. Your end-users shouldn’t notice anything.” was inaccurate or misleading.

Either way, our confidence in your platform is completely shattered. We run a 24/7 operation and we have been relying on Bubble for nearly many years. With this incredibly irresponsible “scheduled maintenance” approach we will immediately seek to move to a different platform that considers their methods more carefully.

7

Hey Jerry, sorry about your app being down. This upgrade should not be impacting any apps in the way you described it.

I recommend you reach out to support

1 Like
8

Talk about jumping to conclusions. The fact that absolutely no-one else is reporting downtime should be a good indicator that your issue has nothing to do with the maintenance.

9

Currently at a client’s site, everything seems to be operational, watching them updating records and change statuses on the appointments, however as soon as they refresh the page everything that was changed is gone. It appears that nothing is actually saving.

Anyone else having issues like this?

10

Checking with the team one sec

11

My projects are all missing CRUD functions. This happens on both the frontend and the backend.

I receive messages like these: There was an issue creating the entry: Unexpected server error: please report the code parameter to the team

or Your request could not be completed, please contact the application administrator, or try again in a few minutes.

12

@contato85 mind sending me in a DM your app IDs? So I can share with the team

13

Same problem here, Im having a demo and I can’t create anything “Unexpected server error” appearing

14

Our engineering team is investigating these reports – will let you know what’s going on shortly

4 Likes
15

We’re seeing a ~10 minute long spike in errors that have now returned to baseline – we’re confirming whether that’s the cause of the issues being reported on this forum thread. If so, it may mean that things are working again, but still confirming

16

Doing another check with everyone.

Status page has been updated

We’ve found an issue with our database upgrade procedure that can cause write operations to fail under specific circumstances. We believe this affects a small number of apps on the main cluster. Given that, we are going to abort and reschedule our upgrade operation, which should restore functionality to the impacted apps. We are working on doing this shortly.

17

As Fede mentioned, we just posted an update here: https://status.bubble.io/incidents/01KK4FPM7W3TZJ23B3WR51BBAH

This was related to our database procedure. We currently think it impacts a small number of applications. It’s going to take us a couple minutes to abort the procedure – once it is aborted, impacted apps will be fully functional again.

Our sincere apologies for this. We’ll share a postmortem of the incident, and make sure we’ve fully investigated and mitigated the issue before attempting this upgrade again.

1 Like
18

Our ETA til abort is about 30 minutes

1 Like
20

Everything appears to be back to normal on my end.

Always fun having production issues on a Saturday! Thanks for the quick response.

1 Like
21

thanks

23

You are the only one jumping to conclusions by assuming I hadn’t already been talking to support about this and confirmed. Indeed, I mentioned that I was informed by Bubble of this in my post so you clearly didn’t read it all. In addition to the database issues, there were DNS validation problems on the Bubble side as well. As you can see from comments after mine, my app wasn’t the only one impacted.

If your comments are not helpful or informative, perhaps you should avoid posting them.

24

I might have been too quick to react. Appears an issue affected a small subset of apps. Seeing it’s your first post, and no other reports at the time, it appeared to be another one of the all too frequent armageddon complaints. Can only speak for myself, but we’ve got 100% uptime past 90 days and 99.9% past 12 months.

1 Like
25

Important Update

What was planned

We need to upgrade our databases, from an old version that is no longer supported by AWS, to a new, supported version that provides significant performance improvements and opens up possibilities for better data handling.

Because of the flexibility Bubble provides for data modeling, the upgrade process requires stopping some functionality in the editor, in order to maintain runmode operations.

What happened

During the upgrade process we realized that a component of how change tracking worked was blocked, affecting live apps for a small subsection of users for three hours (8:30am - 11:30am ET 3/6). We were unable to fix this quickly, so we aborted the upgrade, restoring normal functionality.

Next steps

Having fixed the technical issue above, we plan to retry the upgrade on Friday, March 20th, starting at 8pm ET. Thank you everyone for your patience with this upgrade!

4 Likes