That would be very nice to have some sort of advance notice. Or better yet if these updates are done after business hours?
2 Likes
Thanks for the update Josh. I know how hard it is to keep a system as large as Bubble going 100% of the time. Appreciate your hard work to resolve this matter. I know you and Emmanual are working like crazy right now ā Thanks!
I see thereās an āOh noā message being shown instead of our app. I wish Iād have know about this sooner, weāre going to have some unhappy folks. The page is pretty ominous for our users, Iād love to be able to customize it into something more friendly. Hope weāre back up soonā¦
1 Like
Very sorry about the unexpected downtime. Normally I would schedule it outside of business hours, but we had been hitting CPU capacity limits all day, and it was already causing visible issues. Unfortunately, the maintenance is making it worse. Iām currently waiting for either a) the upgrade to complete, b) AWS support to get back to me on why itās taking so long, or c) the backup instance to be ready to run ā whichever comes first will get things back online again.
3 Likes
Thanks for the update Josh. Transparency and updates makes us all so much more understanding and calmer when we know its being worked on, and what caused it.
When you got things up and running it would be nice to know if we have to worry about this happening again, if it was something that can be mitigated in the future.
1 Like
Okay, the restore process completed (e.g., A happened).
Yes, @gurun ā I plan to do a full diagnosis of what happened hereā¦ canāt have bubble going down for that long 
2 Likes
@josh Great! I have been hitting F5 continously (hope that didnt create another cpu spike hah!), so now its all good! 
We have faith in you and appreciate the feedback, people are starting to build more and more complex and serious apps on bubble now and the stability is quite important as this thread shows.
1 Like
Getting a 502 on the Status page:
Thanks for reporting that. We use a 3rd party provider for that (so it doesnāt go down when we go downā¦ in theory, anyway) ā I just reported the issues to them (Iām also seeing a 502, although if I refresh the page a few times sometimes it loads).
Good 'ol Pingdom.
How can you schedule it outside business hours? We live in a global digital world where there are 24/7 operations.
I would really like to status updates or planned maintenance windows posted, it just saves all these questions, and of course panic in someā¦
Keep up the good work!
1 Like
Better yet an architecture with no single point of failure.
Our daily minimum in traffic tends to be around 03:00 - 05:00 GMT. For users on a dedicated plan, we can schedule maintenance around their appsā traffic minimums.
Iām in the process of reviewing our operational practices in general ā weāve been operating in startup-mode, build features as fast as possible, and weāre now starting to see enough successful, serious apps on our platform that I think we need to adjust that.
For now, we: a) try to avoid downtime at all (ie, we try to do all maintenance operations in a way thatās invisible to users), b) we use the forum to communicate about downtime and outages (like this thread). I know thatās not a perfect system, but itāll tide us over until we get something better in place.
8 Likes
I think the approach is sound. If enough folks are screaming when you go down then you know youāre moving out of startup mode. Those folks should be willing to move to a higher plan with some sort of safety about that built in to the pricing - as I think your pricing is very reasonable.
For someone like me whoās still testing itās a minor pain when bubble goes offline. When I deploy to production though and have more on the line Iād like to pay for more assurance about downtime. Overall great work, Iām a happy bubbler
3 Likes
Anyone else having trouble now too? 4:49pm ET today Oct 25 - Bubble homepage is not loading for me. Our app homepage loaded halfway, but it took about 5 minutes to do so.
Update: everythingās loaded and working smoothly now.