# Something is down? **URL:** \ **Category:** Questions\ **Created:** [October 16, 2023, 6:59pm UTC](https://forum.bubble.io/t/something-is-down/288466 "2023-10-16T18:59:01Z")\ **Posts on this page:** 18\ **Page:** 3
**Author:** ![jlaforge007](https://avatars.discourse-cdn.com/v4/letter/j/8c91f0/32.png) [@jlaforge007](https://forum.bubble.io/u/jlaforge007)\ **Post date:** [October 16, 2023, 8:24pm UTC](https://forum.bubble.io/t/something-is-down/288466/41 "2023-10-16T20:24:30Z")
it appears to be working normally ---
**Author:** ![johnny](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/johnny/32/203020_2.png) [@johnny](https://forum.bubble.io/u/johnny)\ **Post date:** [October 16, 2023, 8:34pm UTC](https://forum.bubble.io/t/something-is-down/288466/42 "2023-10-16T20:34:59Z")
There is a way to opt out of updates but it’s only available on higher tiers, but agree with your points about watching the forum post-code push. Also — is there any code reviews before the code is pushed to production? I think really the only way to get through to flag this as a real big issue to Bubble right now is by submitting a big volume of the same type of bug (which is why these forum posts are made, I think? To see if others are having issues, if so, post a bug report) then their success team escalates. Also — does doing @ Bubble still ping the internal team? My question to that is how long does it actually take to escalate. Although I wasn’t directly affected, would love to see how this is going to be prevented in the future @josh! It’d also be great to get a Postmortem ---
**Author:** ![cpopesku](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@cpopesku](https://forum.bubble.io/u/cpopesku)\ **Post date:** [October 16, 2023, 8:36pm UTC](https://forum.bubble.io/t/something-is-down/288466/43 "2023-10-16T20:36:07Z")
I mean to be fair, since users have full control (just about) over their app, it’d be almost impossible to find all the edge cases. We’re just a little prone to more errors because, at least in the last year, there’s been so many updates. The standard and best practices have changed versus someone who has joined bubble recently. That’s my opinion at least. Could there be improvements, yes. I’d say if you notice issues, check to see the release page from bubble as well. You can tie recent releases to bugs if you notice them. [Releases | Bubble](https://bubble.io/releases) ---
**Author:** ![yusaney1](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/yusaney1/32/217015_2.png) [@yusaney1](https://forum.bubble.io/u/yusaney1)\ **Post date:** [October 16, 2023, 8:38pm UTC](https://forum.bubble.io/t/something-is-down/288466/44 "2023-10-16T20:38:41Z")
Yeah, I know there’s an option for higher tiers, that’s why I remarked for shared clusters. > [@yusaney1](#): > > Can we get an option to opt-out of automatic core updates on the **shared clusters**? They can’t use us as beta testers. ---
**Author:** ![johnny](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/johnny/32/203020_2.png) [@johnny](https://forum.bubble.io/u/johnny)\ **Post date:** [October 16, 2023, 8:40pm UTC](https://forum.bubble.io/t/something-is-down/288466/45 "2023-10-16T20:40:33Z")
Agreed! Looking forward to the postmortem ---
**Author:** ![johnny](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/johnny/32/203020_2.png) [@johnny](https://forum.bubble.io/u/johnny)\ **Post date:** [October 16, 2023, 8:43pm UTC](https://forum.bubble.io/t/something-is-down/288466/46 "2023-10-16T20:43:25Z")
I also do admit that the unstableness is sorta getting old I do get that updates are being pushed at a rate that maybe internal processes aren’t being updated at the frequency they should be but maybe there should be a pause for at least 1 week to figure out the internal processes before continuing to ship potentially broken, buggy code ---
**Author:** ![fabrice.latour04](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/fabrice.latour04/32/111451_2.png) [@fabrice.latour04](https://forum.bubble.io/u/fabrice.latour04)\ **Post date:** [October 16, 2023, 10:43pm UTC](https://forum.bubble.io/t/something-is-down/288466/47 "2023-10-16T22:43:11Z")
Can we please have an explanation as to why the [Status page](https://status.bubble.io/) is all green today even though the main cluster had issues for more than **1h**? ![image](https://us1.discourse-cdn.com/bubble/original/3X/d/6/d616039b0663f64476472ca9b7418a9ac061317a.png) How can we explain to users of our apps why their sessions weren’t working correctly when no issue was reported? What’s the point of subscribing to the Status page if metrics do not represent the true states? Many thanks. ---
**Author:** ![42.decaen](https://avatars.discourse-cdn.com/v4/letter/4/9fc348/32.png) [@42.decaen](https://forum.bubble.io/u/42.decaen)\ **Post date:** [October 18, 2023, 8:02am UTC](https://forum.bubble.io/t/something-is-down/288466/48 "2023-10-18T08:02:22Z")
@josh Has there been any postmortem on this issue? What is Bubble doing to prevent this from happening again? Also, my biggest frustration was that after checking out the status page which was all green, I thought the issue was on my end. So I started to dig everywhere until I (fortunately) came across this post on the forum ---
**Author:** ![josh](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/josh/32/207556_2.png) [@josh](https://forum.bubble.io/u/josh)\ **Post date:** [October 18, 2023, 2:35pm UTC](https://forum.bubble.io/t/something-is-down/288466/49 "2023-10-18T14:35:58Z")
Some follow-up here: In terms of updating the status page, after the last round of changes we made following [my post here](http://forum.bubble.io/t/incident-postmortem-and-updates-on-reliability/284266), this is our process: - We declare an incident whenever: - Our automated monitoring detects changes in metrics or system unavailability that indicate a severe issue - Our customer support team gets 3 or more bug reports pointing towards something having recently broken in our system. Because bug reports are spread out across different members of the team, we have a process for tagging them and looking for patterns to detect if incoming reports are related (we get a constant stream of reports about long-standing issues, corner cases, and bugs that turn out to be on the user’s end, that we have to filter through to determine if something is actually wrong). - Declaring an incident does two things: - It automatically posts a message to our status page that we’re investigating - It pages an on-call engineer to ensure someone begins investigation - The on-call engineer then keeps the status page up-to-date with the progress of their investigation What happened this time around: We shipped a bug that caused privacy rules using the ‘contains’ operator to fail, causing data protected by those rules to be hidden. Using ‘contains’ in privacy rules is relatively uncommon: over the course of the incident, we received 10 total bug reports about it. Those reports starting gradually trickling in around the time this forum thread got started. Because of the relatively low volume and slow trickle (compared to other incidents where we’ve received 100s of bug reports), it took about 45 minutes for our support team to notice the pattern and that we had exceeded the 3 related bug report threshold. By the time they noticed, the engineering team who had shipped the bug was already aware of the issue and working on a rollback because one of the team-members noticed this forum thread. This hit an ambiguity in our process: our support team was unsure if they were supposed to declare an incident given that engineering was already aware of and responding to the situation. Meanwhile, our engineering team wasn’t updating status page because they weren’t officially on-call and aware that this was an ‘incident’ that needed communication. We are fixing both ambiguities going forward: - We are clarifying with the support team that they should always declare an incident if there isn’t currently one ongoing, regardless if engineering is aware of and working on the issue already - We are educating the engineering and product teams on how to declare incidents and making sure they know it’s important to do so in a situation like this even if support hasn’t officially declared one. In terms of the technical root cause of the bug and how it was shipped to production, the issue was with a low-level component of our query evaluation system, and the engineers working on it weren’t aware that it could have implications for the ‘contains’ operator in the context of privacy rules, so didn’t carefully test that use case. Our automated tests do provide coverage for the ‘contains’ operator in general, and do test privacy rules, but we didn’t have a test that specifically covers the combination of using ‘contains’ inside a privacy rule. We’re writing more automated tests to make sure there’s good coverage here. One of the challenges we face in general, which is why our reliability isn’t where I would like it to be, is that Bubble’s features can be combined in an exponential variety of ways, and it’s sometimes possible for there to be bugs that only manifest for certain combinations of features. Our strategy here is to dramatically ramp up the investment in test creation and coverage, which we’re spending time on this quarter, but to be honest I expect getting to full coverage to be a journey, not a quick hit, because of the myriad of ways our users can combine features. Finally, to the point about opting out of automatic updates on shared clusters, this is something we would love to provide, but because of the nature of shared servers, it’s a big technical investment that requires substantial changes to how we generate, package, and host applications. We are working in this direction but it’s far enough off I can’t give an ETA. I will point out, though, that we do offer the Scheduled tier at a lower price point than Dedicated: it’s available on the Growth plan and above. While the Scheduled tier doesn’t offer the same level of control as Dedicated (we still push updates automatically, but we do it on a time-delay), it means you are only exposed to bugs that we don’t notice til the next day. In this case, because we fixed the bug in less than an hour from the first reports, apps on the Scheduled tier were not impacted. You can change your tier on the Settings → Versions tab of the editor. ---
**Author:** ![Benjamin\_Rodgers](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/benjamin_rodgers/32/262514_2.png) [@Benjamin\_Rodgers](https://forum.bubble.io/u/Benjamin_Rodgers)\ **Post date:** [October 18, 2023, 2:51pm UTC](https://forum.bubble.io/t/something-is-down/288466/50 "2023-10-18T14:51:46Z")
Reading about how ‘contains’ in privacy rules is considered ‘uncommon’ made me chuckle. I wonder if anyone at Bubble actually uses Bubble for work purposes? I find that some of my most pressing issues seem to come from design decisions made by someone who hasn’t had to use the feature themselves. Here’s a breakdown of how integral the ‘contains’ operator is in our privacy settings: - User’s company contains transaction - Message contains this user - User access contains edit, view, delete - Company contains user - Thread contains user - User’s role contains customer, production, admin - Invoice contains user - Dozens more… In my app, these aren’t edge cases; they’re the norm. I find the ‘contains’ operator to be absolutely crucial for basic privacy setups, and it’s hard to imagine building a functional app without it. ---
**Author:** ![Benjamin\_Rodgers](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/benjamin_rodgers/32/262514_2.png) [@Benjamin\_Rodgers](https://forum.bubble.io/u/Benjamin_Rodgers)\ **Post date:** [October 18, 2023, 3:02pm UTC](https://forum.bubble.io/t/something-is-down/288466/51 "2023-10-18T15:02:02Z")
> [@josh](#): > > Because of the relatively low volume and slow trickle (compared to other incidents where we’ve received 100s of bug reports), it took about 45 minutes for our support team to notice the pattern and that we had exceeded the 3 related bug report threshold. Given the complexity of filing a bug report—especially when one doesn’t know what exactly is wrong—I’d suggest a “HEY YOU BROKE SOMETHING” button for the community. If enough clicks accumulate in a specific category, then you could start investigating. Also, it’s 2023; why not deploy some AI to monitor the forum? If a thread gains 30+ “me too” or “Bubble’s broke” posts within 20 minutes, perhaps it’s worth looking into. ---
**Author:** ![thekeybeats2](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/thekeybeats2/32/161154_2.png) [@thekeybeats2](https://forum.bubble.io/u/thekeybeats2)\ **Post date:** [October 18, 2023, 3:14pm UTC](https://forum.bubble.io/t/something-is-down/288466/53 "2023-10-18T15:14:40Z")
When I read that “contains” in privacy rules were uncommon it gave me a fright! “Like have I been doing something wrong…?” 😅 ---
**Author:** ![artemzheg](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/artemzheg/32/9651_2.png) [@artemzheg](https://forum.bubble.io/u/artemzheg)\ **Post date:** [October 18, 2023, 3:16pm UTC](https://forum.bubble.io/t/something-is-down/288466/54 "2023-10-18T15:16:13Z")
> [@Benjamin\_Rodgers](#): > > Given the complexity of filing a bug report—especially when one doesn’t know what exactly is wrong—I’d suggest a “HEY YOU BROKE SOMETHING” button for the community. If enough clicks accumulate in a specific category, then you could start investigating. That’s what Bubble devs has been asking during this year suffering from editor bugs… P.S. we’ve tried to collect bug reporting improvements here: > [@Make bug reporting great again (bug reporting improvements)](http://forum.bubble.io/t/make-bug-reporting-great-again-bug-reporting-improvements/274969): > > Problem Current bug reporting system is overcomplicated. This leads to frustration and keeps developers away from reporting (you can easily find tons of messages on the forum or on Twitter about it). How can it be improved 1. Make it possible to report a bug right from the Bubble editor Sometimes ago Bubble [used to have](https://www.youtube.com/watch?v=uX44BUxtdXs) bug reporting link in the editor (in Help section). But after transforming Help dropdown into a Help center sidemenu this link has disappeared. Furthermore, don’t just redirect… ---
**Author:** ![georgecollier](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/georgecollier/32/182327_2.png) [@georgecollier](https://forum.bubble.io/u/georgecollier)\ **Post date:** [October 18, 2023, 3:21pm UTC](https://forum.bubble.io/t/something-is-down/288466/55 "2023-10-18T15:21:39Z")
I still feel the exact same way as after the previous editor issues a month ago: > [@Bubble Service Level Agreement: why doesn't it exist?](https://forum.bubble.io/t/bubble-service-level-agreement-why-doesnt-it-exist/284116/9): > > I’m not technically qualified enough to fairly judge but from a Bubble _user’s_ perspective, **it feels as if** **Bubble’s underlying engine (both run mode and the editor) is built on a house of cards that could all just come tumbling down one day.** It doesn’t particularly matter if that’s true or not, but if that’s how it feels, you can bet I’ll avoid Bubble dependency in the future… > [@NigelG](#): > > Now…there is a sense of dread each morning for how long before something breaks. Echoing this. ---
**Author:** ![fabrice.latour04](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/fabrice.latour04/32/111451_2.png) [@fabrice.latour04](https://forum.bubble.io/u/fabrice.latour04)\ **Post date:** [October 18, 2023, 3:37pm UTC](https://forum.bubble.io/t/something-is-down/288466/56 "2023-10-18T15:37:18Z")
I am wondering why a similar [Notifications for spiking workload unit consumption](http://forum.bubble.io/t/notifications-for-spiking-workload-unit-consumption/280655) can’t be implemented for notifying bugs and issues? This would make the developer debugging process much easier, instead of navigating to the Forum or Status page. Also, why not update the Status page post-mortem to reflect events that happened? Not only would this allow us developers to follow Service disruptions, but in instances where we didn’t witness the bug, we could go back and cross-reference the timestamps issues happened. ---
**Author:** ![Benjamin\_Rodgers](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/benjamin_rodgers/32/262514_2.png) [@Benjamin\_Rodgers](https://forum.bubble.io/u/Benjamin_Rodgers)\ **Post date:** [October 18, 2023, 3:48pm UTC](https://forum.bubble.io/t/something-is-down/288466/57 "2023-10-18T15:48:39Z")
Exactly, a ‘[Down Detector](https://downdetector.com/)’ style feature on the status page would be incredibly useful. It could help us quickly identify if an issue is platform-wide or just related to our specific app. Seems like a no-brainer to implement. ---
**Author:** ![code-escapee](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/code-escapee/32/195485_2.png) [@code-escapee](https://forum.bubble.io/u/code-escapee)\ **Post date:** [October 18, 2023, 5:40pm UTC](https://forum.bubble.io/t/something-is-down/288466/58 "2023-10-18T17:40:23Z")
> [@josh](#): > > In terms of the technical root cause of the bug and how it was shipped to production, the issue was with a low-level component of our query evaluation system, and the engineers working on it weren’t aware that it could have implications for the ‘contains’ operator in the context of privacy rules, so didn’t carefully test that use case. Is that an excuse or a mea culpa? > [@josh](#): > > Our strategy here is to dramatically ramp up the investment in test creation and coverage, which we’re spending time on this quarter, but to be honest I expect getting to full coverage to be a journey, not a quick hit, because of the myriad of ways our users can combine features. @josh as bubble developers are your true customers, can we vote on what’s more important? I believer **test creation and FULL coverage** comes before any other features or updates. Any Bubblers disagree?? ---
**Author:** ![code-escapee](https://sea1.discourse-cdn.com/bubble/user_avatar/forum.bubble.io/code-escapee/32/195485_2.png) [@code-escapee](https://forum.bubble.io/u/code-escapee)\ **Post date:** [October 18, 2023, 5:47pm UTC](https://forum.bubble.io/t/something-is-down/288466/59 "2023-10-18T17:47:26Z")
> [@Benjamin\_Rodgers](#): > > Given the complexity of filing a bug report—especially when one doesn’t know what exactly is wrong—I’d suggest a “HEY YOU BROKE SOMETHING” button for the community. If enough clicks accumulate in a specific category, then you could start investigating. > > Also, it’s 2023; why not deploy some AI to monitor the forum? If a thread gains 30+ “me too” or “Bubble’s broke” posts within 20 minutes, perhaps it’s worth looking into. @Benjamin_Rodgers great suggestions and here are some more @josh: 1. Have a dedicated monitor of the forum for the 12-24 hours after ANYTHING is shipped to production; 2. Trigger a investigation whenever a list of knowledgeable bubblers post a bug 3. Ensure the engineering team is “on call” when anything is shipped to production (see quote below) > [@josh](#): > > Meanwhile, our engineering team wasn’t updating status page because they weren’t officially on-call and aware that this was an ‘incident’ that needed communication. [Previous page](https://forum.bubble.io/t/something-is-down/288466.md?page=2)