I encountered an issue where users were seeing data from other users, like appointments and user-specific details. This was extremely concerning and put my app’s credibility at risk. Thankfully, the problem has been resolved, but it was a serious breach that impacted my confidence in using Bubble.
To anyone still experiencing issues:
Please submit a bug report so the team can collect as much info as possible about this. Go to Contact | Bubble 
click the chat bot icon (bottom right) click send a message report a bug
@josh for your postmortem: I’ve gotten some feedback from clients this morning who were impacted by this issue yesterday. In particular, they were surprised that their slack integration with Status Page didn’t ping ANYTHING about this issue (probably b/c it was “Degraded performance” and not an “outage”). The info is there (kind of - its pretty generic) on the status page - but you have to 1) know what you’re looking for and 2) go looking for it.
I can work with customer perceptions if they know something is going on - they’re pretty understandable about that - but if the primary tool we’ve told them to rely on to understand when issues are occuring doesn’t tell them, then their trust is degraded.
@lostsheep Very fair feedback. You and your clients, and everyone else affected in this thread, have my apologies.
Not to minimize the impact this had on you all, but the reason this didn’t make our status page is that bug reports trickled in pretty slowly on this one. By the time it reached a threshold where it was clear that this was a “drop everything to fix” situation, our engineers already had the fix in the process of being deployed based on the first bug report they saw. We weren’t sure how many apps impacted there were (based on the nature of the bug, it only affected a relatively small fraction of our user base), and we generally use status page for situations that impact a large percent of our userbase.
That said, I know this kind of thing is extremely scary and high impact for the users who are affected, and that some of you are still dealing with the aftermath in terms of cleaning up any affected data or transactions. If you need help or advice from us with the clean up, please reach out to support (we don’t have tools beyond our standard database restore capabilities to instantly fix things, but we can consult case-by-base).
I’ll share more details about our postmortem in our stability update this Friday. Again, we know this was a very disruptive incident for everyone who was affected, and we’re going to do our best to make sure nothing like it ever happens again.
honestly this is one of the most worrying things i have ever seen happen.
things mapped to X showing Y.
here only the users who saw things and use the forum are shown. not the dark number.
its things like these that made me learn xano or stay away from developing any fintech or health apps that actually look at patients ie things which aren’t just nice to have CRMs but have actual implications if they do not work.
thanks for pivoting in the last months to: we need to fix reliability and communicate. yes, this opens some pandoras boxes like the one above but in 2 years it has good chances to yield the stability bubble needs to capture the enterprise.
when i was in customer service for a fintech we probably answered most bugs that came up from email within 30 minutes and product knew about them in 30 minutes as soon as it was more than 3 people affected by same bug. your response time is still a bit slow. you can even build a scraper for this. new bug threads or new bug threads by top 1000 users etc with replies > 3 etc.
Everyone is like that, but we know the end
@josh to be honest, it feels like Groundhog’s Day… again.
To summarize:
- Bubble is aware of when something is shipped to production (usually).
- Bubble shipped something important to production during business hours (yet again).
→ Therefore, for the next 6 hours, Bubble MUST have a team monitoring the forum, bug reports, and servers to detect if there are any errors or bugs with the recent update.
Every post mortem contains yet another explanation why the bug wasn’t detected or anticipated, but there is no explanation why it takes so long to sort it out. Deploying a triage team after every update to prod 
will at least cut down on the response time.
If you want to earn our trust back, please be transparent: What is the threshold for a “drop everything situation”? After @hi.luisacosta and @georgecollier both posted about the bug, your first post was more than 100 and 70 minutes later, respectively, acknowledging that there was in fact an issue.
Whenever a bubble expert (who knows when something is actually a bug) reports a bug within 6 hours of something being shipped to production, the threshold should be reached basically IMMEDIATELY.
(While monitoring the forums isn’t usually necessary to determine bugs, given the hurdles Bubble requires users to jump over to submit a bug report, it’s on Bubble to do so.)
@josh creating a plan where users can get on a dedicated server without the cost for all of the other bells and whistles that come with an enterprise plan would solve the vast majority of the problems we’re facing with staying on Bubble. We would gladly pay 4 or 5 times as much for a dedicated server so we could control when Bubble code gets rolled in and be insulated from external threat related downtime. Unfortunately, right now the only option for a dedicated server is an enterprise plan which costs more than 10x what we pay now. We’re a growing, viable business but that kind of a jump in overhead is very difficult while we’re scaling.
I agree, I’m assuming I was one of, if not, the first to submit a report and then explaining that the following details aren’t required and sending a link to the forum post to get some more attention to the problem quickly, is frustrating in these stressful situations. Especially when you guys can pinpoint quite quickly that it’s related to work being done on what you mentioned last week.
Someone monitoring the forum is an excellent way to stop it escalating so quickly.
I only deploy in the morning when I know I have a free day, all day, to rectify any issues. I’m monitoring immediately after and assumed, with no experience whatsoever, that that’s standard practice 
I wouldn’t even stoop that low. Bubble being a website builder with lock in and no ability to self host has an obligation to keep our sites up and stable. Shopify, Wix, Squarespace, simvoly, etc although less complex platforms don’t have even close to this downtime.
We can’t keep lowering the bar for bubble to the point we are paying “4-5x” normal pricing just to get the basics of reliability they should already be providing as a web hosting company with platform lock in.
Many platforms in the no code space have moved to allowing self hosting and if bubble refuses to then they need to offer hosting that is stable. There’s not really an in between.
@fede.bubble @josh we (@jeallison and I) have several users who as of this afternoon (6/4 1pm CST and onward) are still getting logged in to other users’ accounts. We have multiple screenshots/videos/email support requests from users as recently as the past 2hr.
Your support team keeps telling us things are resolved while we have end users getting logged into Super Admin accounts, potentially getting access to sensitive information, and who knows what other kinds of mayhem occurring since we run automatic payments and invoicing through our platform.
Yesterday your support team was on top of this…today we’re still experiencing it and we’re getting canned responses from your Bubble Support to send us on a fool’s errand.
Will you please look at our last email back to Bubble Support and urgently escalate this to your engineering team ASAP?
We are experiencing 404 errors on a number of our own endpoints (i.e creating backend WF’s that we have in the API connector and call that way)
Just about to submit a bug report - but posting here so other people are aware this is happening.
Seems like there are issues with backend triggers not triggering as well intermittently.
You’d be the second person today, but it’s an unrelated issue.
I’ll flag this to the team so they can take another look at what’s going on
Just been made aware that a date of event changed in my system. How do I even start to go about checking the 10s of thousands of rows of data to see what’s been changed during this glitch?
Do you just promote other tools here 
This is definitely giving me some data anxiety about my own app… Is there corrupted data, will there be more in the future, if I don’t happen to check the forum for 3 days will I miss another potentially breaking update. This feels much worse than the downtime we’ve been having
Agreed. I’m so stressed right now as I literally have no idea how to check all my data.
@fede.bubble is there any logs of the data that changed during the time period since I first posted about this?
hey @LA81638, we’ve created a work group to look into all of this.
Unfortunately this might be a case of writing correctly the wrong data. So the issue was fixed, but it did things while it was ongoing that are harder to troubleshoot.
Hopefully I have more to share on this soon