As you’re aware, Bubble has experienced significant downtime this week. Here’s a breakdown to explain what is happening behind the scenes:
On Monday, May 6, Bubble experienced a DDoS attack, plus one of our databases went down because of expensive queries. Our team responded and deployed fixes that remedied the issues. You can read more details from me here.
On Tuesday, May 7, there were a few brief interruptions. These were directly linked to the database issues on Monday.
Unfortunately, the DDoS attack escalated earlier today, May 8, when the same bad actors attempted another breach, causing several instances of downtime over the course of the day.
At this point, every engineer on our team with expertise in systems or reliability issues is working full time on remediating the root causes of this week’s outages. What we have accomplished so far:
We have fixed the root causes of each of the runaway expensive queries that caused downtime.
We’ve identified the specific DDoS attack vector and put in place a general mitigation that should prevent this entire class of attacks from happening again.
What we are still working on:
Improving the isolation of main cluster databases so that new types of expensive queries cannot impact the performance of apps sharing the same database. We already have some protections in place here, and those catch the vast majority of expensive queries, but there are some outlier cases that can slip through, and we are hardening our protections to substantially reduce the likelihood of a similar occurrence.
Improving our defense-in-depth against DDoS attacks. We want to ensure that at each level of our stack, an attack will be unable to cause damage to our infrastructure. We typically get hit with DDoS attacks (via the apps hosted on our platform) roughly once or twice a week, and we fend off the vast majority of them without anyone being aware. Our goal is to get to the point where we are invulnerable for all practical purposes.
We know your trust in us is being tested right now. At this point, reliability is our #1 concern as a company, and we have a long to-do list of improvements we are making to harden our infrastructure. Given the number of incidents in the past few days, we do not plan to share a public postmortem for each one — we would rather focus on increasing our system stability. Once we deem that we’re in an acceptably stable state, we plan to share even more details. In the meantime, we are going to do our best to make the path to stability as short as possible.
As always, please continue to flag any issues that come up to our team and in this forum.
The editor should be hosted on its own server, your front-end on its own server, and then you should have your shared clusters, and your dedicated clusters. Is this not what you already have going on?
I feel like this should have been top priority from the get-go. Knowing you’re a competing no-code tool in this growing market, you guys should have been/be prepared for attacks like this.
Good luck guys - the next few weeks really will set the course of Bubble for the next couple of years. It seems like you now understand how important reliability is to us, which is great.
Also, another on the improved communication in recent downtime. This is the main thing I’ve noticed improving in the last 6 months.
I’m very happy for Bubble being dedicated to informing us about these technical details and being transparent; at least now we know they are dedicated to reaffirming trust.
Even though some users (including myself) may seem to demand a lot or complain too much, all we do is for Bubble to open their eyes and see that there’s an incredible community behind their apps and that depends on Bubble; we place our trust in this platform.
My sincere hugs to the team and may you all manage to overcome all these adversities, taking into account all the issues that users have been talking about on the forum and via email support.
Thanks for the update,
However, I am not sure what was the fix here , still some of my app’s pages that are fed with API’s do not work :(.
this is the second time this weeks for hours, I am lucky that my app is on post-mvp development stage and not monetized.
Thank you for this update and to the entire team that is working hard to improve things. While it’s unfortunate to have downtime like this, I’m glad to have a team of experts at Bubble working on security/reliability in these situations; if I was self-hosted and facing similar threats I’m sure the outcome would be far worse.
Not to parse your words but hopefully @josh you know that Bubble has been stress testing the trust of EVERY single serious bubble user way before May 1.
No commitment on refunding all WU charges based on all these outages?!? (And I don’t mean one of these misleading waivers that are not waivers)
Still waiting for a refund for a incident 6 weeks ago when I had a multi million WU spike that Bubble only notified me 22 hours AFTER the spike stopped…
Reminder to the legacy plan community T-5 months until everyone is forced onto the new pricing model that’s still finding usage calculation errors, unoptimized, and failures to notify over a year after its release
I would rather Bubble not release a single new feature if it meant we got a more reliable platform. I have a massive backlog of projects and I have serious doubts as to whether I can move forward with Bubble given reliability issues.
We know your trust in us is being tested right now.
I hate to be “that guy”, and i love Bubble and always wish the best. But honestly, you guys are lucky that there is no no-code full stack solution beginner frendly or a open source solution out there(YET). Because you’ve playing with our trust for a LONG time, not just now. Yes i know there is a bunch of paid no-code stuff, but to me Bubble still wins in simplicity and full stack solution for quick mvps and solutions. Take Unity 3D (Game engine) as an example, they were the kings, just like you guys, they went greedy and now Unreal and Godot (open source) are on top and will take YEARS if not never to recover developers trust.
Thanks for the update. Having used many no-code tools, I still find Bubble to be the best as a full-stack platform. However, if reliability becomes an issue, it may be necessary to consider other options. I hope incidents like this week’s won’t occur frequently.
Thanks for the update @josh. It was only a matter of time before Bubble would be heavily targetted. But I trust that you guys are doing everything in your power to mitigate any attacks.
That said, this does resurface a concern I’ve had about using Bubble for years. If hackers were successful in accessing Bubble’s resources, or even an individual account, and did something malicious like deleted our apps, does Bubble have a way to restore those deleted apps?
I ask, because currently, we’re told when we delete an app there is no way to recover it. But that is a bit unnerving in the case of accidental deletions or malicious hacks.
I ask this question on public forums periodically, and it’s like no one wants to ever answer. Hopefully, you can shed some light on this?
on the improved communication in recent downtime. This is the main thing I’ve noticed improving in the last 6 months.
