Hi everyone,
As you all know, we have had a string of outages over the last few weeks. We know outages impact your dev time and your customers, and we know you’re looking for answers. We at Bubble take this very seriously and as always we are here to be transparent about what happened.
Below you can find the post-mortem about these incidents, and if you want more details on anything in particular, please ask in a comment below.
What happened
-
Development branches unavailable. A change that tightened type safety relied on an assumption that older code didn’t hold, leaving about 220 users unable to open a development branch for roughly 20 minutes.
-
Domain reassignment blocked. A refactor of how domain ownership is tracked prevented a small number of customers from moving a domain between applications, persisting a few days for those users before it was corrected.
-
Errors on dedicated instances. A change to how credentials are stored reached production without enough testing against real dedicated applications, causing about 40 minutes of errors across roughly 71 instances.
-
Full platform outage, recurring. When two systems deployed at the same time, new infrastructure targets failed to register and the platform (editor, live applications, and both immediate and scheduled workflows) went down for about five minutes. It self-recovered each time, and recurred twice more before we were able to ship a durable fix.
-
Elevated latency, then dropped traffic. A routine configuration change interacted with leftover code from an earlier migration and overloaded a core service. Page loads ran two to three times normal latency for two to three hours, and a routing component then dropped about 10% of requests for several hours more until it was replaced.
-
Sign-in and signup failures. A database lookup crossed a performance threshold as usage of one sign-in path grew, causing about an hour of failed logins and signups on bubble.io.
-
Real-time search unavailable. Slow queries filled available storage on a search database node and took it offline, breaking real-time search on the main cluster for about two hours.
-
Cache failover disruption. A cache node failed and its replica took about three minutes to serve traffic properly, rather than the few seconds we expect.
-
Mobile web preview unavailable. A large refactor missed one build path used for mobile previews, taking that feature down for a few hours.
-
Log delivery delayed. A roughly three-hour outage at our logging provider backed up ingestion pipelines; logs from that window were delayed, and a portion may not have been captured.
How we’re preventing recurrence
Every incident has a fix in place or underway. Several shipped during the period itself. The rest involves larger structural work, outlined below, that will be completed over the coming months. In the interim, we’ve added monitoring and alerting across each of the systems involved, tightened review and rollout for changes touching systems mid-migration, and put earlier detection in place so these failure modes surface before they reach customers.
-
Retiring two single points of failure — the deployment mechanism behind the recurring outages, already replaced with standard, industry-tested infrastructure, and the routing component behind our largest incident.
-
Completing our core migrations so newer systems no longer fall back on the legacy ones they’re replacing.
-
Upgrading our testing and release pipeline to catch far more issues before code reaches production.
-
Hardening databases, caches, and logging with better alerting and faster automatic failover.
Incident reference
All incidents below are resolved. Ongoing structural work is described above.
| Date | What customers experienced | Duration |
|---|---|---|
| June 26 | Development branches unavailable (~220 users) | ~20 min |
| June 29 | Domain reassignment blocked(small number of users) | ~2 hrs |
| June 30 | Errors on ~71 dedicated instances | ~40 min |
| June 30 | Elevated latency, then ~10% dropped traffic (main cluster) | ~2–3 hrs + partial |
| June 30, July 9, July 13 | Full platform down, recurring (all customers) | ~5 min each |
| July 7 | Sign-in / signup failures on bubble.io | ~1 hr |
| July 8 | Real-time search unavailable (main cluster) | ~2 hrs |
| July 8 | Cache failover disruption (main cluster) | ~3 min |
| July 13 | Mobile web preview unavailable | 14 hrs |
| July 15 | Log delivery delayed / possibly incomplete | ~3 hrs |
Closing
We know we fell short and we are working hard to deliver the best in class experience we want for our community. We are sorry for the disruption, and will reply to your questions in the comments below.
