What caused the recent outages?

We posted a writeup on the Jan 26 outage here: Bubble Status - Issues with caching layer

The outage this morning was unrelated to the events of the 26th. The proximate cause was a buildup of queries on one of our main databases. We’re still investigating the root cause. Unfortunately, one of the things we discovered while investigating was that there was a bug in some of our database monitoring code that’s supposed to provide visibility into which query is responsible for the buildup in that kind of situation, so we’re not sure we’ll be able to retroactively figure out which query caused the issue. We’re definitely fixing the monitoring problem going forward, and we’re still digging into the logs we do have to see if we can spot the problem.

4 Likes