Hi everyone,
TJ from Bubble’s platform team here, you might’ve seen me post a few times last month…
Today, however, I wanted to share an exciting update: We now have 100% of our traffic running through our dramatically simplified balancer service. While the rollout encountered challenges — including temporary outages — things are now running smoothly. This is a significant milestone in our continued work to reduce future risk and strengthen Bubble’s infrastructure.
If you followed along during those outages, you might have seen my brief explanation of Balancer and details in the monthly community update. Now I want to share details about what we did (including a post-mortem), why it matters, and what’s next.
How Balancer works
For those of you who didn’t see the earlier forum responses, “Balancer” is a custom service we built to route traffic from users to Bubble servers. It handled four main responsibilities:
-
Serving SSL certificates to allow for encrypted communication
-
Identifying which cluster an application was running on (which involved database lookups)
-
Maintaining a list of the active Bubble servers, their states, and the load that they are under
-
Routing traffic directly to the next available server in that cluster
While it works, Balancer has become a source of complexity and risk. When things go wrong with Balancer, the effects range from degraded performance to complete loss of access to Bubble, so it was important to make changes.
What we changed
Over the last few weeks, we have been working on re-routing traffic directly to AWS infrastructure by removing three of the Balancer’s core responsibilities:
-
Balancer no longer serves custom SSL certificates.
-
It no longer knows about individual Bubble servers.
-
Now Balancer routes traffic to AWS load balancers, which handle the routing to available servers using industry-standard algorithms.
This eliminates our reliance on Balancer’s custom routing algorithm and positions us to eventually remove Balancer service altogether and run all traffic through AWS infrastructure directly.
We also simplified the logic that updates which Bubble servers are currently serving traffic. This process had been prone to timing issues in our Continuous Deployment pipeline which caused outages in the past.
Finally, we improved our ability to test Balancer, allowing for smoother development as we do further work to optimize our traffic routing infrastructure.
Post-mortem: What happened and how we fixed it
As a part of this rollout, however, we encountered some challenges including the following outages. We recognize how frustrating these issues were, and want to apologize again for the disruption. Below are more details:
-
September 15: In order to roll out fundamental changes to how our custom load balancer works, we stepped up traffic through the replacement pathway incrementally, from 5-40%, with no issues. At 80%, connections began to time out or close, leading to page load issues. We fixed this by rolling back while we investigated the root cause. You can read more about this incident here.
-
September 25: Requests were returned with a “you hit balancer” response, a regression that was immediately reverted. The root cause of this issue was a misunderstanding of how certain configuration parameters were populated, leading to empty values sending requests to a testing response.
-
September 29: An issue occurred during what should have been a routine update. In order to monitor load balancer performance, we made changes to how metrics and logs were gathered. During this work, a new dependency broke the library that handles connections, but only under certain circumstances that our tests could not catch. Our other internal systems alerted us to the traffic reduction and we reverted the change. We then removed the broken package and performed additional testing before a successful update.
Benefits for you
As a result of the rollout, the Balancer service is now significantly less likely to be a cause of system-wide failures because:
-
We’ve replaced custom code that had flaws (it would sometimes unfairly overload certain servers) with AWS’s more robust load balancing system.
-
Connections now route directly through low-latency AWS services, removing processing overhead.
-
Deployments are more reliable without Balancer’s entanglement with our main server updates and the timing issues that previously caused outages.
What’s next
This is one step in a larger effort to simplify Bubble’s infrastructure and reduce opportunities for failure.
Our Platform team is working to fully eliminate the Balancer service by moving to AWS infrastructure and DNS-based routing late this year or early Q1. This will lead to even more stability and reliability for our users, since we can use DNS to route requests rather than needing Balancer to decode the app and where it lives.
Thanks for your patience during the recent outages and for the reports you filed that helped us identify and fix issues quickly.
— TJ and the Platform team
