Greetings,
My name is TJ, an engineering manager at Bubble for both our AI and Scale teams. I would like to give an update to this issue on the WU spike in January. A regression in an optimization we made for real-time updates caused the client-side code to do more redundant work than necessary regarding searches / search invalidations. This regression is what caused an increase in workload units for some customers in January.
Here are the answers to some questions you probably have:
Q: When did it get resolved?
A: The regression was pushed on the morning of January 2, and it was resolved around 11:50 AM ET on Feb 8 for the main cluster, and 9 AM ET on Feb 9 for the scheduled cluster. However, it may have taken some time after this for it to fully resolve on your WU charts since the fix wouldn’t affect the sessions of any end-users who already had an active connection with your app when it was deployed.
Q: Why did we make the original optimization?
A: We are working to help Bubble apps scale significantly. To that end, this fix was intended to reduce inefficiencies with how our servers handle real-time updates.
Q: Why did it take so long from initial reports to resolve the problem?
A: There were a couple of reasons that caused this delay in both identification of the issue and resolution:
- Our initial hypothesis was that this was related to a previous bug report we received in November related to searches in repeating groups (this post). While that turned out to be unrelated, our investigation down that path did cause us to miss the regression that happened on January 2.
- Our process now alerts us more efficiently, allowing us to quickly resolve issues and improve real-time updates and WU cost monitoring.
- As Bubble grows, we are still building out real-time observability tools to alert us to anomalous behavior. While we do have observability around WU, in this case we didn’t have alerting around real-time search WU costs.
- For this point, we are prioritizing WU observability and testing for more of our end-to-end flows.
- Our goal is to significantly improve the scalability of our data operations, allowing your apps to manipulate millions of things instead of just tens of thousands. To achieve this, we must make significant changes on our end. These changes will ultimately result in more efficient scaling for your app without an increase in spend on your side.
Q: What is Bubble going to do to refund / credit our WU due to this mistake?
A: We want to make it right as this was our mistake. If you’ve seen an unexpected increase in workload usage in your app since January and believe it may be related, please visit the Support center to get in touch with a member of our Support team. Simply open the AI-powered chatbot, ask to talk to a person, and provide the details of your situation (including a link to your app editor and a reference to this forum post) to ensure your question is routed to the right team.
–TJ Andrews