Forum Academy Marketplace Showcase Pricing Features

What is Bubble's policy on pushing changes?

Some important questions – what is Bubble’s policy on pushing changes? Why is the current policy the way it is? And/or what should it be?

In the past, Bubble has sometimes intentionally or unintentionally pushed breaking changes into live apps, rather than rolling out the change via a new version. But I thought that after situations where apps broke and users complained, the default is changes are deployed via new versions. But then I saw this posted today in the community update comments…

I thought there was a hard policy against forcing changes onto apps that might break the app. But after seeing this, the policy seems to be kind of squishy.

It’s confusing to me why Bubble would not have a hard policy that changes get put into a new version, and if they aren’t and are pushed onto apps, there has to be an incredibly compelling and urgent reason (like we are fixing a major bug or immediate security vulnerability).

Pushing a change via a new version gives the ability to test before deployment and roll back if needed. Also, new versions have documentation on what’s changed, so that app builders know what to check when they upgrade.

This seems like such a simple and logical policy to have. It would prevent apps from breaking, would give users a lot of comfort, and is part of being a “commercial grade” platform. But because situations like this keep happening, I wonder if I’m missing something?

5 Likes

Pinging @josh and @Bubble for any input on if there’s an official policy when pushing changes to our apps.

1 Like

Hey all –

First, @samnichols , really sorry about what happened with your app. I just checked in with the team and they’re pushing a fix right now (they tried to get it out yesterday but ran into some technical issues). Should be live shortly. Anyway, if you continue to have problems, please keep the support thread you have open with us updated, and we’ll get to the bottom of them.

Addressing the more general question:

Our policy is to never knowingly release a change that will break user apps without creating a new opt-in Bubble version (or experimental feature that will later become a version). There’s been a couple of occasions in the past year where we’ve violated this, generally because of miscommunications or newer people on the team, but each time we discussed the incident internally and reiterated our commitment not to do this.

We do currently release changes that we do not think will break user apps, including bug fixes and performance improvements. This is not an ideal policy, because it leads to incidents like this one: Bubble is a very complicated product, and sometimes we miss ways in which something we think is a harmless change will cause problems. In this case, 99% of repeating groups were unaffected, but for people like @samnichols in the unlucky 1%, a bug like this can be incredibly disruptive.

The reason we have this un-ideal policy is because of technical limitations on our end. Our “versions” tab in the editor doesn’t represent truly different versions of our code. Instead, it’s basically an “if statement” behind the scenes: “if the user app is on version 5, do this; if they are on version 6, do that”. Because of that, we think it’s unsustainable to release all code changes as new Bubble versions: our codebase would quickly become riddled with different branches and totally unmaintainable, and we think that the overall outcome for users in terms of the reliability of Bubble’s platform would be worse rather than better.

Currently, our tools for having truly different versions of the Bubble codebase are the Scheduled cluster and Dedicated clusters. All apps on the Immediate cluster are on the same underlying commit, all apps on the Scheduled cluster are on the same commit, and then each Dedicated cluster is on its own commit.

This bug did in fact make it to the Scheduled cluster, because the number of users it impacted was small enough that we didn’t realize there was a regression until the code had already been shipped to Scheduled. Generally, when we push a regression like this, we get a spike in bug reports that we can flag as a drop-everything-and-fix event; this time around, unfortunately, the bug reports trickled in more gradually and the team didn’t realize what had happened til the code had already moved to Scheduled.

That’s the current state of affairs. We know this is not ideal, but we want to be transparent with you all about what our capabilities are at the moment. We’re working on making our platform more reliable from several different angles:

  • On the infrastructure side, we are doing some deep investments that eventually should give us the capability to potentially have every single Bubble app on its own version of the codebase, so that users have full control about when they upgrade. I can’t give an ETA on this, because this is a major workstream and there’s still a bunch of technical details we’re sorting through, but I could see it potentially happening in 2023.

  • We have a culture around building new automated tests each time a regression ships, to gradually close off gaps in our test coverage so that events like this can’t happen again. While there are always going to be corner cases, our goal is to make it so that instead of bugs that affect 10% or 1% of our apps, the only bugs getting past tests are ones that affect 0.1% or 0.001% of our apps, making incidents like this affecting you personally becoming very rare.

  • We are also in the process of converting our codebase to Typescript, which unlike Javascript is a strongly-typed language. This is a technical measure that will knock out huge categories of potential bugs. It won’t necessarily fix everything (and I think this particular incident likely would not have been prevented by Typescript), but again, it pushes us towards a world where the frequency at which any individual app gets broken gets rarer and rarer.

  • Our Success team is continuing to improve their processes around identifying regressions and escalating them to engineering, with the goal of continually increasing our ability to sort out “we just pushed code that broke something” from “user stumbled over a long-standing bug”, so that we can triage effectively. Our overall response times have been trending steadily downward, though in this case we didn’t identify this as a regression requiring immediate response as fast as we would have liked to

  • We recently added the capability to roll things out as experimental features, and plan to rely on that more heavily going forward as a way to test things out in a lower-stakes way

  • I do want to remind people that for scaling business, you can contact sales to discuss moving to Dedicated. I realize that the price point for this is a deal-breaker for a lot of you, so isn’t a short term fix, but I’m putting it out there because it means that as your business gets increasingly successful on Bubble, there’s a path forward to stay on the platform even as your reliability needs increase.

I hope this provides a bit more transparency into how we think about incidents like this. Again, I know how frustrating this is, especially because it feels out of your control. We can’t guarantee 100% uptime – no tech platform reaches that level of reliability – but we’re aiming to keep pushing our uptime levels higher and higher for all of our users.

13 Likes

Thanks for explaining. Even cloudflare, mongodb, supabase and other have bugs all the time.

I am impressed at the fast fixes and support after things go wrong. Keep it up!

Thanks for the detailed explanation and level of transparency given @josh!

Many thanks, great transparency. This makes the process much clearer and also it’s reassuring knowing that a lot of thought has gone into this issue, both in how to best navigate it short term and also working towards longer-term improvements.

@josh first, I really appreciate the thorough response. Moving forward, I realize it would be a bit of work, but to get to the point where Bubble is seen as a reliable platform to run your business on that isn’t going to push potentially-breaking changes without our input, what do you think about this:

Any and all changes to the main cluster, however small, get pushed to development apps with a notification/email to Bubble users outlining the changes. You give us a week to check out the new version, make adjustments if necessary, and/or report new bugs. then the version gets applied to live apps. The bug in our case was quite obvious, and it was regarding a nested repeating group with 2 levels (parent and child list), but nothing else fancy going on. I’m sure we would have noticed the bug and reported it before it affected our live users if given the chance.

Our app isn’t big yet, by any means, but we are growing at a solid pace and I think getting Bubble to a place where we can regard it as a reliable platform to run our business on long-term vs. a great place to build an MVP means being more careful and communicative about changes. In my opinion, Bubble is absolutely the strongest no-code/low-code platform on the market by a large margin, but our team is constantly weighing whether to plan on using Bubble long-term or rebuild our app custom. If Bubble had a firm policy about never making changes to live apps, I know we (and likely many others) would feel a lot more confident about using Bubble long-term. I know a change like that wouldn’t happen overnight, but I’d love to hear that you’re moving that direction.

2 Likes

@samnichols I agree with both your response here and with @josh’s response above. You have to keep in mind that even if you were running on a custom stack, code regression is a fact of life and unexpected bugs happen every now and then. We run lots of custom apps (off-Bubble) and this kind of stuff happens here and there (even with lots of code and user testing). We are humans and that means it is absolutely impossible to get it right every single deployment.

The important part of this equation is the frequency of disruption (which @josh addressed above as improving) and the quality of communication. I feel like it would be beneficial to at least be aware of deployments that have happened/are happening. Some sort of status page or commit list for those who are interested in the minor changes/technicalities of Bubble as you and I might be. I would guess however that most low-code app builders are just happy to have a platform that allows them to generate revenue, even if from time to time, it is slightly unstable.

1 Like

Yes! The direction we’re moving in is to have the capability to deploy changes to apps much more granularly, instead of to all of them at once. We’d probably use that technical capability to move to a model where we let app creators opt-in to changes, rather than a process like you’re describing where everyone is on the same week-long cadence, but we are definitely aligned that giving users more control and visibility over changes would do a lot to build trust.

We actually have this! Go to Releases | Bubble and click on the “Releases” button

2 Likes

@josh Just to be clear, every deployment that affects the scheduled cluster is listed on this page without exception?

This tracks releases to the Immediate environment, so the time stamps are going to be misaligned with when the code reaches Scheduled, which is typically 9 am ET for the previous day’s code. There are also some deployments that we make to infrastructure services that are not listed here. That said, I think at least 95% of deployments that could cause behavior changes in user applications make it to this page.

3 Likes