Stop pushing Bubble updates without user consent

Idea
1

I’ve come to realize that Bubble pushes minor updates to the engine and editor on a daily basis. These minor updates are not part of the scheduled Bubble updates and it seems users have no option to opt out of these pushes. The issue here lies in the constant bugs that we face on our live applications and our editors due to these minor pushes without any control.

This needs to change for the stability of the Bubble ecosystem.

An “opt out” option needs to be implemented so that users can have control over their current editor and live builds. Even a weekly “minor release” that users can opt out of would be helpful to stabilize the ecosystem rather than having daily pushes.

I don’t know what the QA process is on the Bubble side of things, and I imagine that these minor pushes help them to identify bugs. If this is the case, it is unacceptable. Our live apps are mission critical and the editor is critical for product development. We are paying and trusting that Bubble will be stable. I don’t know how this can be quickly remediated, but it’s something that’s been on my mind and something I believe will significantly improve the Bubble experience.

I’d like to close by saying that I greatly appreciate the Bubble team, community, and platform. It’s a special ecosystem that I want to see thrive and I am posting this from a place of love, not anger.

Much love to all and happy Bubbling,
Gal

16 Likes
2

Minor updates are included even if you choose not to be on Immediate Releases?

Screen Shot 2021-10-08 at 3.38.55 PM
Screen Shot 2021-10-08 at 3.38.55 PM759×75 5.63 KB

1 Like
4

From my observations and the implications I’ve received via Bubble support over the past couple of months, yes.

7

I thought releases aren’t made till the next day if you’re on scheduled releases?

8

I’m not certain. Even if that is the case, releases should not be released daily. They need to be on a schedule and updated based on user consent. That is why you don’t see companies like Adobe or Microsoft pushing out daily minor updates. It causes an unstable platform with consistent bugs in production. Bubble can package minor updates into bi-monthly OPTIONAL updates for users to choose from.

I get HP updates that are “CRITICAL” but I still get the option to choose… same with Apple. It’s paramount to have the choice for stability’s sake.

9

+1 we need this

2 Likes
10

+1 : My app is not a hobby, it’s how I serve my customers. Need stability.

3 Likes
11

+1. I found 2 more major issues where alignment settings had simply changed! My app was basically unusable. We are well beyond the fascination age of apps where people used to be more forgiving and “hang on” to give the app time to mature. These days, if the app doesn’t work people just leave. We can’t have the software we are building change beneath us without going through some sort of opt-in release process. We need improved stability and reliability here!

3 Likes
12

Agree. I have heard Bubble has hired new people, I hope they can hire some testers to test their engine out of production.

Otherwise it seems that we and our users are in the group of free testers all the time.

1 Like
13

Hi all, couple thoughts on this.

First of all, we very much realize that Bubble is a mission-critical piece of infrastructure for a lot of our users, and that when we ship buggy code, it can be anywhere from a major inconvenience to a disaster for our customer. We feel really bad when this happens, and generally scramble to get a fix out. This last couple weeks we’ve seen a couple of mistakes on our end – we’re very sorry about that.

The primary defense mechanism against this kind of issue is our QA and automated tests. All new features are tested by hand, and we run all new code through a suite of automated tests that try as many common scenarios we can think of, including scenarios we generated as a result of past bugs that have slipped to production. That said, our test coverage isn’t as good as we’d like it to be; we have a lot of corner-case feature interactions where one Bubble feature can cause bugs interacting with another Bubble feature, and we don’t have coverage for all of them. We just engaged a QA contracting firm to help expand our test coverage. We’re paying for the equivalent of a team of five full-time testers, which might expand if the initial trial goes well, and they’re currently getting up to speed on our product and starting building tests. QA is one of the areas we think we can throw money at to get improvements after our recent fundraising round, so we’re crossing our fingers this test effort goes well.

In terms of release schedules, to clarify how this works:

  • For customers on free or shared hosting plans (ie, not dedicated), we have two clusters of servers; the main cluster, and the scheduled release cluster. On those clusters, we can’t deploy code to apps individually; deploying code to one of the clusters updates every app hosted on that cluster.
  • For the main cluster, we release code as soon we’re done testing it. You can see a log of those releases on the “Releases” tab of this page: Releases | Bubble
  • Once code has been on the main cluster for 12 hours without us doing another release or seeing evidence of new bugs (generally overnight), we mark that code as “stable”. We do sometimes see bugs in stable code, but they tend to be fairly subtle and / or impact only a tiny fraction of apps; more visible bugs generally get noticed by the team.
  • The scheduled cluster contains apps where users have selected to be on the “Scheduled releases” tier, which is an option for apps on our Professional plans and above. The scheduled release tier gets updated 9am ET every day that there is new stable code. So if we don’t mark our code as stable for a few days, there may be a few days in between releases, or if deploying is going smoothly and there aren’t any bugs, we might release code every day.
  • Apps on dedicated plans have the option to upgrade their cluster to the latest stable code once it becomes available. Currently, dedicated apps can choose not to upgrade for extended periods of time, although in the future we may put a limit on how out of date a dedicated cluster can get because we’ve seen a number of issues caused by apps being on very old versions of code.

So to summarize:

  • Our primary mechanism to keep the platform stable is testing. Our testing isn’t as thorough as we would like it to be, and we’re currently investing more money in fixing that
  • As an additional layer of protection, which we hope becomes unnecessary over time, apps can pay to be on the scheduled or dedicated tiers, which only get code that has been in production without us noticing issues with it for a period of time.

We may tweak these release schedules in the future, but we’re more excited about improving this by hiring testers, because eventually, if we don’t find bugs in our human or automated testing, they’ll make it to end users no matter how we structure our releases. Either we find them, or you find them – there’s not really another option.

We also do think it’s important to keep shipping code at a steady cadence, for a few different reasons. There are certain times we have to roll out code: we find a security issue, a web browser makes a change that breaks apps, we get hit by a new form of DDoS attack or encounter unexpected user behavior that causes issues with our systems, or a third party provider we use deprecates a feature or makes a change on their end. Moreover, our community has the expectation that we keep investing in and improving the platform: part of the bet of building your business on top of us is stability, but the other part of the bet is that Bubble is going to keep evolving and improving and the ecosystem will keep growing. One of the reactions to my most recent community update was that the rate of features we’ve shipped over the last two years has been disappointing, which is feedback we agree with and are working to change. Finally, from a change management point of view, frequent small releases tend to be safer than infrequent, massive releases: when something goes wrong during a small release, it’s usually easy to pinpoint the problem and roll back or fix-forward, whereas for massive releases where a lot is changing at once, it’s harder to diagnose issues and the likelihood of unintended consequences from multiple changes interacting with each other in bad ways is much higher. For all those reasons, we do think it’s important keep shipping code on a regular basis, but we very much agree that we need to improve the quality of those releases.

21 Likes
14

Thank you Josh for the thorough and transparent reply. I need a bit of time to digest the process to give a thorough and well thought-out response. I still firmly believe that daily releases are not the answer and will need to consider the above to understand how this affects my own users.

2 Likes
15

Yes, thanks Josh. Really appreciate your reply and attention to this issue. Investigating proven release models in use by other infra providers might also yield some improvements to the process. I don’t think there’s anything Bubble-specific to this issue.

16

I have 25 years in the enterprise dynamics application space. This is THE most difficult challenge. Striking the right balance between a SaaS based app, where the authors have no say on new code releases, to PaaS based apps, where the authors are responsible for all of their own updates including security patches.

I can appreciate the difficulty of the challenge for all Low-Code vendors as they want to deliver the ability for authors to change code on their own without the need for technical concepts like GitPush and GitCompare which are not low-code, low-technical skill concepts.

The underlying platform code will ALWAYS change when you have a site based on dynamics and logic operators. So there will always be a level of maintenance required due to the reasons stated. (Security patches to underlying technologies, warding off bad actors etc. )

These can be solved with the locked down SaaS approach, but then authors lose flexibility in core logic development and how applications can be easily modified to fit any business case or desired user experience.

From what I can tell, Bubble is making a wonderful attempt to put a UI overtop of Serverless JavaScript and combine that nicely with UI design logic. Very noble cause but not an easy task and not something that has been solved by large companies like Microsoft, Adobe or SalesForce.com either. That is why there is still heated discussions for SaaS versus PaaS at the CTO level of most modern enterprises and enterprise vendors have huge R&D teams dedicated to low-code application orchestration.

So kudos to the Bubble Team for their efforts at the code and flexibility level.

Reverse Kudos for how this solution handles single app, multiple viewport conditional design. Really falling short here and in the end, hard for me to take it seriously until they resolve this element of the solution. Too bad because the rest of it is really stellar for what it is trying to uniquely do, but experience design and rendering device compatibility is just as important as business logic and CRUD operations in a visual UI. Without both working seamlessly, then you just have another decoupled solution with some kind of headless solution doing rendering and something else doing business logic and data persistence. This just means lots of technical delivery which is what LowCode is designed to abstract.

Keep at it Bubble, but I am personally wishing for way more effort on device level conditional logic for the UI that is visually defined. This should go down to the most granular levels of CSS styling declarations. Systems as old as DotNetNuke had the notion of element level themes to help address conditional UI and that is at least 20 years ago. Most leading CMS engines have the same component level conditional theming that is very granular.

I hope this little commentary helps in some way.

Cheers all and be safe and well in these trying times people and remember it is one planet and one life. Embrace both as much as possible with an eye on diversity and mutual respect.

4 Likes
17

Thank you for the update, Josh but why 12 hours? I think 72 may be the minimum viable time for the Free Tier users to start contacting your support. Especially taking into account that most people on Free Tier are not professionals and their apps are not complex enough.

1 Like
18

Do you testers have a copy of real complex apps for testing? That would be the best option like having copy of 50 of most complex apps built with Bubble instead of purified tests.

1 Like
19

Thank you Josh for the thoughtful response. I agree that there are indeed two elements in the Bubble “bet”. I would propose that stability should take priority over evolving and improving.

Meaning: if the platform is not stable, my customers will leave. If my customers leave, I will not be around to experience the evolution and improvements.

2 Likes
20

Not a new topic. It’s been my recommendation for a while that, while Bubble inherently CANNOT get away from evolving their codebase continuously (think Apple iOS updates that break apps and force them to constantly regression test), they SHOULD get on a cadence schedule like most enterprises so that at a minimum Bubble devs may know generally when they need to regression test.

On dedicated, users have MUCH more control over the regression testing cadence, however the required frequency to keep the box updated and avoid issues is still very fast, and much of resource’s dev time is spent endlessly regression testing instead of enhancing apps. I’ve sent 1 or 2 recommendations to Bubble on how to solve this for growing enterprises sitting on dedicated clusters, which I think will be very critical for apps that cross over the line of MVP into success stories, whom serve tens of thousands of users… and that have a decision to remain on Bubble permanently or seek conventional stack rebuilds.

Bubble has a hard line to toe here: please MVP builders, where Bubble’s fledging capabilities are key, and please apps entering productionalized rigor, where stability and version control of the code baseline is key.

1 Like
21

Just wanted to give a part 2 update on my experience with this issue. It would be one thing if some fields suddenly weren’t aligned and the under-the-hood updates simply caused a few look and feel inconsistencies. But this has been far more than that. I still have bugs coming in from my beta group where elements aren’t even displaying. I use a lot of responsive settings and MANY of them have changed.

Based on some of the great posts in this thread, I think I need to be tolerant of SOME level of impact to my app in exchange for the constant improvement to the platform. But I would greatly appreciate getting a process in place that protects against sev-1 level impact to our user bases. Perhaps the beefed up internal QA team can focus their efforts there first. Specifically, all conditionals and responsive settings need to be rigorously tested to make sure elements still display.

1 Like
22

Just today I’m seeing inconsistent rendering in my app… No changes made on my end.

23

@gbenchanoch, what kind of inconsistencies? Any example pics you can share?