[New Feature] Workflow Performance Improvements

I have no idea, I noticed a drastical improvement on a Personal plan. I guess things only keep getting better…

1 Like

Update: yesterday, we noticed that this change was breaking certain actions. As a few of you guessed in this forum thread, the issue was with actions that rely on database updates performed by previous actions. When actions were batched together, the database updates didn’t write to the database in time for the actions that relied on the database updates to see a change.

I’ve temporarily disabled batching (so the performance improvements are sadly reverted for the time being) while we investigate this issue. This should fix the issue.

3 Likes

Following up on my last reply to this thread: we’ve determined a fix to the problem that also preserves the performance gains in most cases. We’re going to hold off on the deploy until January just to be safe. Everything should be working fine now. Sorry for the trouble this caused and thanks to everyone who posted about the issue in this thread; the insights here really helped us to track down the bug!

6 Likes

Thank you! Great work.

Thank you sir! For our app, we mitigated a lot of the issue with API-chaining everytime we needed to ensure a conditional gets re-evaluated. The performance boost was noticeable and appreciated.

A small cultural suggestion, if I may? Is it possible for Bubble team shoot out an email when any anticipated prod change is scheduled to go in place, like an advice or notification of maintenance 24 hours prior? In the same nature as the automated cluster alerts or DB downtime alerts.

This way app owners can be at the helm and ready to halt any rouge processes should something unanticipated break in prod. We had 3 prod users get emailed something like 700 times inside a previously-stable recursive API that altered its behavior before we realized what was occurring and ceased the process. Really no big deal, but if a recursive Stripe charging API was running or something, things could’ve been a bit more serious. Such a small shift in operating procedure could give prod continuity oversight a serious boost! Appreciate it to both you and team for continuously striving to improve performance.

13 Likes

Second this. There needs to be some better standards in place for updates for production apps.

Tough part on their end is they can’t predict the endless number of ways we setup our apps, so they might think it’s a minor update but when in reality it’s critical to some - making it hard to tell us to be ready because I think they push updates almost daily.

5 Likes

Hello @henryd,

So is it deployed already or you will be deploying it in January?

Thanks a lot.

Love the intent here but it seems to have broken something along with it.
All of a sudden doing “Create a new thing…” in a custom event in one of my apps has ceased to work.
I can see it in debug_mode but when I check the database in the app editor it is nowhere to be found. Has someone else experienced similar and know what to do about it?

@anders.eidergard does it happen to have a conditional (ie “Only When…”) tied to the action?

Yes, it does.
It’s checking a custom state value on the page.

I tried making it all into a single workflow instead of in a custom event and that worked.
The flow is the exact same with the only difference being that it now runs in a single workflow instead of part of it being in a custom event.
A bit more work since I can’t reuse that custom event and have to make changes in several places so that’s a bummer, but at least it now works for my users.

This is great! Thanks so much.

Hey @anon38627393, thanks for the reply and sorry for the late response on my end. I agree that we definitely could have done better in the way we rolled this out. Going forward, we’re planning on rolling out these workflow changes as a Bubble version, which should prevent all of the issues you mentioned in your reply happening. With a Bubble version, app owners are in control of when an update is rolled out so there’s no unexpected outages and (hopefully) no need for update alerts 24-hours prior to a major deployment. That said, if you have any suggestions about how this process could be better, we would love to hear them. We really appreciate your input and help with this whole process, and thanks for bearing with us as we improve our operational procedures.

3 Likes

Hey @gf_wolfer, I just responded to @anon38627393 but I want to agree with you and let you know that we’re definitely aware of this situation on the engineering side. We don’t want to have to bother users with every update we push, as we update pretty frequently, so we only release a change as a Bubble Version if we know that it’s going to break user’s apps. Here, we thought it wouldn’t break anything, but we didn’t realize that workflows that are structured in a certain way are in fact broken by these changes.

(If anyone’s curious, here’s what broke: workflows that have these two actions in the same set of consecutive database updates: 1. update the database 2. another database update that relies on an “Only if” conditional, which is constructed with a “Search for … : count” dynamic expression". This issue was resolved last week, so if you notice workflows of this kind, or any kind, not working, please submit a bug report!)

Going forward, we’re going to use the Bubble Version system for updates like these so that apps don’t unexpectedly break. We hope this will prevent issues like this from happening in the future.

3 Likes

Hi @ryanck, the workflow performance improvements are not currently in effect because we disabled them temporarily in order to fix a bug some users were having. The improvements will be deployed next week, so that if anything goes wrong we can quickly respond (most of the engineering team is off the the holidays right now).

1 Like

Hi @anders.eidergard, sorry that your app isn’t working, although I’m glad you were able to find a workaround. If you haven’t already, would you mind submitting a bug report? This will help us fix the bug.

Thank you much @henryd. The notion of a version increment solves this issue nicely. Appreciate your engagement here with the user base, very much. How about this thought: are the Bubble updates pushed out as specific times of the day? I recognize it’s probably both impossible to A) know if an update could potentially or not potentially break existing app structures and hence concretely know whether or not it should be a dedicated version increment, and B) to actually make every single update a version option. Hence, because we’re dealing with this rock and a hard place, perhaps if the high-frequency updates were put on scheduled release cadence, we could still anticipate potentials. AKA I know updates are pushed around 11AM EST every day, so I know prod ops needs to check health and logs at 11:10 every day.

Working for a particular large, and technically-disparate, large bank comes to mind here, where they managed hundreds of possibly-high-impacting prod updates per week. In events where testing the impacts of each one was impossible, the hard-rule was there was a set time and frequency for prod pushes (which was once and only once per week), so that all affiliates/departments/tech teams knew when to be on call and to watch for sudden defects and failures. The flip side of this is, if a dev missed the cutoff for his particular change, he had to wait for the next prod push round. No exceptions.

That sort of model could work here. Food for thought.
Appreciate you much.

EDIT: P.S. SEV1’s were always an exception. And the real cadence looked like all updates: once per week, always followed by a minor push 36 hours later of all the fixes that the weekly push broke haha. Emergency SEV2 fixes only, no additional dev, for that mini-push.

Maybe we can implement a simple Request For Comments system? During development/design, there can be a post made to the form that can solicit comments about how these changes may affect our apps. I understand if it can’t be made out on a public forum, but maybe even in a private one.

1 Like

Hi @henryd,

Is it already deployed?

Thanks a lot.

Hi @ryanck, we’re re-rolling out batching at the beginning of next week. When we do, we’ll roll it out as a bubble version, so you’ll have a chance to test the changes in development before rolling out to production to prevent further hitches.

6 Likes

FYI: just rolled this out. Took a bit longer than we had hoped because of some issues with testing, but we’re confident that the version we deployed today is stable. To enable batching, upgrade to Bubble version 11.

7 Likes