Sudden WU Spike - My Customer is Being Overcharged

Efe · 2024-09-10T20:11:47.561Z

For those interested, the root of the issue was in the fetching data actions within the child elements of a reusable element called “Screen - Analytics.”

This reusable element had been consuming a normal level of WU (daily 5-20k) for months, but four days ago, it suddenly started consuming 20x more WU (reached up to 600k). As I mentioned in my previous reply below, I removed this element from all pages.

Sudden WU Spike - My Customer is Being Overcharged Bugs

My mistake. I accidentally shared the “Dev” version on the Workload Usage specific graph. Thank you for pointing that out. I wish the situation were as simple as you mentioned, but unfortunately, it’s not. Here’s the graph for the “live” version. [image] As you can see, 98.6% of it is from Fetching Data processes. Not from a scheduled workflow. [image] Scheduled Workflows, on the other hand, consume far less WU. My recursive workflow simply makes a basic API call. It’s unrelated to my cu…

However, I noticed that despite being removed from all pages, it was still consuming WU. At this point, I suspect that somehow, even though this reusable element wasn’t visible or existing anywhere in the app, it was still triggering the data fetching process. But I have no idea how that was happening.

I then deleted the reusable element from the app entirely, and after doing so, WU consumption returned to normal.

image1869×742 88.3 KB

Although WU consumption is back to normal, this doesn’t mean the issue is fully resolved because I had to remove a crucial feature from my app. For that reason, I will continue to work closely with Bubble Support to get further assistance.

So my money would definitely be on a recursive workflow. @adamhholmes

I think you might want to reconsider that! (Just joking, I absolutely see your point.)

Thanks again, everyone!

georgecollier · 2024-09-10T20:17:20.740Z

Efe:

I suspect that somehow, even though this reusable element wasn’t visible or existing anywhere in the app, it was still triggering the data fetching process. But I have no idea how that was happening.

Then it was definitely still somewhere in your app (perhaps you hadn’t deployed the change that really removed it from everywhere).

We’re trying to help but have very little to go on, so a recursive workflow is the most likely culprit, followed by a Do every X seconds, followed by a nested search, or some combination of all three.

Efe:

I’ve been professionally developing Bubble apps for over 5 years

For what it’s worth, years of experience doesn’t really count for anything on Bubble. There are people around that have been doing Bubble for six months that are better than people who’ve been around since 2014!

Efe:

the root of the issue was in the fetching data actions within the child elements of a reusable element called “Screen - Analytics.”

So what expression were you using? Were you using display data / display list actions anywhere on your page? It evidently caused 40,000 searches in one hour alone, so something must’ve been triggering that.

fede.bubble · 2024-09-10T23:07:28.515Z

Have you connected with the support team yet?

randomanon · 2024-09-10T23:17:03.769Z

georgecollier:

So what expression were you using?

How has this not been answered by him this deep into the thread?

boston85719 · 2024-09-11T04:50:49.379Z

Efe:

I then deleted the reusable element from the app entirely, and after doing so, WU consumption returned to normal.

While reading through all the posts in this thread, this was going to be my suggestion. Glad you got to it.

The reason for it to have been my suggestion is that I have seen a bug in WUs that has still not been addressed by Bubble. I had a backend workflow, consuming a small amount of WUs which totaled around 1.8 WUs per day for an API call that was not necessary. I deleted the backend workflow, yet it still shows as running and consuming WUs every day. This is in development and when I look at the WUs it is in the Development.

This seems to be a bug that is causing things to be out of sync with the reality of the app setup. Unfortunately, I have not taken the time to investigate whether an app optimization run will actually delete the backend workflow or not in a similar fashion it does for reusable elements, data types and fields etc.

Screen Shot 2024-09-11 at 11.44.58 AM1371×860 61.4 KB

Screen Shot 2024-09-11 at 11.44.44 AM869×511 28.3 KB

Again, this is a known bug that I have raised and gotten no resolution to, and I believe recalling other users reporting via forum of this bug previously.

Likely, the reason for the usage spike, obviously the fact of the bug is the real case, but why such a high usage, is because since deleting the reusable element, the source of the data search was likely unconstrained and therefore searching and returning all entries (as support pointed out) because despite having been deleted, it likely was not deleted from the page in the app it was originally placed on, so the ‘element not found’ notice was likely not visible, as the reusable element itself was placed either in a group not visible or a popup (so on preview can not see the ‘element not found’)…also, probably an issue with the issue tracker not picking up things as consistently as it should, so the ‘app wide’ alert of the deleted reusable element not found on a page was not shown in editor.

Please make sure you report this as a bug to support. I will be having to report again about my ‘deleted workflow’ still running and getting charged. The more users reporting about the buggy behavior around WUs the better and faster it will start to function appropriately all the time.

georgecollier · 2024-09-11T06:48:32.403Z

randomanon:

How has this not been answered by him this deep into the thread?

Probably because the answer is in there! I hope OP reports back whether it’s a bug or not with details of what actually caused it.

Efe · 2024-09-11T07:11:09.060Z

Sudden WU Spike - My Customer is Being Overcharged Bugs

Have you connected with the support team yet?

Yes, 5 days ago and it’s still under investigation.

boston85719:

The reason for it to have been my suggestion is that I have seen a bug in WUs that has still not been addressed by Bubble.

I’m sorry you’re experiencing this. Creating a bug report and getting results is really a tedious thing. I waste a lot of time trying to explain my problem in full detail. It’s also very disappointing that it’s not resolved.

boston85719:

This seems to be a bug that is causing things to be out of sync with the reality of the app setup. Unfortunately, I have not taken the time to investigate whether an app optimization run will actually delete the backend workflow or not in a similar fashion it does for reusable elements, data types and fields etc.

There must be something wrong here. One of the issues we often face as a team is losing parts of our work during merges. We’ve encountered this many times. For example, we have a page in our app that was updated weeks ago in terms of UI. But when we merge with a newly created branch, the page somehow reverts to its old UI. The page with the old UI doesn’t exist in either the main branch or the merged branch. There seems to be a synchronization issue somewhere.

boston85719:

Again, this is a known bug that I have raised and gotten no resolution to, and I believe recalling other users reporting via forum of this bug previously.

This is exactly why I created this forum post. I’m sure there are other developers suffering from WU bugs, and maybe they are experiencing very similar problems to mine. Contacting Bug Support alone doesn’t seem to enough. I hoped that this forum post could speed up the resolution process for these bugs, and you’ve contributed greatly to this topic. I’m grateful for that.

boston85719:

Likely, the reason for the usage spike, obviously the fact of the bug is the real case, but why such a high usage, is because since deleting the reusable element, the source of the data search was likely unconstrained and therefore searching and returning all entries (as support pointed out) because despite having been deleted, it likely was not deleted from the page in the app it was originally placed on, so the ‘element not found’ notice was likely not visible, as the reusable element itself was placed either in a group not visible or a popup (so on preview can not see the ‘element not found’)…also, probably an issue with the issue tracker not picking up things as consistently as it should, so the ‘app wide’ alert of the deleted reusable element not found on a page was not shown in editor.

I’m a bit confused on this point. My reusable element was only being in one place. The reason I made it a reusable element was to manage it more easily in the editor. The strange part is that this element had been there for months, yet suddenly it started consuming huge amounts of WU. Normally, if an element isn’t drawn on a page (when not visible), it shouldn’t fetch data, as we know. But in this case, I think a bug may have occurred about this thing.

boston85719:

Please make sure you report this as a bug to support.

I’m already in contact with Bubble Support. They are still investigating, but the process is very slow. In the meantime, I’ve already deleted and deployed the buggy reusable element. I hope they don’t respond with “we can no longer investigate because this element is not exist anymore” because I couldn’t sit while losing money, waiting for them to investigate.

Thank you for sharing your thoughts, @boston85719.

I hope we get to see the time when serious things like WU consumption are bug-free.

sem · 2024-09-11T07:49:05.144Z

@Efe

Although I do not have a solution obviously, I can say that we have the same experience as you. In any case we can not fully trust Bubble all the time when merging. We always need to double check. 95% of the time it goes fine. Those 5% where something is not working as it should can hit you hard.

I think a lot of users do not experience this either because their App is not heavily used in the real world or the owner is not measuring lots of things around user behavior, error logging and such so they simply do not see it.

Good to know that we also had issues with undo. We saw weird behavior with undo where after undo not all steps of a workflow were added back to the app. Eventually, after weeks, Bubble support said that they could reproduce the issue and that its behavior can occur in various situations. Since there was no easy fix, they advised only to undo small steps and check.

These small things can add up if you are not familiar with it.

I think the best approach is to make small changes and for each small change save it. If something weird happens you can at least go back to a state that you know.

As for WU’s, I am very happy that we decided to move data processing to Supabase. We started this transition when we got notified of the price change. I feel sorry for everybody still running heavy data processing on Bubble or backend workflows that go haywire.

eren · 2024-09-11T08:11:28.212Z

Hi all,

I believe there are also bugs in WU consumption and data fetching, too… I won’t go into all the metrics now, as it would only cause confusion. Instead, I’ll share the key points I found important within a specific date range.

For those who don’t want to read the entire post, here’s our observation: Bubble’s hidden elements consume data. At least for today, this is the case in my application.

Last 30 Days - 12 Agu to 11 Sep

image1587×747 47.7 KB

You can see the increase, which started in the evening hours of the 5th.

image1563×727 43.8 KB

On this evening, API Workflows were initially responsible for WU consumption. Then, “fetching data” took the lead with a significant increase.

image1873×901 71.9 KB

Now let’s go to the peak time of workload on the same day.

image1648×771 61.4 KB

Upon closer examination, I noticed that the “search” part was causing this.

image1647×771 46.7 KB

I investigated the element causing this issue. The only way this could happen was if the relevant conditions were met, leading to a significant increase in page loads.

Here is the expression:

Expression part 1: It checks if the page is working under a certain domain or vice versa.
Expression part 2: It checks a feature on data directly assigned to a group. If it is active, it searches for an LLM type that matches certain data and becomes visible. This search returns just over 10,000 results.

image866×834 79.9 KB

Is this possible? Yes! If this condition holds and the number of page views increases significantly, my app’s WU consumption will undoubtedly increase.

Let’s conclude by examining the page metrics. I pulled 30 days of data and analyzed it. When my WU usage was overloaded—and it still is—sometimes the page views were even lower.

image1690×634 63.5 KB

After these investigations, we concluded it was a bug and created a report. Note how the dates align closely with when @Efe started experiencing similar issues.

My teammate @Batuhan Merguz, who was investigating this problem yesterday, understood the situation. Once we realized this, the alarming part was that every user interacting with the chatbot on our client’s website was causing the chatbots on other users’ pages to also fetch data!

For instance, if this data stream consumes 10 WU, and 10 users interact with the chatbot simultaneously, it becomes 100 WU for each message. As long as they continue talking, it increases exponentially. If new users join, the situation results in an exponential increase in WU consumption, leading to the explosion in usage we observed.

The repeating group was fetching data even though it was not visible .

Here’s what we tried to resolve the issue:

1- We tried to figure out what went wrong by deleting the condition piece by piece, nope, it kept repeating in every case.
2- We unchecked the condition “visible”. It still keeps fetching data.
3- We completely hid the repeating group, it keeps fetching data.
4- We deleted the repeating group and it stopped fetching data.

We are waiting for the Bubble team to begin investigating, and we are still pulling WU like crazy.

Note: Efe is also my teammate, but he is talking about a different app.

georgecollier · 2024-09-11T08:17:46.630Z

eren:

it searches for an LLM type that matches certain data and becomes visible. This search returns just over 10,000 results.

150 WU in one search

eren:

The repeating group was fetching data even though it was not visible .

eren:

3- We completely hid the repeating group, it keeps fetching data.

Don’t know whether this is the root cause of the issue, but the idea that an invisible repeating group should not load data is a misconception. Making an RG non-visible does not mean that the data will not be fetched. Often, that will be the result, but as soon as you reference the hidden RG from anywhere else in a way that requires the RG’s data, the RG’s data will be fetched. This generally happens when you reference it somewhere that was visible (or some condition in another visible element required data from that RG).

So, for this:

eren:

here’s our observation: Bubble’s hidden elements consume data.

The answer is yes, they do in some cases, and that’s what’s always been the expected behaviour.

Whatever it is, I hope you’re able to get it fixed as nobody wants to be billed loads of WU if they’re not supposed to be. If it’s a bug then let us know if it gets confirmed, and if not… best leave this up as an example!

Efe · 2024-09-11T08:34:50.850Z

sem:

Those 5% where something is not working as it should can hit you hard.

Well, it definitely hits hard.

sem:

I think a lot of users do not experience this either because their App is not heavily used in the real world

I totally agree. We often encounter weird issues and I wonder, “If Bubble is so buggy, why isn’t anyone talking about it?” Then I thought, maybe there are just really a few heavily used apps.

sem:

Good to know that we also had issues with undo.

We have the same.

sem:

I think the best approach is to make small changes and for each small change save it. If something weird happens you can at least go back to a state that you know.

Isn’t that super exhausting? I mean, we’re using Bubble to move fast, but these kinds of situations really make me feel disappointed.

sem:

As for WU’s, I am very happy that we decided to move data processing to Supabase. We started this transition when we got notified of the price change. I feel sorry for everybody still running heavy data processing on Bubble or backend workflows that go haywire.

I’ve also thought about moving to a third-party service for database management, but I haven’t had the courage to start yet. For heavy apps like ours, moving all the data is a big step. When you decided to go with Supabase, did you consider any other alternatives? If so, what made you choose Supabase?

Thank you for your reply,
Best

eren · 2024-09-11T08:37:38.002Z

georgecollier:

150 WU in one search

How did you find this figure?

georgecollier:

Don’t know whether this is the root cause of the issue, but the idea that an invisible repeating group should not load data is a misconception. Making an RG non-visible does not mean that the data will not be fetched. Often, that will be the result, but as soon as you reference the hidden RG from anywhere else in a way that requires the RG’s data, the RG’s data will be fetched. This generally happens when you reference it somewhere that was visible (or some condition in another visible element required data from that RG).

So, for this:

georgecollier:

The answer is yes, they do in some cases, and that’s what’s always been the expected behaviour.

We have already checked the possibilities that I have not referred to elsewhere, those that are mentioned here, and those that are not mentioned here at all. In short, I hope it is understood that I am talking about retrieving data from an element that is not referenced anywhere. Otherwise, when writing a post, we would have to give examples from the entire Bubble manual, which would make it impossible to write a post.

A little advice: Please do not always feel obliged to give advice. There are better ways to provide guidance, such as blogging or appearing on different social platforms.

randomanon · 2024-09-11T09:12:27.786Z

eren:

2- We unchecked the condition “visible”. It still keeps fetching data.
3- We completely hid the repeating group, it keeps fetching data.
4- We deleted the repeating group and it stopped fetching data.

The only time visibility matters for data is in a Popup.

boston85719 · 2024-09-11T10:21:17.905Z

@eren best practice for hidden RG to not load unless necessary is to leave main data source empty. Put conditional onto RG for when data is needed and property to change is data source, and that is where you place search.

ihsanzainal84 · 2024-09-11T17:02:29.292Z

This is why 98% of the time I avoid putting Searches as data sources. I’ve experienced similar issues in the past, when WU first launched and I was learning to optimize for it.

Instead I will load data in a workflow to store into states. To keep things updated I have an expression somewhere in a group or a plugin that will retrieve and store the latest item. This will be a Search but it returns a single item instead. I’ll have some workflow that will detect if the latest item in the list state with that single item does not match then add the latest item into the list. A plugin like Floppy with it’s Expression Watcher has built in actions to trigger events when a value changes so that makes it easier to manage.

I’ve not had any unexpected behaviours in my apps with this approach since I have full control over when I want to load data.

sem · 2024-09-11T18:25:26.567Z

I think that it is exhausting but also that in many cases it is still for many users the best bet. We should not forget that managing complex hard and software ourselves also can mean problems. The feeling is different of course in that case. On the other hand, NoCode and AI is moving fast so more and more other options are very viable as replacement.

As for Supabase, we wanted a large community, backed by serious entrepreneurs/investors, open source, proven technology, full ownership, managed, good pricing structure. Honestly, there are not much other options and Supabase does many things very very good. And for us, it is golden that we can use Postgres as it is meant to be. Oh boy…what a power and what a speed.

We started with a few tables. Db triggers when something changes in Bubble and updated Supabase. Now almost everything is in Supabase. Planning to have data and business rules as much as possible out of Bubble. It will mean we can scale easier at much lower price points and are less vulnerable to any of the companies we do business with

eren · 2024-09-12T06:52:19.353Z

Thank you for your input.

As you suggested, there are dozens of ways to accomplish things. On this page, there are already elements that should hold a single item. Normally, that RG should already fetch the list. Of course, it should do this when it becomes visible.

For us, this is not just a process related to WU. For years, we have been building applications for our own projects and for our clients. In applications with many page loads or workflows, whether WU is present or not, we always had to optimize for high performance. I come from the days when Bubble was very slow, and we had to jump through hoops to show some things correctly(for UX, etc.). In many other applications, the way I used above has been used and is still being used. This usage has not caused any issues either from a WU perspective or a performance perspective.

For example, the most suitable usage to edit the current situation seems to be fetching data with a GET API. This way, Bubble will not make a new call without fully refreshing the page or straining it.

However, we are not applying these optimization techniques that come to mind. We are waiting for news from the Bubble team. Because we believe this is a bug related to data fetching and is important for the sustainability of the platform. Even if it is not a bug, we need to understand which behavior has changed.

You will see in the above metrics that the page load numbers fluctuate between 25K-50K. And WU consumption started to spike after a certain date. Of course, saying we made no changes doesn’t mean much, but it is an old feature, so we know we didn’t touch “that part” and there are no references to it in our release notes. As a less valid proof, I can show the date of Aicado’s GPT’s PH launch, and if you read the post, you will see that our “history” feature has been there at least since that day. We also have an old branch to show to the Bubble team.

image1179×230 18.2 KB

We also noticed a release note that aligns with the start date of overconsumption.

image1736×568 21.8 KB

That’s why I created a post here. Maybe there are people who are going through the same thing as us, and we thought the Bubble team could make it easier to clarify.

georgecollier · 2024-09-12T06:58:58.888Z

eren:

We also noticed a release note that aligns with the start date of overconsumption.

Do your search expressions use unique ID or :each item’s Thing in their expressions?

georgecollier · 2024-09-17T08:14:15.903Z

So, what was the bug? Has it been fixed? Or was it something in this particular app?

Efe · 2024-09-17T09:58:27.929Z

Yes, at first they thought it was a problem with the recursive workflow. Fortunately, they did not insist on it. They said that they started receiving other bug reports about this issue. As a result, the technical team confirmed that it was a bug. They are currently investigating further. I think this post encouraged other developers who had the same problems to create a bug report. So thanks to everyone who contributed to this post.