CRAZY 5000% workload usage increase from daily avg

jacob18 · 2024-07-19T03:22:36.331Z

A critical issue I experienced with the API Connector plugin in Bubble: While I was out of the office, our application made an excessive number of API calls, specifically 43,838 times in an hour, which caused us to exceed our monthly WU allocation in just one day.

Here are the details of the issue:

Issue Description:

Upon returning to the office, I discovered that our application had made 43,838 API calls in a single hour.
This unexpected behavior caused us to exceed our allocated Work Units (WU) for the month.

Initial Troubleshooting:

I immediately deleted the API Connector plugin, suspecting it might be the cause.
Despite removing the plugin, the API calls continued.
I then identified and deleted the relevant backend workflow, which finally stopped the calls.

Concerns and Questions:

Why did the application make 43,838 API calls in an hour? This number seems unusually specific. Does it indicate any specific error or pattern?
Logs Review Request: Could you please review the logs to determine what triggered such an excessive number of API calls?
Prevention Measures: What steps can I take to prevent this issue from recurring in the future?

animisha45 · 2024-07-19T03:51:59.460Z

Hi @jacob18 ,

jacob18:

I immediately deleted the API Connector plugin, suspecting it might be the cause.

API connector Plugin will not cause a surge in WUs by itself. The issue would lie in an action or a database search etc.

jacob18:

Why did the application make 43,838 API calls in an hour? This number seems unusually specific. Does it indicate any specific error or pattern?

I believe it could have gone up if you didn’t catch it on time. This is not a specific error on the application. I would say it had something to do with a condition that allowed the API to run that many times.

Have a look at the the logs in your application. It will help you find out and debug why the api kept running. It is possible that the call was recursive and some condition allowed it to not stop.

These kind of calls happen sometimes. Hope you are able to figure it out.

animisha45 · 2024-07-19T03:54:42.027Z

Additionally, have a look at the following for some preventative measures:

manual.bubble.io

Workload notifications | Bubble Docs

manual.bubble.io

Infinite recursion protection | Bubble Docs

If you implement these, it will make sure this does not happen again.

I would also recommend reaching out to bubble support if you are unable to figure out why this happened or for further clarification.

Jici · 2024-07-19T04:09:31.603Z

API can also be related to backend WF or Data API. Can you share screenshot of chart and your backend WF involved? It may be someone that used this open endpoint to “attack” your app…

jacob18 · 2024-07-19T04:30:59.290Z

hi @animisha45 thanks for the replies. Yes you’re right, i had a backend workflow to call the API but it is only supposed to run when a specific button element is clicked.

regarding the specificity of 43,838 API calls in an hour it was consistent throughout the middle 3 hours where i completely left it alone, with each hour calling 43,838 times. not more not less.

I am looking through the logs but it does not display anything, there is only a pie chart of what caused the spike which i already identified, but nothing else more than that. edit: sorry i just realised that i had to do a manual search for the logs, it will not display by default. Will look through it now.

I’ve contacted support yesterday but so far still no response.

jacob18 · 2024-07-19T04:36:51.821Z

@Jici unfortunately in a state of panic i deleted everything, because after removing the plugin, it was still calling. so i had to remove the back end WF and all elements that has connected actions.

However i do remember that an option in one of the editors : expose this API as public… was checked by default. Could this be a reason why?

The funny thing is in the call, there is a token included in the headers that expires in 1 hour. So it should throw a 401 unauthorized right?

jacob18 · 2024-07-19T04:49:02.517Z

image1832×644 27.3 KB

jacob18:

Will look through it now

this is the logs, when i click zoom in on this workflow it goes blank. Can anyone one tell me what this means?