[Ongoing Issue] Problem with Logs -> Capacity charts

johnny · 2021-01-08T20:07:15.060Z

Thanks for letting us know, Josh!

josh10 · 2021-01-08T20:12:25.902Z

As long as AWS doesn’t go down, I can deal with some analytics not showing!

anon38627393 · 2021-01-08T20:22:52.761Z

Rate limit hit. Sounds like growing pains to me, as Bubble usership grows as a whole.

Thanks for investigating! (Constant cap monitoring is actually semi-important to our app)

josh · 2021-01-08T23:31:31.041Z

Thanks for everyone’s patience. We have a path forward to get this back up and running again, but unfortunately:

It requires rebuilding parts of our code, so it will likely be at least a day or two until we’re back online
We’re going to lose the historical data: no way of porting it over to the new system

Really sorry for the disruption here

ryanbeaucher · 2021-01-08T23:35:25.365Z

Hi Josh, thanks for the update.
Question-- is this issue affecting other workflows…?
I’m getting a bug notice in my logs about an API Connector workflow not running:

" Workflow error - Sorry, we ran into a temporary bug and can’t complete your request. We’ll fix it as soon as we can; please try again in a bit! "

Just trying to figure out if it’s just me breaking my own things or not!
Cheers~

chris1111 · 2021-01-09T01:00:44.347Z

Can you please clarify Josh on what data has been lost? We ran a test of our platform (taking 2 weeks to plan) during the time Bubble had this outage. Is there a way we can get back the data lost? Critical we get that data - we have lost 1,000 man hours of test time (a 100 person event for an hour) if we cannot get that data back.

JohnMark · 2021-01-09T01:32:40.561Z

Hello @chris1111 Contact directly the support@bubble.io

josh · 2021-01-09T02:23:25.003Z

Hi @ryanbeaucher , likely unrelated to this. That said, that error message generally means there’s a bug on our end, not your end, so please feel free to file a bug report and we can investigate.

@chris1111 – answered you in a DM. Short version: we’ll likely be able to recover historical data on a manual, one-off basis, and will consider doing it in special circumstances like these, but won’t do it for people generally.

josh · 2021-01-10T19:05:10.857Z

Hi all, update on the situation. The good news is if you check your apps, you’ll see that the charts are working again! However, there are a few caveats:

We’re still confirming that the approach we took to fix it solves the underlying backend issue that caused the initial outage. We believe it does, but there’s some hopefully-small probability that we’ll need to disable the charts again in the next few days
You won’t see data prior to when the charts stopped working. As mentioned above, we do still have that data, but in a manual, hard-to-access form. If for some reason it’s extremely important for you to look at the historical data, please email support@bubble.io with an explanation for why you need access, and we’ll help out on a case-by-case basis.
We had to change the library we were using to render the charts. The new charts aren’t quite as feature-rich as the old ones were: they don’t currently support real-time updating, and you can’t hover them to read specific datapoints. We wanted to get something out the door ASAP, so went with this solution, but we’d be open to hearing feedback on the new charts and gradually improving them over time

Again, sorry for all the issues here!

dehacked79 · 2021-01-11T04:45:16.421Z

Given that the subscription model is capacity based, I rely heavily on the real time charts to see the effects of my coding on available resources.

jared.gibb · 2021-01-11T05:02:33.853Z

Working on your site affects overall site capacity?!? That’s news to me. There’s so much to learn in bubbleland

dehacked79 · 2021-01-11T05:55:28.405Z

Not coding specifically, but testing. Some user actions trigger many workflows so it’s critical to see the impact on resources so you can decide if the processes is ok, needs to be rewritten, split up into smaller steps or perhaps moving some items into backend workflows. Hitting resource limits is very nasty as workflows can get terminated mid-cycle.

adamhholmes · 2021-01-11T18:11:31.685Z

Is anyone seeing very different reporting than usual with the new charts?

Previously, with the old charts, our app was running at about 2-3% of capacity most of the time, with occasional spikes up into the 20-30% range at busy times.

Now, since the new charts are showing, it’s pretty much off the scale for the past 3 days!! (we haven’t had any increase in traffic or usage)

image954×563 12.9 KB

The other chart doesn’t show any period of max capacity for the same period

image972×491 10.1 KB

Are the new charts measuring the same thing as the old ones? Or are they not working correctly yet? Has anyone else seen similar reporting?

dehacked79 · 2021-01-11T19:09:17.207Z

Mine is showing lower cpu spikes and higher work flows. Unfortunately I have not kept more detailed records to see if somehow my work flows are actually executing faster. Wouldn’t that be neat…

lakossaack · 2021-01-11T19:13:37.526Z

Interesting that you’re seeing such spikes. I cannot imagine why the charts/analytics would impact your app.

It seems that the detailed capacity section is up, can you look further into what is causing the spike? Perhaps a silly question, but have you added any thick plugins or APIs recently?

adamhholmes · 2021-01-11T19:35:39.510Z

No we haven’t made any changes over the period shown, and there’s nothing in our logs or scheduler out of the ordinary.

I guess I’ll just have to keep an eye on it over the coming days…

josh · 2021-01-12T03:48:53.046Z

@adamhholmes Mind opening a bug report with us and sharing which app it is? It’s possible there’s a bug with the new charts and there’s something wrong with the data they’re displaying , rather than an actual change to your app. It shouldn’t be showing > 100% CPU usage, especially if your app is running normally and not having an outage

ryanck · 2021-01-12T09:28:51.471Z

Hi there,

Yes, it’s happening to me as well.

crathbun · 2021-01-12T19:44:06.005Z

Same issue.

image1318×799 64 KB

what is interesting is if I look at the 24 hour view, the CPU usage goes off the charts towards the right of the graph.

image1318×799 64 KB

But if I look at just the last 6 hours I don’t exceed 50%:

image788×490 21.5 KB

aless · 2021-01-12T20:13:44.818Z

Thanks everyone for the input - an unforeseen change from our update to fix logs, is that capacity, which we used to show in relative terms to a baseline of ‘Capacity used per seconds as a percentage of max allowable capacity’ became the absolute sum of all capacity used over a given resolution of time. Because of this, the capacity is off by 120x on the 1h view (because granularity is 120 seconds), off by 720x on the 6h view (because granularity is 720 seconds), etc.

We’ll be pushing a fix for that.
Thank you for your patience,
Best,