Depsite upgrading to the Professional plan, my site continues to hit max capacity

@brandon8 that makes sense and. yeah, Bubble makes all of that stuff really easy.

So, the value we get from Bubble is pretty big when we think about it being a full stack solution with a database/CMS storage, hosting/bandwidth, build time, login features, etc.

The thing with Bubble and pricing and what you get for that is a little weird though, in the sense that we don’t see a lot of other solutions priced in the same way.

Like, if you built a JAMstack type static site (using cloud functions and APIs for database and stuff), there’s a lot you can do totally for free in the prototyping/proof-of-concept phase as you can get away with the free/community level of Gatsby, free version of Netlify perhaps for deployment over their CDN, the free level of some headless CMS (like Contentful or Prismic) and a free or very low cost.

And lots and lots of Bubble projects at the hobby, personal and even some at the professional level could (if they were built “in code”) be delivered in this way, without any actual cash outlay at all, especially if you’re just a one-person team. And, the main difference you’d see in sites like this vs Bubble is that their page load speed would be orders of magnitude faster than what you see in Bubble. (And this is probably the main frustration with Bubble on these lower tier plans – your app spends a lot of time waiting for the shared server to serve your page.)

But, even in that more modern and initially lower-cost architecture, once you get serious, you’re likely going to need to upgrade to the “pro” level of whatever CDN provider you’re using (e.g., Gatsby Cloud, Netlify, etc.) and that goes to around $20/month. The same thing might happen with your CMS of choice and you might start consuming enough compute on whatever cloud functions thing you’re using, your database, etc. And all of those individually cheap $20-$100 per month things start adding up.

(But the sheer amount of compute/storage/bandwidth you’d be buying there is in fact a lot more than what Bubble gives you at similar costs.)

And a lot of these things have a per-user model where in some cases, your costs go up for each team member (where at least with Bubble you get 1 collaborator at the Pro level).

OTOH, in a JAMstack or similar model, if you go over capacity with your CDN, you just get charged for what additional bandwidth you use during spikes like the one you experienced, @brandon8. (And this tends to be very low cost and at least your site doesn’t “go down”.)

In either model, there tends to be a big jump in cost to get to the “production/enterprise” (whatever you want to call it) level. Where Bubble is now $475/529/month for the Production level, a similar-ish level with other approaches might be like $99/mo for your CDN, a similar amount for your CMS and/or database, and various other service costs (so like, maybe half of what Bubble costs at the low end to $400-ish bucks a month, depending on a LOT of different factors). And of course, it’s not an apples-to-apples comparison (for the money you’d be buying A LOT more compute, bandwidth, etc.).

Another big difference is that, where Bubble is charging per project, in other architectures you might be able to have multiple projects (repositories) all under the same account and as long as you’re within your max pages/storage/build minutes/whatever it doesn’t matter how many projects you’re running (this varies immensely between different services, but it’s generally true that in one account you could be servicing multiple and in some cases unlimited projects).

This is one of the places where the Bubble pricing model really starts to fall down. (Like just to support my plugins, I have multiple personal plans and then I have a Professional plan with a couple of extra capacity units for my VR calendar thingy. I wish I could just bundle them up on something like a Pro+ plan at around 200-250 bucks and have a bunch of these little projects in that plan.)

Basically, in the world of web app stuff today, cost grows like a step function. There’s a metric s-ton of “free” (or free-ish, right?) and then there’s a small step up to the “OK, OK, I gotta be on the paid version of a couple of things and the pro version of that one thing” level and this can last you for a long time, and then there’s a BIGGER step up where you’re in the “this thing costs real money” category.

With Bubble, the “free”/free-ish level starts at $29/month, and the “small step” is to $129. If the basic page serving stuff wasn’t soooo vastly different with respect to other approaches, this wouldn’t seem so bad to my mind. And I totally remember that getting to that small step with my VR calendar project. It’s like, “damn, this thing is ramen profitable and now Bubble gonna take my ramen!” right?

I didn’t really mean for this reply to be a commentary on Bubble pricing, but I just wanted to note that for where you app is now in terms of users/hour, even if you were on some other architecture, you’re now at that “small step” level and you’d be paying $100/mo (or right in that neighborhood) for something or in total for all of the somethings that power your site. (Though as I’ve said too many times in this post already, the value-for-money of the somethings would be rather higher than what Bubble gives us at that price point.)

3 Likes

I hope this isn’t an insulting question, but are you absolutely certain you don’t have a path in your app that is essentially infinitely calling a page render? Those analytics look like an example of mutual recursion gone wild.

Just a thought.

4 Likes

Not insulting at all! That’s actually a pretty solid theory. My app basically has three pages:

  • Homepage
  • Search
  • Single Product

I’m in the three so much that I feel like I would have spotted something. Likely isn’t something using a workflow because otherwise the workflow count would also spike (right?).

Are there any common scenarios off the top of your head where such a thing would happen?

Page load was fast and responsiveness was excellent, except on “page scroll” where the new item populations were a little bit laggy but not “broken” just noticeable.

Ryan
Hong Kong
96 Mbps

1 Like

Keith, this means so much that you took the time to write all this! Full of fantastic insights and gives me a lot to chew on. I know others have said the same in the forum, but it seems like the dream solution would be for Bubble to allow us to pay for monthly access to the Bubble builder and then have the option to bring in your own hosting. Something like that would make Bubble such a great long-term solution for scaling without having to completely ditch the platform down the road (assuming you refused to bring on developers).

Again, much appreciated for these thoughts and I’ll definetely be reading it 10+ times throughout the night as a ponder the meaning of life :joy:

2 Likes

Can you make your app into a single page app. Perhaps that will reduce your issues a bit?

Now I’m just gonna shoot in the dark here because I don’t have a handle on the full situation, but you point out that the workflows running seems pretty normal/stable while all the other graphs show the spike when you pinged your Discord.

It seems the workflows running is the anomaly: why would the workflows continue to run once the site visits drop off?

As you pointed out, 44K visits was impossible - given your reach, so something in your “no code” is triggering “re-visits”.

Have you followed all the basics pointed out here: Understanding Workflow Execution Rules | Bubble

The following is a great procedure used in coding debugging and while it seems super-simple, it’s often the basics that elude us:

  1. What did you expect your code to do? Get very clear on your expectation for without this, it’s impossible to know where your “code” is going off the rails. The more specific you can be, the better.
  2. What happened instead? Again, get very clear on exactly what happened when things went off the rails.
  3. What does your expectation depend on? You expect a certain result and yet you are getting another result. For it to run the way you expect, what actually needs to happen (or not happen)?
  4. How can we test the things that our expectations depend on? Maybe you can duplicate your pages or workflows one at a time and put them on a special personal “testing page” and run them (where your customers don’t have access to them). Do it one workflow at a time and just keep re-triggering the workflows and see what your graphs do. You could even duplicate your entire bubble app and strip it down one piece at a time and see what happens and see what your numbers do. It might even be economical to pay for the new app in bubble (basic) and then cancel once you’ve figured it out.
  5. The last step is to make reality match expectation. This is pretty self-explanatory.

I realize that is relatively opaque, being behind the bubble.io abstraction, but if you follow this procedure, it is the most likely way to bring you to a resolution.

Good luck and please let us know how it goes.

3 Likes

Djryanash is saying the same thing I mentioned, but in much more detail. I believe you have a subtle logic issue that is causing these issues.

The trick will be to figure out how to reproduce the issue.

I’ve had errors like this in traditional coding environments such as React and Angular. Looking debugging environments like Chrome Dev Tools can be helpful.

Good luck!

2 Likes

One thing to keep in mind in that graph is the “zoom level” of each graph. That massive pageview spike makes the rest of the traffic look like near 0, when in reality there are several hundred pageviews on each dot. When you look at the Workflow graph you can see the highest number is just 18. So for a workflow to be associated with this spike (outside of a single workflow that causes a loop outside the workflow, we should see a graph that looks similar to the Pageview graph.

Regardless, love the insights you posted! I went through that video and did learn some new things I didn’t know!

Situation Update
I had yet another “artificial spike” happen again yesterday. I backed up even more to see if I could see a time pattern that could associate it with a backend workflow (maybe something random firing off every X time) but there was no consistency. (The following picture shows the first spike I referenced above followed by the second random spike)

Using the “Server Capacity Usage Details” section I narrowed the time down to that spike and the only thing to learn is that most of the resources are going to “Page Load” specifically on the Index page.


And then when looking at the Server logs in that time period, the only thing that shows up is the HTTP Request’s & HTTP Responses from visitors and nothing was out of the ordinary. No single user visiting the home page a bunch of times, nothing. I had every single box in the “Advanced” section to see if anything else showed but it’s just the HTTP information.

I’ve gone through every workflow with every page and re-usable element & every backend workflow. Nothing! Historically speaking I can’t go more than 24 hours without a spike in pageviews so I’m sure sometime today I’ll see that spike again, but until then I will say the site seems to be running well right now and that is with about 500 users/hour holding about a 10% capacity.

I actually wouldn’t expect the issue to be in a backend workflow. This looks far more like a front end issue to me. If it was a react app, my first thought would be state being updated causing a render causing a state update causing a render ad infinitum.

2 Likes

It seems to me that something is “scraping” the data from your website. There’s not much you can do about it, as the IP addresses often change with each request. Since they’ve already collected most of your data, that’s probably why the next peak is lower. It is also possible that if people received many errors, they tried to refresh and refresh several times, which caused the first spike.

This is just a thought.

5 Likes

I was thinking the same thing. It happens to any e-commerce site and your price watching site screams “Scrape me!”.

Wish i could advise you on how to detect/prevent it but I’m no expert.

2 Likes

Yeah this is a good thought! I knew scraping would be a thing but my site is so new to the public I wouldn’t anticpate the scrapers coming so soon. Also, since all of the product data shows up on my /search and /product pages, wouldn’t that be the place I’d see high volume views as well?

1 Like

Well, look at the date and time. It’s the exact same day and time from last week.

Clearly it’s an automated thing, whether from your site or another.

That in itself is good to know for you and others though.

1 Like

@brandon8, further to what others have suggested about scraping, you could try implementing a rate limiting rule in Cloudflare. (Which may or may not work if the loads are from different IPs.)

@troy.roberge The first post in this thread was only a few days ago, so not more than a week.

Totally right. Misread graph. His first graph showed different data, thought it was a previous week. Went from views of 25,000 to 45,000.

Totally get the confusion! Took me a while at first getting used to Bubble’s “auto zoom” type reporting where, depending on the selected time frame, the numbers can visually appear totally different.

Here to just follow the updates. I’m making all my “bets” on Bubble. I’m doing my best to accept the fact that you can’t scale without Xano. I wish we never had issues like this.

@brandon8 if I were you I’d be sending Bubble 1000 emails until I got a call with an engineering team.

Good luck my friend hope you’ll sort this out soon :fist:

How do you get updated live data from your selected retailers? Do you scrape their data or did you just ask for JSON data?

Also, I’m assuming to keep prices updated you do recursive workflows? Or is it something else in Xano?

Fantastic site its honestly way easier to use the pcpartpicker in some ways.