Urgent: Page Load DDoS

Need help
1

My app isn’t loading, all pageviews just return the text in browser: App too busy (1)

Looking at capacity, it seems that i’m getting 300k pageviews an hour on my /signup page. I can’t see this in Google Analytics or Cloudflare, but looks like I’m getting hammered from somewhere. Anyone seen this before/know what to do?

2

Contact bubble.io support

3

I have, and filed a bug report, but they only operate from 9am EST so looking for faster possible solutions.

4

I know that you can have Bubble turn on Cloudflare’s Under Attack Mode when you are suffering from a DDoS attack

Maybe contact your domain provider and see if they can do anything.

Also read this article Distributed Denial of Service DDoS Attacks it might help

5

Thanks @bodiebloore

Perhaps a naive question, but do you know why if I’m routing my DNS through cloudflare, turning attack mode on in my dashboard isn’t helping? I can’t even see the requests that the Bubble logs page is showing

6

Out of curiosity, what kind of WU usage are you seeing from this?

7

No WU usage as not running any workflows, just hitting the server extremely hard (tens of millions of request an hour now)

8

@grace.hong @nick.carroll

9

Done a bit more digging here. The issue is, and what I don’t understand about how Bubble handles this is:

  • I have Cloudflare on my domain; Bubble also applies it to my domain on their end
  • This creates an orange to orange problem, whereby Bubble’s Cloudflare overrides mine
  • This would be fine except Bubble’s Cloudflare completely missed the DDoS/just let all of the traffic too. We’re now talking billions of requests.

Can’t work out why they’d do this/it seems like a huge letdown. Had to completely take the domain off bubble.io, and build a subdomain which essentially mirrors {appname}.bubbleapps.io, just so I could layer my own Cloudflare on top/enable DDoS mode

1 Like
10

For anyone reading in the future:

  • Don’t expect Bubble to do anything.
    Not only is their Cloudflare setup not designed to prevent attacks in the first place, but I suspect unless you’re on the highest enterprise tier then you won’t get any meaningful response within the time of the attack.
    It’s beyond me how our domain could get 2.5 billion requests in a day and Bubble do nothing to identify/prevent the DDoS.
  • Wait it out.
    There basically wasn’t anything we could do other than wait. There were some mitigation strategies we could use to take control of our Cloudflare, but it was like playing whack-a-mole and the attackers kept following our moves.
11

Thank you and best of luck - sounds like an extremely frustrating situation.

One question - I thought Bubble applied WU to any activity, including page loads, which would be impacted if bots loaded your page hundreds of thousands of times.

I’m trying to figure out if there is any pricing risk related to bot attacks driving a big increase in workload units, which then results in a large bill from Bubble.

12

Someone can correct me here on how this all works @ed727:

  • We initially had a small workflow (0.5 WU or similar) on the page load, this was racking up WU usage (high but not absurd).
  • I took the workflow off (wasn’t critical) which stopped WU rising, and the page load itself didn’t consume any. Server was just getting hammered independently of the fact that no WU was being accumulated.
  • If the attackers had chosen a different page, or been more sophisticated, they could’ve definitely racked up an absurd amount of WU
13

Hi @macklpgr ,
thanks for reporting on this. Would you mind sharing a bit more how you managed to add the DDOS protection “on top” with cloudflare? I assume you give cloudflare control of your domain and then you create sub-domains in the cloudflare UI and make all other necessary settings?

14

Hey @geo1
Sorry so basically we couldn’t add actual DDoS protection. What we did is:

  • Noticed that all the traffic was coming to example.com/signup.
  • Usually with Cloudflare this would be easy to solve - just set up a page/redirect rule that sends all requests pointed to that page somewhere else (we chose google.com)
  • The problem is, because Bubble also uses their own Cloudflare instance on each domain, you get an ‘orange-to-orange’ problem, where Bubble’s Cloudflare overrides ours and basically renders ours useless. We couldn’t set up page/redirect rules, or even see the (billions!) of requests in our dashboard.
  • To fix, we moved our app to app.example.com (i.e. change the domain inside of Bubble), redirected all traffic from example.com/{path} to app.example.com/{path} (using page rules), and finally redirected (with a high priority rule) all traffic from example.com/signup to google.com.

This stopped any of the DDoS traffic from hitting our servers/solved the issue, but is incredibly ugly as:

  • We obviously didn’t want users to see a subdomain (app.example.com) in their browser.
  • The attackers, at various points, redirected their attack to our subdomains (meaning we had to repeat the process above).

It’s honestly just mindblowing that Bubble weren’t able to help at all. We’re a 7 figure ARR app, and even now Bubble haven’t gotten back at all (other than to say we’ve passed this on to engineering). I’ve never really been worried about platform lock-in etc before, but this was an absolute shitshow and Bubble may have just as well said you’re on your own here. That actually would’ve been more helpful to me, rather than thinking they might be able to help.

1 Like
15

Sorry to hear that @macklpgr and props to your and your team for minimizing the damage.

Would it be possible if you could update this thread if ever you get a response from Bubble’s support?

16

Echoing Nino, sorry to hear about the situation, and hats off to you and your team because it sounds like you managed it quite well.

I do have a question, though, and I swear I am not trying to rub salt in the wound (that should go without saying because it’s not what I do out here)… I am just super curious and genuinely trying to verify something that I always believed to be true.

If you are a 7-figure ARR app (which means you have far surpassed us huddled masses out here in gen pop), shouldn’t you be on a dedicated instance for lots of reasons, including the ability to configure custom Cloudfare settings and having access to a dedicated account manager and technical success manager?

To be clear, I am not saying Bubble should have left you hanging, but my understanding has always been that for apps like yours (and kudos to you, by the way, because you must be in the top 1% of Bubble apps), being on a dedicated instance is an important piece of the puzzle when it comes to protecting your Bubble investment. I am wondering at this point if that understanding is incorrect or misguided because I offer a lot of advice out here in the forum, and I try to make sure I am always learning and taking in new information.

Anyway, please feel free to ignore this question, and if anyone thinks the question is a diversion from the original topic, flag it and I will remove it myself. That being said, it feels like a legitimate question to me, so I thought I’d ask to see if OP (or anyone else) can share their thoughts.

17

That’s a fair point @mikeloc, maybe we should be! At the same time though, we’re managing fairly well within the current plan, and the only real reason to upgrade it seems would be crisis technical support i.e. during the incident last week. Other than that I don’t think there are major reasons to need to upgrade.

In hindsight, may have been worth it to be on a higher tier, but hopefully can understand it would have also felt potentially ROI negative to pay a few $k a month for support just in the off chance that something like this happened. I also think ability to manage DDoS isn’t something that should require that level of spend (given that it’s essentially free in a vanilla Cloudflare setup).

@ntabs will update if I hear anything back that could be useful to users that search this in future/undergo similar issues. Currently nothing since 6pm UTC on Thursday (the day of the incident), just saying they’d passed on info. It’s now midday UTC Monday.

2 Likes
18

Interesting, thanks for the clarification.

My wild guess is that since loading the static part of the page is handled by cloudflare, Bubble’s servers aren’t hit for a worfkflow or to send data, and therefore there’s no workload unit charge for the page load.

Saw your later posts - yes, troubling.

19

Hey @ed727, that’s correct, in an instance where either Cloudflare has detected/is mitigating the DDoS, or where you’ve used Cloudflare to redirect traffic away.

The orange-to-orange problem though basically means that you can’t do the above with Bubble, as Bubble prevents you from using your own Cloudflare and theirs isn’t set up to deal with DDoS.

So basically we were getting tens of millions of requests an hour to our Bubble app (although Bubble would only register half a million of these as pageviews in the logs screen). These are definitely pageviews (they’re not being handled or blocked by Cloudflare) they just weren’t incurring WU as far as I can see.

20

This topic was automatically closed after 70 days. New replies are no longer allowed.