I sympathize - But please keep in mind this happens… Hackers are always finding new ways of intruding. My company runs tens of enterprise software apps… Even w/ weekly security reviews, monitoring services, etc, there still are inventive little suckers (hackers) out there who find ways in. So I wouldn’t blame Bubble squarely - Web security changes every day.
They’re DDOS’ing you with 20x the traffic that bubble, as a whole, normally gets
We’re consulting with cloudflare for mitigation strategies that will not harm the main cluster. Most of the traffic is coming from botnets located in Russia, Indonesia, and India.
Update:
You’re peaking at about 10k requests per second. We’ve added a 5-second JS challenge every 30 minutes that should mitigate some of the bot traffic without taking your site completely offline. Meanwhile we’ll keep evaluating options at our disposal to keep you online as much as possible.
This kind of ransom usually comes from hackers who rent others’ botnets for limited periods of time, so I expect to see a number of attacks over the coming days until we figure out a long-term solution or they get bored.
Note that the JS challenge limited the rate of the 21:00 DDoS (requests came in at about 1/3 the rate of the previous attack) and completely neutralized it from the Bubble side.
@kodjima33 Sorry about what we had to do to your app on Sunday. This has been a useful learning, and we’ll get rid of the JS challenge once your attacker gets bored.
Sort of? CF admitted that they don’t trigger automatic DDoS protection until ~10krps, which will easily swamp any single bubble app (but obviously not the cluster), and most customers who ask for the limit to be changed ask for it to be raised.
2021Q1 includes some major initiatives for getting more immediate insight into cluster performance (or, in less verbose terms, more fine-grained monitoring and alerting) that should enable us to be more proactive about this sort of thing in general.