3/22/2023 Incident Postmortem

[johnny]

Haha thanks! I have forum notifications turned on for my laptop. It varies nowadays, I used to be quicker not so much anymore

[zelus_pudding]

I used to be quicker not so much anymore

Lol, didn’t notice

[davidhvse]

Thanks. As I’ve said when the attack occured, the best way to mitigate it is probably by enabling under attack mode in the cloudflare’s panel and if it’s not working, enabling bot fight mode too in the Cloudflare panel.
I’ve got surprised by not seeing at least under attack mode enabled during the attack and just seeing it enabled in forum and editor when the attack was about going to get mitigated, and not in the first place. Due to that, I supposed the lack of access/knowledge of cloudflare’s tools.

Thanks to all the team for dealing with such unfortunate event, next time (hope it doesn’t happens again) I’m pretty sure the response will be better.

[LA81638]

Every adversity, every failure, every heartache carries with it the seed of an equal or greater benefit.

Napoleon Hill

[johnnyweb]

Thanks @josh for the awesome transparency. Theres some significant takeaways for all of us to implement in our own builds as well. I for one will be looking more closely out our own Cloudflare configs and get some better doco for the team setup.

Will be good to see those who move to dedicated plans having some of these issues mitigated as well.

[James33]

Appreciate the radical transparency. Shit happens and it’s how you respond that builds trust.

[princenocode]

Thanks a lot for your transparency and above all for your level of responsiveness which enabled us to limit the damage to our customers​:clap:

[nathan9]

Thanks @josh this is very transparent.

When you do break the clusters up to smaller clusters how will you monitor the clusters on status.bubble.io. I hope we can see the separate cluster statues and that we can identify what clusters we are on. Lastly a Canadian only cluster would be amazing for us Canadians.