#lessons
grep -l "lessons" /portfolio/* · 9 entriesA generated config file that doubled in size
What I took from Cloudflare's November 2025 outage: a permissions change made a generated file double in size, and the proxy crashed on a limit nobody expected to hit.
1 min readA bad content update, everywhere, at once
What I took from the July 2024 CrowdStrike outage: a configuration update, not a code release, crashed millions of Windows machines because it went to all of them at once.
Locked out by our own outage
What I took from Facebook's October 2021 outage: a backbone change took the network down, and the tools to fix it were on the same network.
1 min readOne bad rule, deployed everywhere at once
What I took from Cloudflare's July 2019 outage: a single regular expression, pushed globally in one go, and why config changes need the same staged rollout as code.
One typo, a whole region
What I took from the February 2017 S3 outage: a routine command with one wrong input, and the guardrails that should sit around every runbook command.
1 min readOur DNS provider went down and took us with it
What I took from the October 2016 Dyn attack: when one managed DNS provider fails, every site that relies only on it disappears, however healthy its servers are.
1 min readThe day a dependency disappeared from the registry
What I took from left-pad: an eleven-line package was unpublished, builds broke everywhere, and a private mirror would have made it a non-event.
1 min readOne extra second that pinned every CPU
What I took from the June 2012 leap second: a kernel timer bug, Java and MySQL spinning at full CPU, and a fix that was just setting the clock.
When the storage failure took the backups with it
What I took from the April 2011 EBS outage: a network change, a re-mirroring storm, and why one availability zone wasn't as isolated as people assumed.