#disaster-recovery
grep -l "disaster-recovery" /portfolio/* · 4 entriesThe DR drill that found what the runbook missed
As DR lead, the drill is where the plan meets reality. One service failed over cleanly on paper and then couldn't reach a dependency still pinned to the old region.
1 min readDR drills on the calendar, not after the outage
As DR lead, the plan is only as good as the last time it was run. Putting drills on the calendar, and tracking every finding to closure, kept it honest.
1 min readLocked out by our own outage
What I took from Facebook's October 2021 outage: a backbone change took the network down, and the tools to fix it were on the same network.
1 min readBackups we had never restored
After a very public database loss that year, I tried restoring our own lab backups. Several of them had been empty for months.
1 min read