The problem
DR plans tend to be written once, reviewed once a year, and tested for real during an outage. By then the system has changed, and the plan describes something that no longer exists.
How I measured it
# when was each application last drilled? (drills.csv: app,last_drill_date,result)
awk -F, -v cutoff="$(date -u -d '-180 days' +%F)" 'NR>1 && $2<cutoff {print $1, $2}' drills.csv
# open findings from past drills, oldest first (findings.csv: id,app,opened,status)
awk -F, 'NR>1 && $4!="closed" {print $3, $2, $1}' findings.csv | sort
Why it kept happening
A drill is never urgent. There's always something more pressing, until the day it's the most pressing thing there is.
The process change
- A calendar: every application has a drill slot on the calendar, at least twice a year for the critical ones. Booked months ahead, so it's planned work.
- A script: each drill follows the runbook exactly as written. If someone has to improvise, that's a finding.
- A real end-to-end test: after failover, a real request through the whole path. A green health check doesn't count.
- Findings with owners: every gap gets a ticket, an owner and a date, and the next drill checks it's fixed.
What I learnt
The findings were the point. A drill that found nothing was usually a drill that didn't test enough. And improvising during a drill told us exactly where the runbook was wrong.
What I do now
- Book drills months ahead, like any other planned work.
- Follow the runbook word for word. Every improvisation is a finding.
- Track findings to closure and check them in the next drill.
stage 1 looks at when your DR plan was last run → see the engagements