The page
I'm the disaster-recovery lead for nearly every application on the platform. The part of that job that matters most is the drill, because a DR plan nobody has run is a guess.
Triage
During one drill, a service failed over to the secondary region and its health checks went green. Then real requests started failing.
# where is DNS pointing now?
dig +short app.example.com
# Route 53 health check status from each checker region
aws route53 get-health-check-status --health-check-id <id> \
--query "HealthCheckObservations[].[Region,StatusReport.Status]" --output table
# is the replica promoted and writable?
aws rds describe-db-instances --region us-west-2 \
--query "DBInstances[].[DBInstanceIdentifier,DBInstanceStatus,ReadReplicaSourceDBInstanceIdentifier]"
# what endpoints is the app actually configured with?
grep -rn "us-east-1" ./config ./helm 2>/dev/null
The grep found it.
Root cause
One downstream endpoint was configured with the primary region written in. The health check only tested that the service was up, not that it could reach its dependencies, so failover looked healthy and wasn't.
The fix
Moved the endpoint to region-aware config, added a dependency check to the health check, and added a step to the runbook: after failover, test a real request end to end before calling it done.
What I learnt
Health checks that only check the service itself will tell you a broken failover worked. And the drill found this in a calm hour instead of a real outage.
What I do now
- Drill regularly, and treat findings as the point of the drill.
- Health checks should test real dependencies, at least shallowly.
- Search config for hard-coded regions and endpoints before every drill.
stage 1 includes a look at whether your DR plan would work → see the engagements