What happened
In April 2011 a network change in one availability zone in AWS us-east-1 sent traffic for the EBS storage cluster onto a network that couldn't carry it. Storage nodes lost contact with their replicas. When the network came back, a large number of volumes tried to re-mirror at the same time, used up the free capacity in the cluster, and got stuck. The control plane for the whole region slowed down as requests piled up behind the stuck zone.
AWS wrote it up in detail: Summary of the Amazon EC2 and Amazon RDS Service Disruption.
Why it mattered
A lot of teams had their servers and their backups in the same zone, or used EBS snapshots as their only copy without testing a restore anywhere else. When the zone went bad, so did the plan B.
How you'd spot it today
# volumes AWS itself marks as impaired
aws ec2 describe-volume-status \
--filters Name=volume-status.status,Values=impaired
# open AWS Health events (the Health API is served from us-east-1)
aws health describe-events --region us-east-1 \
--filter eventStatusCodes=open
# on the instance: is the disk stalling? watch await and %util
iostat -x 1
And the check that matters most, does a copy exist somewhere else?
aws ec2 copy-snapshot \
--source-region us-east-1 \
--source-snapshot-id snap-0123456789abcdef0 \
--region us-west-2 \
--description "cross-region copy"
What I took from it
Redundancy only counts if the failure you're planning for can't reach both copies. "Another server in the same zone" protects you from a dead server. It doesn't protect you from the zone.
The re-mirroring storm is also a good picture of a retry problem. Every volume did the sensible thing on its own, and together they made it worse.
What I do now
- Keep at least one backup copy in a different region or account, and restore from it on a schedule.
- Ask what a recovery process does when everything recovers at once. Add backoff and limits.
- Don't let one zone's health be a hidden dependency for the whole region's control plane in your own designs.
if your backups live next to the thing they back up → page me