What happened
On 28 February 2017 an engineer debugging S3's billing system in us-east-1 ran a command from an established playbook to remove a few servers. One input was entered wrong, and far more capacity was removed than intended, including two subsystems that the rest of S3 depended on. Restarting them at that scale took hours. AWS's own status dashboard depended on S3, so for a while it couldn't show the outage.
AWS's summary: Summary of the Amazon S3 Service Disruption.
AWS changed the tool to remove capacity more slowly and to refuse to take a subsystem below its minimum.
What a safer runbook command looks like
#!/usr/bin/env bash
set -euo pipefail
TARGETS=$(aws ec2 describe-instances \
--filters "Name=tag:role,Values=billing-worker" "Name=instance-state-name,Values=running" \
--query "Reservations[].Instances[].InstanceId" --output text)
COUNT=$(wc -w <<<"$TARGETS")
MAX=3
echo "About to terminate $COUNT instances: $TARGETS"
if (( COUNT > MAX )); then
echo "Refusing: more than $MAX instances. Check the filter." >&2
exit 1
fi
# AWS supports --dry-run on many EC2 calls: it checks permissions without acting
aws ec2 terminate-instances --dry-run --instance-ids $TARGETS || true
read -rp "Type the count to continue: " CONFIRM
[[ "$CONFIRM" == "$COUNT" ]] && aws ec2 terminate-instances --instance-ids $TARGETS
Show what will happen, cap how much can happen, make a human confirm the number.
What I took from it
The engineer followed the runbook. The runbook let a single wrong input do enormous damage. That's a tooling problem, and blaming the person would have left the tool exactly as dangerous.
The status page detail stuck with me too. Your monitoring and your communication channel shouldn't depend on the thing they report on.
What I do now
- Put a blast-radius limit in any command that removes or changes capacity.
- Show the plan before the action:
--dry-run,terraform plan, a printed list. - Host the status page somewhere that survives your own outage.
stage 0 looks at what your runbooks would let happen → get the free audit