The problem
When something big broke, five people would join the bridge and start fixing things at once, sometimes the same thing, sometimes in opposite directions. Updates to stakeholders came from whoever had a spare minute.
How I measured it
# from the incident chat export: distinct people posting in the first 30 minutes
jq -r --arg start "$START" '.messages[] | select(.ts < ($start|tonumber)+1800) | .user' incident-chat.json | sort -u | wc -l
# how many changes were made during the incident, and by how many people (CloudTrail write events)
aws cloudtrail lookup-events --start-time "$START_ISO" --end-time "$END_ISO" \
--lookup-attributes AttributeKey=ReadOnly,AttributeValue=false \
--query "Events[].Username" --output text | tr '\t' '\n' | sort | uniq -c
Lots of people posting, lots of people making changes, and no single thread of decisions.
Why it kept happening
Everyone wanted to help, and nobody had the job of deciding. Without that role it's normal for the most senior person to start fixing things, which leaves nobody watching the whole picture.
The process change
An incident commander on the 24/7 rotation, separate from whoever is fixing:
- Commander: runs the incident. Decides, assigns, keeps the timeline. Doesn't type commands.
- Ops lead: makes the changes, and says each one out loud before making it.
- Comms: sends updates to stakeholders on the agreed schedule.
For smaller incidents one person holds all three. For big ones they're split, and the commander says so at the start: "I'm IC, Sam has ops, Priya has comms."
What I learnt
The commander's most useful skill was not touching anything. Someone watching the whole incident catches the change that's about to conflict with another one.
What I do now
- Name an incident commander at the start of every significant incident.
- Keep the commander off the keyboard.
- Say each production change out loud before making it.
stage 3 includes help running incidents like this → see the engagements