The problem
Every production change, from a DNS record to a database upgrade, went through the same weekly change board. Small changes waited a week. The board approved almost everything, so the risky changes got the same five minutes as the trivial ones.
How I measured it
# changes.csv: id,type,submitted,approved,outcome
# approval rate and the average wait in days
awk -F, 'NR>1 {n++; if ($5!="rejected") a++; w+=($4-$3)/86400} END {printf "approved %.0f%%, avg wait %.1f days\n", 100*a/n, w/n}' changes.csv
# which change types caused incidents
awk -F, 'NR>1 && $5=="caused_incident" {c[$2]++} END {for (k in c) print c[k], k}' changes.csv | sort -rn
Approval was close to 100%, and the incidents came from a handful of change types.
Why it kept happening
One process for every change meant the board couldn't tell the difference between them. When everything needs approval, approval stops meaning anything.
The process change
Three tiers, decided by the change type, not by the person:
- Standard: pre-approved, low-risk, repeatable (a DNS record, a scaling change within limits). Logged, no board.
- Normal: peer-reviewed by someone else on the team, with a rollback plan.
- High-risk: the change types that had caused incidents. Board review, a scheduled window, and a named person watching it go out.
A standard change that caused an incident moved up a tier. That kept the list honest.
What I learnt
The board got better at its job once it had less to do. Its time went to the changes that actually needed a second look.
What I do now
- Tier changes by risk, using your own incident history to decide the tiers.
- Pre-approve the boring changes so the review has time for the dangerous ones.
- Move a change type up a tier when it causes an incident.
if your change process slows everything equally → page me