The problem
Reliability reviews tend to go one of two ways: a long status meeting where everyone reads out their week, or a meeting that gets cancelled because there's nothing urgent. Either way the numbers don't get looked at.
How I measured it
The review only works if the numbers are ready before it starts. A small prep script pulled the week together:
# availability per service for the last 7 days from access logs (5xx / total)
for svc in payments ledger gateway; do
awk -v s=$svc '$9 ~ /^5/ {e++} END {printf "%-10s %.3f%%\n", s, 100-(100*e/NR)}' logs/$svc/access-*.log
done
# incidents opened this week by severity
awk -F, -v since="$(date -u -d '-7 days' +%F)" 'NR>1 && $2>=since {c[$3]++} END {for (k in c) print k, c[k]}' incidents.csv
Why it kept happening
Without a fixed agenda, the meeting fills up with whatever's loudest. Without numbers prepared beforehand, half the time goes on finding them.
The process change
Thirty minutes, same time every week, same agenda:
- SLAs: each service against its target for the week. Only discuss the ones that missed or are trending down.
- Incidents: high-severity ones since last week, with postmortem status.
- Action items: overdue ones only.
- Toil: is any rotation over the 25% cap?
- One decision: the single most important thing to change this week.
The numbers went out an hour before. If a service was green and steady, it didn't get mentioned.
What I learnt
People came because it was short, and because it ended in a decision. Skipping the green services kept it to half an hour.
What I do now
- Prepare the numbers before the meeting, automatically.
- Discuss exceptions only. Green and steady gets no airtime.
- End with one decision and one owner.
stage 3 runs this review with you every month → see the engagements