The page
On-call engineers were busy all week and the same problems kept coming back. Restarts, manual cleanups, the same request from the same team. Nobody had time to fix the causes because they were busy handling the effects.
Triage
Before setting a cap you need a number. We tagged tickets as toil (manual, repetitive, no lasting value) and counted from an export:
# tickets.csv: id,engineer,category,hours
# share of on-call hours spent on toil
awk -F, 'NR>1 {t+=$4; if ($3=="toil") s+=$4} END {printf "toil: %.0f%%\n", 100*s/t}' tickets.csv
# the repeat offenders: toil hours by ticket summary
awk -F, 'NR>1 && $3=="toil" {h[$5]+=$4} END {for (k in h) print h[k], k}' tickets.csv \
| sort -rn | head -10
Root cause
Toil grows if nothing pushes back. Each manual fix is quicker than automating it, today. Added up over months, it took most of the week.
The fix
On-call toil was capped at 25%. When a rotation went over, the next priority was automating or removing the top item on the list, ahead of new feature work. The repeat-offender list from the export made it obvious what to automate first.
What I learnt
A cap turns "we should automate this someday" into a rule with a trigger. And measuring toil, even roughly, ends a lot of arguments about whether it's a problem.
What I do now
- Tag toil in the ticket system, so it can be counted.
- Set a cap and decide in advance what happens when it's exceeded.
- Automate the top repeat item first. It's usually most of the hours.
stage 3 keeps an eye on toil every month → see the engagements