The problem
Alerts got added during incidents, in a hurry, and pointed straight at the pager. Months later nobody knew what half of them meant or what to do when they fired.
How I measured it
# export of alert rules: name, routes_to, owner, runbook_url
# paging alerts with no owner or no runbook
jq -r '.[] | select(.routes_to=="pager") | select(.owner==null or .runbook_url==null) | .name' alert-rules.json
# how many of those fired in the last 30 days
jq -r '.[].alertname' fired-30d.json | sort | uniq -c | sort -rn > fired-counts.txt
Plenty of paging alerts had no owner or runbook, and some of them fired often.
Why it kept happening
Adding an alert was easy and had no rules. Removing one felt risky ("what if we need it?"), so nobody did. The set only grew.
The process change
Three conditions before an alert could route to the pager:
- Owner: a team, not a person.
- Runbook: a link that says what the alert means and the first steps.
- Action: something a human has to do now. If nothing needs doing at 3am, it goes to a ticket queue instead.
New paging alerts went through a short review. Existing ones got a month to meet the rule, and anything that didn't moved to the ticket queue. Nobody deleted alerts in the first pass, which made it easier to agree to.
What I learnt
The rule worked because it was about the pager, not the alert. Teams could still create any alert they liked. Only the ones that met the bar could wake someone up.
What I do now
- Owner, runbook and action before anything pages.
- Route "useful but not urgent" alerts to tickets.
- Re-check the paging set every quarter against the rule.
this is what stage 0 checks first → get the free audit