The page
The on-call rotation was getting woken up most nights. Almost every page resolved on its own within minutes. People had started acknowledging alerts without opening them, which is exactly when a real one gets missed.
Triage
I exported a month of alert history and counted.
# alerts per rule, noisiest first
jq -r '.[].alertname' alerts-last-30d.json | sort | uniq -c | sort -rn | head -15
# how many resolved within 5 minutes (flapping)
jq '[.[] | select((.resolvedAt - .firedAt) < 300)] | length' alerts-last-30d.json
# when do they fire? count by hour of day
jq -r '.[].firedAt | strftime("%H")' alerts-last-30d.json | sort | uniq -c
A handful of rules made up most of the pages, and most of those resolved within five minutes.
Root cause
Static CPU and queue thresholds set close to normal peaks, evaluated on a single data point. Any short spike crossed the line, the alert fired, and the next sample cleared it.
The fix
For each noisy rule: alert on a sustained condition (several minutes, not one sample), move the ones that didn't need a human at night to a ticket instead of a page, and add a runbook link to every alert that still paged. Two rules we deleted outright, because nobody could say what they'd do if it fired.
What I learnt
Every alert that pages for nothing makes the next real one less likely to be read. That makes alert noise a reliability problem in its own right.
What I do now
- Count pages per rule every month and fix the top few.
- Every paging alert needs a runbook and an action. If there's no action, it isn't a page.
- Alert on sustained conditions or on symptoms users feel, not on single spikes.
this is the first thing stage 0 looks at → get the free audit