#on-call
grep -l "on-call" /portfolio/* · 5 entries1,300 monitors, and which ones may wake a human
Running 1,300+ Datadog monitors is mostly about deciding which ones page, which ones open a ticket, and which ones shouldn't exist.
One incident commander, not five people typing
During big incidents, everyone jumped in and nobody was in charge. A named incident commander on the 24/7 rotation made the response calmer and faster.
1 min readCapping on-call toil at 25%
On-call time was disappearing into repeat manual work. A 25% cap on toil, measured from tickets, gave the rest back to fixing the causes.
An alert can't page until it has an owner and a runbook
Anyone could add an alert that paged on-call. A simple admission rule (owner, runbook, action) stopped the noise at the door.
A pager that cried wolf
On-call was getting paged most nights, nearly always for nothing. Counting where the pages came from showed a handful of alerts doing almost all the damage.
1 min read