#alerting
grep -l "alerting" /portfolio/* · 4 entries1,300 monitors, and which ones may wake a human
Running 1,300+ Datadog monitors is mostly about deciding which ones page, which ones open a ticket, and which ones shouldn't exist.
RDS out of storage at 3am
A production database ran out of disk overnight. The alert existed, but it fired when there was already almost nothing left.
An alert can't page until it has an owner and a runbook
Anyone could add an alert that paged on-call. A simple admission rule (owner, runbook, action) stopped the noise at the door.
A pager that cried wolf
On-call was getting paged most nights, nearly always for nothing. Counting where the pages came from showed a handful of alerts doing almost all the damage.
1 min read