Skip to content
rajesh.dhar
2022 · lead sre · datadog

1,300 monitors, and which ones may wake a human

Running 1,300+ Datadog monitors is mostly about deciding which ones page, which ones open a ticket, and which ones shouldn't exist.

1 min readby Rajesh Dhar
storyalertingdatadogon-call

The page

With 1,300+ monitors, coverage is the easy part. A page could come from any of them, and nobody can hold that many in their head.

Triage

Every so often I audit the whole set through the API:

# pull all monitors (paged)
for p in 0 1 2; do
  curl -s "https://api.datadoghq.com/api/v1/monitor?page_size=1000&page=$p" \
    -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY"
done | jq -s 'add' > monitors.json

# how many notify the pager?
jq '[.[] | select(.message | test("@pagerduty"))] | length' monitors.json

# paging monitors with no runbook link
jq -r '.[] | select(.message | test("@pagerduty")) | select(.message | test("runbook"; "i") | not) | .name' monitors.json

# monitors with no team tag (no owner)
jq -r '.[] | select([.tags[] | startswith("team:")] | any | not) | .name' monitors.json

Root cause

Monitors get added during incidents and almost never removed. Each one made sense on the day. Over time the set fills up with alerts nobody owns.

The fix

A few rules for every monitor:

  • It has a team: tag. No owner, no monitor.
  • If it pages, it has a runbook link and a clear action.
  • If there's no action for a human at night, it opens a ticket instead.
  • If nobody can say what it's for, it gets deleted.

What I learnt

At this scale, the alerting policy matters more than any single monitor. The API audit is what turns the policy into something you can check.

What I do now

  • Audit monitors through the API on a schedule: owner, runbook, routing.
  • Decide page vs ticket when the monitor is created, not during the incident.
  • Delete freely. Monitors are cheap to recreate and expensive to ignore.

stage 1 runs this audit on your monitors → see the engagements

Related

← all posts