The page
A platform with more than 116 production services, and one question that matters at 3am: are users being hurt, and how fast? Threshold alerts on CPU or latency can't answer that. They fire on anything unusual, whether or not anyone notices.
Triage
I own the SLOs and error budgets. The first check on any service is what its SLO says right now:
# list SLOs and their current status through the Datadog API
curl -s "https://api.datadoghq.com/api/v1/slo?limit=100" \
-H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
| jq -r '.data[] | [.name, .type, (.thresholds[0].target|tostring)] | @tsv'
Root cause
Of the alerting problem in general: an alert that fires on a symptom users don't feel trains people to ignore it. An alert tied to the error budget fires when users really are losing service, at a rate that matters.
The fix
Alerting runs on burn rate. A burn-rate monitor fires when the budget is being used faster than the SLO allows, over two windows so it's both quick and not jumpy. In Datadog that looks like:
burn_rate("<SLO_ID>").over("30d").long_window("1h").short_window("5m") > 14.4
At 14.4×, a 30-day budget would be gone in about two days, which is worth waking someone up for. A slower burn goes to a ticket.
What I learnt
Burn rate puts the user's experience in charge of the pager. It also gives a clear answer to "is this worth stopping for?", which thresholds never do.
What I do now
- Page on fast burn, ticket on slow burn.
- Give every SLO an owner and look at budgets weekly, not only during incidents.
- Start with one availability and one latency SLO per service. Add more only when a question needs one.
this is the work stage 0 samples → get the free audit · see a sample report