The problem
We had SLOs and error budgets. When a budget ran out, nothing in particular happened. Feature work carried on, and the budget kept going negative.
How I measured it
# SLO status and remaining error budget for each SLO, last 30 days (Datadog API)
FROM=$(date -u -d '-30 days' +%s); TO=$(date -u +%s)
for id in $(jq -r '.[]' slo-ids.json); do
curl -s "https://api.datadoghq.com/api/v1/slo/$id/history?from_ts=$FROM&to_ts=$TO" \
-H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
| jq -r '[.data.slo.name, .data.overall.sli_value, .data.overall.error_budget_remaining["30d"]] | @tsv'
done
Some services were well past their budget and still shipping at the usual pace.
Why it kept happening
Without an agreed rule, slowing down is a fight every time, and it happens at the worst moment: right after an incident, with a deadline coming. Shipping wins those fights.
The process change
An error budget policy, agreed with product and engineering leads in a calm week, written down:
- Budget above 50%: ship normally.
- Below 25%: releases need a rollback plan reviewed by SRE.
- Exhausted: reliability work comes first for that service until the budget recovers. Releases limited to fixes.
- Exceptions: allowed, with sign-off from the product owner, and written down.
The policy applies per service, so one struggling service doesn't freeze everyone.
What I learnt
The policy's real value was that the decision was already made. When a budget ran out, nobody had to argue for slowing down. They just pointed at the page.
What I do now
- Write the error budget policy before you need it, with product in the room.
- Apply it per service.
- Allow exceptions, but make them explicit and signed off.
stage 1 sets SLOs and the policy behind them → see the engagements