The page
When I joined as Expert SRE and Design Authority for SRE Patterns & Practices, the job was to write down how SRE should work across the platforms, and then get teams to actually work that way. Production deployments had SLAs, and the first question I asked was how each one was measured.
Triage
For a few, the honest answer was that it wasn't measured continuously. So I started with the simplest possible check from the access logs:
# share of 5xx responses in a combined-format access log (status is field 9)
awk '$9 ~ /^5/ {e++} END {printf "error rate: %.3f%%\n", 100*e/NR}' access.log
# requests slower than 2 seconds, if the log has request time as the last field
awk '$NF > 2.0 {s++} END {printf "slow: %.3f%%\n", 100*s/NR}' access.log
Rough numbers, but real ones, and they started the right conversation.
Root cause
An SLA on paper is a promise. Without a measurement behind it, nobody knows if it's being kept until a customer says it isn't.
The fix
I led the SRE transformation program and was its change-management lead too, because most of a change like this is teams changing how they work. The pieces:
- Weekly operating reviews, looking at each service's reliability against its SLA.
- SLA reviews on live production deployments, so each one had a measurement and an owner.
- On-call toil capped at 25%.
- Blameless postmortems for high-severity incidents.
What I learnt
The technical part (metrics, dashboards, alerts) is the easier half. The other half is a weekly habit of looking at the numbers together, kept up long enough that it becomes how the place works.
What I do now
- Ask "how is this measured?" before "what's the target?"
- Start with rough numbers from logs you already have. Perfect SLIs can come later.
- Put the review on the calendar every week, and keep it.
stage 1 is where this kind of work starts → see the engagements