The problem
During incidents, we updated customers when we had news. Sometimes that meant three updates in an hour and then nothing for six. The quiet stretches were when customers escalated to their account manager, and then to ours.
How I measured it
# gaps between customer-facing updates on an incident ticket (timestamps in UTC, one per line)
awk 'NR>1 {print ($1-prev)/3600 " hours"} {prev=$1}' \
<(grep "public_update" ticket-4821.log | cut -d' ' -f1 | xargs -I{} date -u -d {} +%s)
# escalations to account managers, by incident
grep -c "escalated to account" incidents/*.log | sort -t: -k2 -rn | head
The incidents that drew escalations were the ones with the longest gaps between updates, not the longest incidents.
Why it kept happening
"Update when there's news" sounds reasonable, and it leaves the timing to whoever's busiest. During an incident, that's everyone.
The process change
Update intervals set by severity, agreed with the customer at the start:
- Highest severity: every hour, even if there's nothing new.
- Lower severities: every business day.
Every update had the same shape: what we know, what we're doing, when the next update is. The person running the incident set a timer. "No change, next update at 3pm" counted as an update.
What I learnt
Customers can't see the work, only the updates. A predictable schedule lets them stop watching their inbox and tell their own bosses when to expect news.
What I do now
- Agree the update interval at the start of the incident.
- Always end an update with the time of the next one.
- Send the update on time even when there's nothing new to say.
if your customers find out from Twitter first → page me