Skip to content
rajesh.dhar
2021 · expert sre · postmortems

Postmortems when the instinct is to find someone to blame

High-severity incidents got blameless postmortems. Building the timeline from systems, not memories, made it easier to talk about decisions instead of people.

1 min readby Rajesh Dhar

The page

After a high-severity incident, the first meeting tends to start with "who pushed that change?" Once people feel blamed, they stop sharing the details you need to actually fix anything.

Triage

I started every postmortem by building the timeline from systems first, before anyone's memory:

# service logs for the window, in order
journalctl -u payments-api --since "2021-03-02 01:00" --until "2021-03-02 03:00" -o short-iso

# what changed in the code in that window
git log --since="2021-03-01 22:00" --until="2021-03-02 03:00" --pretty="%h %ad %an %s" --date=iso

# what changed in Kubernetes (events are short-lived, so capture them early)
kubectl get events -A --sort-by=.lastTimestamp > events-$(date +%F).txt

A timeline from logs and commits is about what happened, not about who remembered what.

Root cause

Of the culture problem: people were explaining incidents in terms of mistakes, which made the mistake the cause. The question that helped was "what made that action seem reasonable at the time?" That usually pointed at missing tests, missing alerts or a confusing tool.

The fix

A postmortem template with a systems-built timeline, contributing factors (not a single root cause), and action items that each have an owner and a date. No names in the "what went wrong" section. High-severity incidents always got one.

What I learnt

Blameless doesn't mean nobody's accountable. Teams are accountable for the action items. Individuals aren't punished for the system's gaps, so they keep telling you where the gaps are.

What I do now

  • Build the timeline from logs, commits and events before the meeting.
  • Ask what made the action reasonable, not who took it.
  • Every action item gets an owner and a due date, and someone checks.

if your postmortems end with a name → page me

Related

← all posts