The page
A certain kind of performance escalation kept coming in, and it kept going to the same two or three senior people because they were the only ones who knew where to look. Every time, the junior specialists waited for us.
Triage
The diagnosis always followed the same steps. So I started collecting them, starting with the bundle we asked every customer for:
# collect a diagnostic bundle in one go
TS=$(date +%F-%H%M)
mkdir -p /tmp/diag-$TS
uname -a > /tmp/diag-$TS/system.txt
java -version 2>> /tmp/diag-$TS/system.txt
df -h > /tmp/diag-$TS/disk.txt
free -m > /tmp/diag-$TS/memory.txt
jstack <pid> > /tmp/diag-$TS/threads-1.txt
sleep 10
jstack <pid> > /tmp/diag-$TS/threads-2.txt
cp /opt/app/logs/*.log /tmp/diag-$TS/
tar czf diag-$(hostname)-$TS.tgz -C /tmp diag-$TS
Two thread dumps a few seconds apart show what's stuck versus what's just busy.
Root cause
It was a knowledge problem. The steps weren't hard. They just weren't written down anywhere a new person would find them.
The fix
I wrote it up as a white paper: the symptoms, the bundle, how to read two thread dumps side by side, and the three causes we usually found. Then I ran brown-bag sessions walking through real (anonymized) cases, and had juniors lead the next few escalations with me watching.
What I learnt
If only one person can solve a class of problem, that person is a single point of failure. Writing it down was the cheapest redundancy I've ever added.
What I do now
- When I solve the same thing twice, I write it down the second time.
- Teach with real cases. Abstract examples don't stick.
- Let people lead with backup before they lead alone.
this is how stage 3 hands work over → see the engagements