The page
A corporate customer had a production issue that wasn't going to be fixed in an afternoon. It ended up taking three days. I was their main contact for all of it.
Triage
The technical work ran in the background with the product team. My part was keeping a clean record of what had happened, so every update was accurate.
# pull every error from all nodes into one ordered timeline (UTC)
grep -h "ERROR" node*/app.log | sort -k1,2 > timeline.txt
head -50 timeline.txt
# when did it start? first occurrence of the key error
grep -m1 -h "ConnectionPool exhausted" node*/app.log
# write down when each step happened, in UTC
date -u +"%Y-%m-%dT%H:%M:%SZ"
Root cause
Under the hood it was a resource leak that only showed up under their load pattern, and it took the product team a while to confirm. The thing that was actually going wrong on the customer side was silence. In the first few hours, before I took it over, they'd gone long stretches without hearing anything, and they'd started assuming nobody was working on it.
The fix
I set an update schedule and stuck to it, even when there was nothing new: what we know, what we're trying, when the next update is. I kept one running timeline document that they could forward internally without rewriting.
The leak was patched on day three.
What I learnt
During a long incident, the customer can't see the work. All they see is the updates. "No news yet, next update at 3pm" keeps trust in a way that a perfect update twelve hours later doesn't.
What I do now
- Promise a time for the next update and keep it.
- Keep one timeline, in UTC, that anyone can read cold.
- Separate "what we know" from "what we think". People forward these.
if your incidents go quiet for hours → page me