The page
We pushed a config fix through Chef and expected the problem to go away. It went away on about half the servers. The other half still had the old setting.
Triage
# when did each node last check in, and with what run list?
knife status --run-list | sort -k2
# only the web role, to narrow it
knife status "role:web"
# is the chef-client service even running on them?
knife ssh "role:web" "sudo systemctl is-active chef-client"
# what would change on a node, without changing it
sudo chef-client --why-run
knife status showed half the web nodes hadn't checked in for weeks.
Root cause
An OS update a few weeks earlier had restarted those servers, and chef-client hadn't been enabled to start on boot there. Nobody noticed because nothing alerted when a node stopped converging. Meanwhile people had made a few manual changes on those boxes, so they'd drifted.
The fix
Enabled and started chef-client everywhere, ran --why-run first to see what
it would undo (some of the manual changes were real fixes that needed to go
into the cookbook), then let it converge. Added an alert for any node that
hadn't checked in within a couple of hours.
What I learnt
Configuration management only works if it keeps running. A node that stops converging looks healthy right up to the moment you need it to change.
What I do now
- Alert on last check-in time for every managed node.
- Run a dry run before converging a node that's been out of contact for a while.
- Fold manual fixes back into code, or they'll be undone the next time it runs.
stage 1 looks for drift like this → see the engagements