The page
Monday morning, an application server that had been fine on Friday was throwing errors. The change log for the weekend was empty. Everyone agreed nothing had changed.
Triage
When people say nothing changed, I ask the server instead.
# config files modified in the last 3 days
find /etc /opt/app/conf -type f -mtime -3 -ls
# who logged in, and when
last -F | head -20
# what they ran with sudo (RHEL: /var/log/secure, Debian: /var/log/auth.log)
grep -i sudo /var/log/secure | tail -50
# files from packages that no longer match what was installed
rpm -Va | grep -v "^\.\.\.\.\.\.\.\.T"
find turned up a JVM settings file edited on Saturday. sudo logs showed who.
Root cause
Someone had been asked to "quickly" raise a memory setting over the weekend and did it by hand. They had a typo in one flag, the service restarted on Sunday's maintenance reboot, and the new setting kicked in then. No ticket, because it was "just a config tweak".
The fix
Revert the file and restart, done in minutes. The longer conversation was about how a one-line change got made without a record, and that turned into keeping config under version control with a diff for every edit.
diff -u app.conf.bak app.conf
What I learnt
Change control is what lets you answer "what changed?" in one minute instead of one hour. The person who made the change wasn't careless. The process made the untracked way the easy way.
What I do now
- Ask the system what changed as well as the people.
- Put config in version control so every edit has a diff and a name.
- Make the tracked path the easier one, or people will route around it.
if "nothing changed" keeps being wrong → page me