The page
On a Friday afternoon, integrations between two internal systems started failing with handshake errors. Nothing had been deployed. The error message said "certificate has expired", which is the kind of clue you only appreciate afterwards.
Triage
# what certificate is the server presenting, and when does it expire?
openssl s_client -connect api.internal.example.com:443 \
-servername api.internal.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates
# for a certificate file on disk
openssl x509 -in /etc/pki/tls/certs/api.pem -noout -enddate
# exit code 1 if it expires within 30 days (2592000 seconds)
openssl x509 -in /etc/pki/tls/certs/api.pem -noout -checkend 2592000
notAfter was that morning.
Root cause
The certificate was issued years before by someone who'd since moved teams. It wasn't in any inventory, wasn't on anyone's calendar, and there was no alert on expiry. The "owner" field in our heads was empty.
The fix
A new certificate that afternoon, then a list. I scanned the internal endpoints we knew about and wrote down every certificate, its expiry date and a named owner.
for h in $(cat hosts.txt); do
printf "%s " "$h"
openssl s_client -connect "$h:443" -servername "$h" </dev/null 2>/dev/null \
| openssl x509 -noout -enddate
done
What I learnt
Certificates are outages scheduled in advance. The date is printed right on them. They only catch you out because nobody's looking at the date.
What I do now
- Every certificate gets a named owner and an expiry alert at 30 and 7 days.
- Automate renewal wherever the platform allows it.
- Scan the endpoints you think you know, because there's always one you don't.
this is the kind of gap stage 1 maps → see the engagements