What happened
On 30 June 2012 a leap second was added at 23:59:60 UTC. A bug in how the Linux kernel handled timers after the leap second left futex-heavy programs waking up over and over. Java and MySQL processes on a lot of servers went to full CPU at the same moment, and several big websites had problems that night.
The odd part was the fix. Setting the system clock again, even to the current time, cleared the bad timer state:
date -s "$(date)"
How you'd spot it
# per-thread CPU, which shows a process spinning rather than working
top -H
# the kernel logs the leap second when it inserts it
dmesg | grep -i "leap second"
# what time source the box is using and how far off it is
ntpq -p
chronyc tracking # on newer systems using chrony
A sudden CPU spike on many unrelated hosts at exactly midnight UTC points at time, not load.
What I took from it
Time is a dependency like any other, and it breaks in ways that hit every server at once. You don't get the usual protection of one host failing while the others carry on.
Since then, most big providers smear the leap second instead of adding it in one step: the clock runs a tiny bit slow over many hours, so no system ever sees 23:59:60. AWS Time Sync and Google's public NTP both do this.
What I do now
- Treat simultaneous failures across unrelated hosts as a shared cause: time, DNS, a certificate, a config push.
- Use a time source that smears leap seconds, and don't mix smeared and non-smeared sources.
- Keep "set the clock" style recovery steps in the runbook when the vendor gives one. They feel too simple to be right until you need them.
stage 0 looks for the shared causes first → get the free audit