Skip to content
rajesh.dhar
2014 · senior technical analyst · openssl

Heartbleed week: patching everything at once

Heartbleed meant finding every OpenSSL we ran, patching it, restarting what still had the old library loaded, and then replacing the keys.

1 min readby Rajesh Dhar

The page

On 7 April 2014 Heartbleed was published: a bug in OpenSSL's heartbeat extension that let anyone read memory from a server, including private keys. Versions 1.0.1 to 1.0.1f were affected, 1.0.1g fixed it. Every system with TLS was suddenly a question mark.

Triage

First, what's vulnerable:

# version of the library, and the build date
openssl version -a

# what the package manager thinks is installed
rpm -q openssl          # RHEL / CentOS
dpkg -l openssl         # Debian / Ubuntu

# check a live endpoint from outside
nmap -p 443 --script ssl-heartbleed host.example.com

The trap came after the upgrade. Services that were already running still had the old library loaded in memory. The package was patched and the process wasn't.

# processes still mapping a deleted (old) libssl or libcrypto
lsof -n | grep -E "libssl|libcrypto" | grep DEL

Root cause

A missing bounds check in the heartbeat code. The server trusted the length the client sent and returned that much memory.

The fix

Patch, then restart every service that lsof showed holding the old library, then check again from outside with nmap. After that, the part people skipped: new private keys and new certificates, because the old keys might already have leaked. Patching alone didn't undo that.

What I learnt

"Patched" has three steps: the package, the running processes, and anything the bug could have exposed. Most teams stopped after the first one.

What I do now

  • Keep an inventory of what runs which TLS library, before the next one.
  • Verify a fix from the outside, the way an attacker would see it.
  • After a key-exposure bug, rotate the keys. Patching doesn't take back what already leaked.

Details: heartbleed.com


stage 0 checks what's exposed before anything else → get the free audit

Related

← all posts