What happened
On 4 October 2021, a command run during routine maintenance on Facebook's backbone network took down the connections between their data centres. Their DNS servers, unable to reach the data centres, withdrew their BGP routes as designed, so Facebook, Instagram and WhatsApp vanished from the internet. The tools engineers would normally use to fix it ran over the same network. People had to get physical access to data centres, which took time because the access systems were affected as well. It lasted around six hours.
Facebook's write-up: More details about the October 4 outage.
How you'd spot it from outside
# does the domain resolve at all?
dig +short facebook.com
dig +short NS facebook.com
# ask the authoritative servers directly
dig @a.ns.facebook.com facebook.com +norecurse
# where does the path stop?
mtr -rwc 20 facebook.com
DNS failing everywhere at once, with authoritative servers unreachable, points at routing rather than the servers.
What I took from it
The question I ask every team now: if our network or identity provider is completely down, how do we get in to fix it? Most answers rely on the thing that's down.
What I do now
- Keep a break-glass access path that doesn't depend on the production network or SSO, and test it.
- Keep incident comms (chat, status page, phone tree) independent of your own infrastructure.
- Add an audit check to backbone-scale commands that can't be bypassed by the command itself.
stage 0 asks how you'd get in during the worst case → get the free audit