What happened
On 21 October 2016 Dyn, a large managed DNS provider, was hit by a DDoS from the Mirai botnet, mostly hijacked cameras and home routers. For hours, users in large parts of the US couldn't resolve names for big sites that used Dyn. Their servers were up. Nobody could find them.
Dyn's summary: Dyn Analysis Summary of Friday October 21 Attack.
How you'd spot it
# who is authoritative for the domain?
dig +short NS example.com
# ask each authoritative server directly, skipping caches
dig @ns1.provider-a.net example.com A +norecurse
dig @ns1.provider-b.net example.com A +norecurse
# follow the whole chain from the root
dig example.com +trace
# how long resolvers will cache the answer (second column)
dig example.com A +noall +answer
If every NS record points at one provider and that provider stops answering, you're down as soon as the TTL runs out.
What I took from it
DNS is the first dependency of everything, and it's often the one with no redundancy. A second provider, with records kept in sync, would have turned that day into a non-event for a lot of companies.
TTLs cut both ways. Long TTLs keep answers cached through a short provider outage. Short TTLs let you move traffic fast. Pick on purpose.
What I do now
- Use two authoritative DNS providers for anything customer-facing, synced from one source of truth (Terraform works well for this).
- Monitor resolution from outside your network, against each provider.
- Know your TTLs and why they're set that way.
if one vendor going down takes you with it → page me