The page
On 20 October 2025, AWS us-east-1 had a long outage. A race condition in
DynamoDB's automated DNS management left the regional endpoint
(dynamodb.us-east-1.amazonaws.com) with an empty record. Anything that needed
to resolve it couldn't, and the failure spread to other AWS services that
depend on DynamoDB, including EC2 launches and load balancers, for most of the
day.
AWS's summary: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region.
Triage
The useful question during it was not "are we in us-east-1?" but "what of ours touches us-east-1 at all?"
# does the endpoint resolve?
dig +short dynamodb.us-east-1.amazonaws.com
# where our code and config name the region
grep -rn "us-east-1" --include=*.tf --include=*.hcl --include=*.yaml --include=*.json .
# services calling the global STS endpoint (served from us-east-1) instead of a regional one
grep -rn "sts.amazonaws.com" .
aws configure get sts_regional_endpoints
Root cause
Of the surprise, which was ours: a few dependencies we didn't think of as "us-east-1" really were. Global endpoints, a third-party SaaS hosted there, a CI step pulling from a bucket in that region.
The fix
Made the list into tickets: regional STS endpoints everywhere, region-aware config for anything that had one written in, and a note on each third-party dependency of where it's hosted.
What I learnt
Most teams don't know their real dependency graph until part of it fails. A big public outage is the cheapest time to learn it, if you write things down while it's happening.
What I do now
- During any big provider outage, write down everything of yours that wobbled. That list is your real dependency map.
- Use regional endpoints for STS and other services that offer them.
- Ask vendors where they're hosted. Their region is your region.
if you found out the hard way what you depend on → page me