#aws
grep -l "aws" /portfolio/* · 9 entriesus-east-1 DNS: finding out what we actually depended on
When DynamoDB's DNS record in us-east-1 went empty, services that 'didn't use us-east-1' had problems anyway. The outage was a free dependency map.
1 min readA text-to-speech pipeline that had to keep up with the news
Building an AI text-to-speech pipeline on AWS Lambda, and the throttling problem that showed up when a lot of articles landed at once.
1 min readThe DR drill that found what the runbook missed
As DR lead, the drill is where the plan meets reality. One service failed over cleanly on paper and then couldn't reach a dependency still pinned to the old region.
1 min readDR drills on the calendar, not after the outage
As DR lead, the plan is only as good as the last time it was run. Putting drills on the calendar, and tracking every finding to closure, kept it honest.
1 min readA security group change that cut the app off from its database
A tidy-up of security group rules removed the one that let the app reach its database. Flow logs and CloudTrail found it in minutes.
1 min readA change board that approved everything
Every change went to the same weekly board, and the board approved nearly all of them. Tiering changes by risk let the board spend its time on the risky ones.
1 min readRDS out of storage at 3am
A production database ran out of disk overnight. The alert existed, but it fired when there was already almost nothing left.
One typo, a whole region
What I took from the February 2017 S3 outage: a routine command with one wrong input, and the guardrails that should sit around every runbook command.
1 min readWhen the storage failure took the backups with it
What I took from the April 2011 EBS outage: a network change, a re-mirroring storm, and why one availability zone wasn't as isolated as people assumed.