#story
grep -l "story" /portfolio/* · 35 entriesPHP2Shell: patch what you need, delete what you don't
Two critical PHP flaws landed on a Tuesday. One service got patched within the day. A handful of old PHP scripts got something better than a patch: they got switched off for good.
An AI agent's plan that looked right and wasn't
AI agents are in my daily loop for infrastructure code. One produced a tidy change whose plan would have replaced a load balancer. The guardrails are why it didn't.
1 min readus-east-1 DNS: finding out what we actually depended on
When DynamoDB's DNS record in us-east-1 went empty, services that 'didn't use us-east-1' had problems anyway. The outage was a free dependency map.
1 min readA text-to-speech pipeline that had to keep up with the news
Building an AI text-to-speech pipeline on AWS Lambda, and the throttling problem that showed up when a lot of articles landed at once.
1 min readThe DR drill that found what the runbook missed
As DR lead, the drill is where the plan meets reality. One service failed over cleanly on paper and then couldn't reach a dependency still pinned to the old region.
1 min readA Terragrunt plan that wanted to destroy production
A harmless-looking refactor produced a plan to destroy and recreate a production database. Reading the plan, not the summary line, caught it.
1 min readJenkins to GitHub Actions without a release freeze
Leading a CI migration while teams kept shipping. Running both pipelines side by side and comparing their outputs made the cutover boring.
1 min read1,300 monitors, and which ones may wake a human
Running 1,300+ Datadog monitors is mostly about deciding which ones page, which ones open a ticket, and which ones shouldn't exist.
Holding SLOs across 116 services without paging everyone
Owning SLOs, error budgets and burn-rate alerting across 116+ production services, and why burn rate beat thresholds for deciding when to wake someone.
1 min readLog4Shell: finding every Java app we ran
The patch was the easy part. The hard part of Log4Shell was finding every place log4j-core existed, including inside jars inside jars.
1 min readA security group change that cut the app off from its database
A tidy-up of security group rules removed the one that let the app reach its database. Flow logs and CloudTrail found it in minutes.
1 min readRDS out of storage at 3am
A production database ran out of disk overnight. The alert existed, but it fired when there was already almost nothing left.
Postmortems when the instinct is to find someone to blame
High-severity incidents got blameless postmortems. Building the timeline from systems, not memories, made it easier to talk about decisions instead of people.
Capping on-call toil at 25%
On-call time was disappearing into repeat manual work. A 25% cap on toil, measured from tickets, gave the rest back to fixing the causes.
Standing up an SRE practice inside a global fintech
Design authority for SRE patterns and practices, change-management lead for the transformation, and weekly operating reviews of SLAs on live production.
1 min readPods OOMKilled because nobody set limits
One service's memory leak kept getting other pods killed on the same nodes. Nothing had requests or limits set, so the scheduler had no idea.
1 min readA platform upgrade that ran out of capacity halfway
A rolling platform upgrade took cells out one at a time, and halfway through there was nowhere left to put the apps.
A Terraform state lock nobody released
Every Terraform run against production failed with a state lock error. A cancelled pipeline had left the lock behind, and the safe fix was checking before forcing it.
Chef runs that drifted half the fleet
A config fix rolled out through Chef and only half the servers picked it up. chef-client had quietly stopped running on the rest weeks earlier.
1 min readA secret printed in a Jenkins build log
A database password showed up in plain text in a build log. The fix was more than masking it: secrets moved to Vault and every pipeline stopped holding them.
Anomaly detection that flagged every Monday morning
Adding machine-learning analysis to monitoring looked great until it called every Monday 9am an anomaly. The baseline didn't know what a week looked like.
1 min readA pager that cried wolf
On-call was getting paged most nights, nearly always for nothing. Counting where the pages came from showed a handful of alerts doing almost all the damage.
1 min readThe APM agent that slowed down the app it was watching
After rolling out deeper instrumentation, one Java service got slower. The monitoring was the load.
1 min readFour monitoring tools and no single answer
Answering 'is the customer journey healthy?' meant opening four tools. Pulling them into one view changed how fast incidents got diagnosed.
1 min readSynthetic checks caught it before users reported it
A login flow broke overnight. Real-user monitoring was quiet because nobody was using it yet. A scripted check caught it hours before the morning.
Turning a hard escalation into a white paper
The same kind of escalation kept landing on the same few people. Writing up how we diagnosed it, and teaching it in brown bags, spread the load.
1 min readA lab that stopped because snapshots ate the datastore
Every VM on one datastore froze at once. The cause was months of forgotten snapshots growing quietly until the disk was full.
Backups we had never restored
After a very public database loss that year, I tried restoring our own lab backups. Several of them had been empty for months.
1 min readThe change that wasn't in the ticket
A server started failing after a quiet weekend. Nothing was in the change log, but the file system remembered what had changed.
The certificate nobody owned
An internal service broke on a Friday because its certificate expired. It had been renewed by hand once, years earlier, by someone who'd moved on.
1 min readHeartbleed week: patching everything at once
Heartbleed meant finding every OpenSSL we ran, patching it, restarting what still had the old library loaded, and then replacing the keys.
Keeping a customer calm through a three-day incident
The technical fix took three days. What kept the customer on side was a fixed update schedule and a timeline they could forward to their bosses.
Reproduce it in the lab before you touch production
A corporate customer's issue only happened on their system. Rebuilding their setup in the lab found it, without anyone experimenting on production.
1 min read"The internet is down" tickets that were really DNS
Half the connectivity tickets I took said the internet was down. Most of the time the line was fine and name resolution wasn't.
1 min readThe database that fell over at 9am every day
My first infrastructure job. A client's system went down at the same time most mornings, and chasing the bottleneck live cut their downtime by 83%.
1 min read