2026
PHP2Shell: patch what you need, delete what you don't
Two critical PHP flaws landed on a Tuesday. One service got patched within the day. A handful of old PHP scripts got something better than a patch: they got switched off for good.
An AI agent's plan that looked right and wasn't
AI agents are in my daily loop for infrastructure code. One produced a tidy change whose plan would have replaced a load balancer. The guardrails are why it didn't.
1 min read
2025
A generated config file that doubled in size
What I took from Cloudflare's November 2025 outage: a permissions change made a generated file double in size, and the proxy crashed on a limit nobody expected to hit.
1 min readus-east-1 DNS: finding out what we actually depended on
When DynamoDB's DNS record in us-east-1 went empty, services that 'didn't use us-east-1' had problems anyway. The outage was a free dependency map.
1 min readRules for AI agents touching infrastructure
AI agents are in the daily loop for triage, infrastructure code and postmortems. Writing the rules down as repo policy made using them safe and repeatable.
1 min read
2024
A bad content update, everywhere, at once
What I took from the July 2024 CrowdStrike outage: a configuration update, not a code release, crashed millions of Windows machines because it went to all of them at once.
Postmortem action items that actually close
Good postmortems, and action items that sat open for months. Tracking them like real work, with owners, dates and a monthly review, got them closed.
A text-to-speech pipeline that had to keep up with the news
Building an AI text-to-speech pipeline on AWS Lambda, and the throttling problem that showed up when a lot of articles landed at once.
1 min read
2023
The DR drill that found what the runbook missed
As DR lead, the drill is where the plan meets reality. One service failed over cleanly on paper and then couldn't reach a dependency still pinned to the old region.
1 min readA Terragrunt plan that wanted to destroy production
A harmless-looking refactor produced a plan to destroy and recreate a production database. Reading the plan, not the summary line, caught it.
1 min readDR drills on the calendar, not after the outage
As DR lead, the plan is only as good as the last time it was run. Putting drills on the calendar, and tracking every finding to closure, kept it honest.
1 min readJenkins to GitHub Actions without a release freeze
Leading a CI migration while teams kept shipping. Running both pipelines side by side and comparing their outputs made the cutover boring.
1 min read
2022
1,300 monitors, and which ones may wake a human
Running 1,300+ Datadog monitors is mostly about deciding which ones page, which ones open a ticket, and which ones shouldn't exist.
What happens when the error budget runs out
SLOs and error budgets only change behaviour if something happens when the budget is gone. A written policy, agreed in advance, made that decision easy.
1 min readHolding SLOs across 116 services without paging everyone
Owning SLOs, error budgets and burn-rate alerting across 116+ production services, and why burn rate beat thresholds for deciding when to wake someone.
1 min read
2021
Log4Shell: finding every Java app we ran
The patch was the easy part. The hard part of Log4Shell was finding every place log4j-core existed, including inside jars inside jars.
1 min readLocked out by our own outage
What I took from Facebook's October 2021 outage: a backbone change took the network down, and the tools to fix it were on the same network.
1 min readA security group change that cut the app off from its database
A tidy-up of security group rules removed the one that let the app reach its database. Flow logs and CloudTrail found it in minutes.
1 min readA change board that approved everything
Every change went to the same weekly board, and the board approved nearly all of them. Tiering changes by risk let the board spend its time on the risky ones.
1 min readRDS out of storage at 3am
A production database ran out of disk overnight. The alert existed, but it fired when there was already almost nothing left.
One incident commander, not five people typing
During big incidents, everyone jumped in and nobody was in charge. A named incident commander on the 24/7 rotation made the response calmer and faster.
1 min readPostmortems when the instinct is to find someone to blame
High-severity incidents got blameless postmortems. Building the timeline from systems, not memories, made it easier to talk about decisions instead of people.
Capping on-call toil at 25%
On-call time was disappearing into repeat manual work. A 25% cap on toil, measured from tickets, gave the rest back to fixing the causes.
2020
A weekly operating review people actually attend
Running weekly operating reviews on live production. The agenda that kept it short, useful and attended, and the numbers prepared before anyone joined.
Standing up an SRE practice inside a global fintech
Design authority for SRE patterns and practices, change-management lead for the transformation, and weekly operating reviews of SLAs on live production.
1 min readPods OOMKilled because nobody set limits
One service's memory leak kept getting other pods killed on the same nodes. Nothing had requests or limits set, so the scheduler had no idea.
1 min readThe production readiness review
New services reached the platform missing basics: no limits, no health checks, no runbook. A short readiness review before first traffic caught them while they were cheap to fix.
1 min readA platform upgrade that ran out of capacity halfway
A rolling platform upgrade took cells out one at a time, and halfway through there was nowhere left to put the apps.
2019
A Terraform state lock nobody released
Every Terraform run against production failed with a state lock error. A cancelled pipeline had left the lock behind, and the safe fix was checking before forcing it.
Chef runs that drifted half the fleet
A config fix rolled out through Chef and only half the servers picked it up. chef-client had quietly stopped running on the rest weeks earlier.
1 min readInfrastructure changes go through pull requests
Production infrastructure was changed in portals and over SSH. Moving it into code with reviewed pull requests made every change visible and reversible.
1 min readOne bad rule, deployed everywhere at once
What I took from Cloudflare's July 2019 outage: a single regular expression, pushed globally in one go, and why config changes need the same staged rollout as code.
A secret printed in a Jenkins build log
A database password showed up in plain text in a build log. The fix was more than masking it: secrets moved to Vault and every pipeline stopped holding them.
An alert can't page until it has an owner and a runbook
Anyone could add an alert that paged on-call. A simple admission rule (owner, runbook, action) stopped the noise at the door.
2018
Anomaly detection that flagged every Monday morning
Adding machine-learning analysis to monitoring looked great until it called every Monday 9am an anomaly. The baseline didn't know what a week looked like.
1 min readA pager that cried wolf
On-call was getting paged most nights, nearly always for nothing. Counting where the pages came from showed a handful of alerts doing almost all the damage.
1 min readThe APM agent that slowed down the app it was watching
After rolling out deeper instrumentation, one Java service got slower. The monitoring was the load.
1 min readFour monitoring tools and no single answer
Answering 'is the customer journey healthy?' meant opening four tools. Pulling them into one view changed how fast incidents got diagnosed.
1 min readNo service goes live without monitoring
New services kept reaching production with no APM, no synthetics and no alerts. A short onboarding checklist made monitoring part of launching.
1 min readSynthetic checks caught it before users reported it
A login flow broke overnight. Real-user monitoring was quiet because nobody was using it yet. A scripted check caught it hours before the morning.
2017
Turning a hard escalation into a white paper
The same kind of escalation kept landing on the same few people. Writing up how we diagnosed it, and teaching it in brown bags, spread the load.
1 min readA lab that stopped because snapshots ate the datastore
Every VM on one datastore froze at once. The cause was months of forgotten snapshots growing quietly until the disk was full.
Backups we had never restored
After a very public database loss that year, I tried restoring our own lab backups. Several of them had been empty for months.
1 min readShadow, co-lead, lead
Junior specialists were either kept off hard escalations or thrown in alone. A three-step path in between made them ready faster and safer.
1 min readOne typo, a whole region
What I took from the February 2017 S3 outage: a routine command with one wrong input, and the guardrails that should sit around every runbook command.
1 min read
2016
Brown bags instead of the same three experts
Hard escalations queued up behind the same three people. A regular brown-bag session, built from real cases, spread what they knew.
1 min readOur DNS provider went down and took us with it
What I took from the October 2016 Dyn attack: when one managed DNS provider fails, every site that relies only on it disappears, however healthy its servers are.
1 min readThe day a dependency disappeared from the registry
What I took from left-pad: an eleven-line package was unpublished, builds broke everywhere, and a private mirror would have made it a non-event.
1 min read
2015
The change that wasn't in the ticket
A server started failing after a quiet weekend. Nothing was in the change log, but the file system remembered what had changed.
"Can't reproduce" stopped being an answer
Too many tickets closed as 'can't reproduce' came back. Requiring reproduction notes before closing made the in-house lab part of the process.
1 min read
2014
The certificate nobody owned
An internal service broke on a Friday because its certificate expired. It had been renewed by hand once, years earlier, by someone who'd moved on.
1 min readHeartbleed week: patching everything at once
Heartbleed meant finding every OpenSSL we ran, patching it, restarting what still had the old library loaded, and then replacing the keys.
Updates on a clock, not when there's news
Customers during an incident judged us by how often they heard from us. A fixed update schedule turned that into something the team could keep.
2013
Keeping a customer calm through a three-day incident
The technical fix took three days. What kept the customer on side was a fixed update schedule and a timeline they could forward to their bosses.
Escalations that lost context at every handoff
Every time an escalation changed hands, the next person started from scratch. A short written handoff fixed it without slowing anyone down.
Reproduce it in the lab before you touch production
A corporate customer's issue only happened on their system. Rebuilding their setup in the lab found it, without anyone experimenting on production.
1 min read
2012
One extra second that pinned every CPU
What I took from the June 2012 leap second: a kernel timer bug, Java and MySQL spinning at full CPU, and a fix that was just setting the clock.
2011
When the storage failure took the backups with it
What I took from the April 2011 EBS outage: a network change, a re-mirroring storm, and why one availability zone wasn't as isolated as people assumed.
2010
"The internet is down" tickets that were really DNS
Half the connectivity tickets I took said the internet was down. Most of the time the line was fine and name resolution wasn't.
1 min readThe database that fell over at 9am every day
My first infrastructure job. A client's system went down at the same time most mornings, and chasing the bottleneck live cut their downtime by 83%.
1 min read