#process
grep -l "process" /portfolio/* · 16 entriesRules for AI agents touching infrastructure
AI agents are in the daily loop for triage, infrastructure code and postmortems. Writing the rules down as repo policy made using them safe and repeatable.
1 min readPostmortem action items that actually close
Good postmortems, and action items that sat open for months. Tracking them like real work, with owners, dates and a monthly review, got them closed.
DR drills on the calendar, not after the outage
As DR lead, the plan is only as good as the last time it was run. Putting drills on the calendar, and tracking every finding to closure, kept it honest.
1 min readWhat happens when the error budget runs out
SLOs and error budgets only change behaviour if something happens when the budget is gone. A written policy, agreed in advance, made that decision easy.
1 min readA change board that approved everything
Every change went to the same weekly board, and the board approved nearly all of them. Tiering changes by risk let the board spend its time on the risky ones.
1 min readOne incident commander, not five people typing
During big incidents, everyone jumped in and nobody was in charge. A named incident commander on the 24/7 rotation made the response calmer and faster.
1 min readA weekly operating review people actually attend
Running weekly operating reviews on live production. The agenda that kept it short, useful and attended, and the numbers prepared before anyone joined.
The production readiness review
New services reached the platform missing basics: no limits, no health checks, no runbook. A short readiness review before first traffic caught them while they were cheap to fix.
1 min readInfrastructure changes go through pull requests
Production infrastructure was changed in portals and over SSH. Moving it into code with reviewed pull requests made every change visible and reversible.
1 min readAn alert can't page until it has an owner and a runbook
Anyone could add an alert that paged on-call. A simple admission rule (owner, runbook, action) stopped the noise at the door.
No service goes live without monitoring
New services kept reaching production with no APM, no synthetics and no alerts. A short onboarding checklist made monitoring part of launching.
1 min readShadow, co-lead, lead
Junior specialists were either kept off hard escalations or thrown in alone. A three-step path in between made them ready faster and safer.
1 min readBrown bags instead of the same three experts
Hard escalations queued up behind the same three people. A regular brown-bag session, built from real cases, spread what they knew.
1 min read"Can't reproduce" stopped being an answer
Too many tickets closed as 'can't reproduce' came back. Requiring reproduction notes before closing made the in-house lab part of the process.
1 min readUpdates on a clock, not when there's news
Customers during an incident judged us by how often they heard from us. A fixed update schedule turned that into something the team could keep.
Escalations that lost context at every handoff
Every time an escalation changed hands, the next person started from scratch. A short written handoff fixed it without slowing anyone down.