The problem
Some infrastructure was in Terraform. A lot of it wasn't, and was changed in the cloud portal or by hand over SSH. When something broke, the first hour went on working out what had changed.
How I measured it
# resources in the subscription that Terraform doesn't know about
az resource list --query "[].id" -o tsv | sort > cloud.txt
terraform state list | xargs -I{} terraform state show -no-color {} 2>/dev/null \
| awk '/^ *id *=/ {gsub(/"/,"",$3); print $3}' | sort > managed.txt
comm -23 cloud.txt managed.txt | wc -l
# drift on what Terraform does manage
terraform plan -detailed-exitcode -no-color > /dev/null; echo "exit $?" # 2 means drift
A large share of resources weren't managed, and the managed ones had drift.
Why it kept happening
The portal was quicker for a one-off change, every time. Code review felt like overhead for "just one setting", so the quick way won.
The process change
Production infrastructure changes go through a pull request, with no exceptions for small ones:
- Change in Terraform or a Chef cookbook, in Git.
- Jenkins runs
terraform plan(or a Chef--why-run) and posts the result on the PR. - SonarQube checks the code. A quality gate failure blocks the merge.
- One reviewer approves. Jenkins applies after merge.
Portal write access for production was reduced to a small break-glass group, and every break-glass change had to be brought back into code within a day.
What I learnt
It only stuck once the PR path was nearly as fast as the portal. The plan on the PR did most of the work: reviewers could see what the code would do.
What I do now
- Every production change in code, reviewed, applied by the pipeline.
- Show the plan on the PR.
- Keep a break-glass path, and require it to be reconciled into code.
stage 2 sets this up for your team → see the engagements