Skip to content
rajesh.dhar
2023 · lead sre · terraform, terragrunt

A Terragrunt plan that wanted to destroy production

A harmless-looking refactor produced a plan to destroy and recreate a production database. Reading the plan, not the summary line, caught it.

1 min readby Rajesh Dhar

The page

A refactor moved some resources into a shared module. Tests passed, the PR looked clean. The plan said 1 to add, 0 to change, 1 to destroy. The thing it wanted to destroy was a production database.

Triage

The summary line hides a lot. I read the plan as data:

cd live/prod/database
terragrunt plan -out=tfplan

# every resource the plan would delete or replace
terragrunt show -json tfplan \
  | jq -r '.resource_changes[] | select(.change.actions | index("delete")) | "\(.change.actions|join(",")) \(.address)"'

The database showed up as delete then create, under a new address.

Root cause

Moving a resource into a module changes its address in state. Terraform saw the old address gone (destroy) and a new one appear (create). The real resource hadn't changed at all.

The fix

Tell Terraform it's the same resource with a moved block:

moved {
  from = aws_db_instance.main
  to   = module.database.aws_db_instance.main
}

The plan came back with no changes. Then prevent_destroy went on every stateful production resource, so a future mistake fails loudly:

lifecycle {
  prevent_destroy = true
}

What I learnt

The plan is the review. A clean PR diff can still produce a terrible plan, and the summary line doesn't tell you which resources are involved.

What I do now

  • Fail CI if a production plan contains a delete, unless someone approves it explicitly.
  • Use moved blocks for refactors and check for "no changes".
  • prevent_destroy on databases, buckets and anything else with data in it.

stage 0 checks what your pipeline would let through → get the free audit

Related

← all posts