The page
Every plan and apply against the production workspace started failing:
Error: Error acquiring the state lock
Lock Info:
ID: 2b6a...
Operation: OperationTypeApply
Who: jenkins@build-agent-03
Nobody was running an apply.
Triage
The tempting move is force-unlock straight away. If an apply really is still
running somewhere, that's how you corrupt state. So first, check.
# is anything still running on that agent?
ssh build-agent-03 "pgrep -af terraform"
# the azurerm backend uses a blob lease as the lock
az storage blob show \
--account-name tfstateprod --container-name tfstate \
--name prod.terraform.tfstate \
--query "properties.lease"
# check the last pipeline run that touched it
# (Jenkins console log for the job that owns the lock)
No terraform process on the agent. The lease was held. The last apply job had been aborted mid-run when someone cancelled the pipeline.
Root cause
A cancelled Jenkins job killed Terraform without giving it the chance to release its lock. The apply itself had finished its changes before the cancel, so state was fine. Only the lock was stale.
The fix
terraform force-unlock 2b6a...
terraform plan # should show no changes if the last apply completed
The plan came back clean. Then the pipeline got a timeout that sends Terraform an interrupt and waits, instead of killing it, so it can clean up.
What I learnt
A lock error is the tool doing its job. The fix is quick once you've proved nothing's running. Skipping that check is where state corruption comes from.
What I do now
- Confirm no run is active before
force-unlock, every time. - Let CI stop Terraform gracefully (SIGINT and wait) instead of killing it.
- Run a plan right after unlocking to see whether the last apply finished.
stage 2 is where this kind of pipeline gets built → see the engagements