The page
During a platform upgrade, apps started reporting instances down. The upgrade was rolling, one cell at a time, which should have been safe.
Triage
# platform VMs and their health, per instance group
bosh -d cf instances --ps
# CPU, memory and disk on the cells
bosh -d cf vms --vitals
# from the app side: which instances are down, and why
cf app my-service
cf events my-service
The cells that were still up were close to full on memory. New app instances couldn't be placed, so they sat waiting.
Root cause
The foundation had grown since the last upgrade. With one cell drained for the upgrade, the remaining cells no longer had room for everything. The math worked on the day the platform was sized, and nobody had redone it.
The fix
Paused the upgrade, added cells, and resumed. Then a standing rule: keep enough spare capacity to lose at least one cell (more for bigger foundations), and check it before every upgrade.
What I learnt
Rolling upgrades are only safe if you can afford to lose what you're rolling. Headroom is a requirement of the upgrade, not a nice-to-have.
What I do now
- Before any rolling change, check you can lose one unit and still fit everything.
- Track capacity as a trend, not only on the day you size it.
- Know how to pause a rollout halfway, and practise it.
if your upgrades make you nervous → page me