The page
One morning every VM on one of our lab datastores was paused, all with the same message about no more space for the virtual disk. Nothing new had been deployed.
Triage
# on the ESXi host: how full are the datastores?
df -h
# snapshot delta files, which grow with every write after the snapshot
find /vmfs/volumes/ -name "*-0000*.vmdk" -exec ls -lh {} \;
# list VMs, then the snapshot tree for one
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/snapshot.get <vmid>
From PowerCLI, the whole picture in one line:
Get-VM | Get-Snapshot | Select-Object VM, Name, Created, SizeGB | Sort-Object SizeGB -Descending
Snapshots from months earlier, some of them huge.
Root cause
People took a snapshot before testing something, which is good practice, and then never deleted it. A snapshot keeps every change since it was taken in a delta file, so the files grew with every write. The datastore filled, and VMs that couldn't write paused.
The fix
Free some space first so the VMs could resume, then consolidate the old snapshots one VM at a time. Deleting large snapshots needs room to merge, so the order mattered.
vim-cmd vmsvc/snapshot.removeall <vmid>
After that, a weekly report of snapshots older than a few days, sent to the person who took them.
What I learnt
Snapshots are for short-term rollback. Left alone they become slow-growing storage leaks that hit every VM on the datastore at once.
What I do now
- Put an age limit on snapshots and report on anything older.
- Alert on datastore free space well before it's critical.
- Name snapshots with an owner and a date, so the report has someone to send to.
if capacity surprises keep happening → page me