The page
Paged at 3am: the application couldn't write to its database. The RDS instance
had gone into storage-full.
Triage
# instance state and storage settings
aws rds describe-db-instances --db-instance-identifier prod-db \
--query "DBInstances[0].[DBInstanceStatus,AllocatedStorage,MaxAllocatedStorage]"
# free storage over the last day, 5-minute minimums
aws cloudwatch get-metric-statistics --namespace AWS/RDS \
--metric-name FreeStorageSpace \
--dimensions Name=DBInstanceIdentifier,Value=prod-db \
--start-time "$(date -u -d '-24 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 300 --statistics Minimum --output table
# what's using the space, on MySQL
mysql -h prod-db... -e "SELECT table_schema, ROUND(SUM(data_length+index_length)/1024/1024) AS mb
FROM information_schema.tables GROUP BY table_schema ORDER BY mb DESC;"
mysql -h prod-db... -e "SHOW VARIABLES LIKE 'general_log%';"
Free space had been dropping in a straight line for days, then faster that evening. Data size hadn't changed much.
Root cause
The general query log had been switched on (writing to a table) to debug something and never switched off. It logged every query, and it grew until it filled the disk. The free-storage alarm was set at a level that left minutes, not hours.
The fix
Increased storage to get writes going again, turned the general log off, cleared the log table, and turned on storage autoscaling with a ceiling:
aws rds modify-db-instance --db-instance-identifier prod-db \
--max-allocated-storage 1000 --apply-immediately
Then reset the alarm to fire on the trend (days of headroom left), not on a nearly-empty disk.
What I learnt
An alert that fires when the disk is already full is a notification, not a warning. The slope was visible days earlier.
What I do now
- Alert on time-to-full, not just on a low threshold.
- Turn on storage autoscaling with a sensible ceiling.
- Put an expiry on debug settings in production, with a ticket to turn them off.
stage 0 checks whether your alerts arrive in time → get the free audit