The page
The application started timing out on every database call. The database itself was healthy and accepting connections from a bastion host.
Triage
# from an app host: can we even open the port?
nc -zv prod-db.internal.example.com 5432
# what the database's security group allows in
aws ec2 describe-security-groups --group-ids sg-0db123 \
--query "SecurityGroups[0].IpPermissions"
VPC Flow Logs in CloudWatch Logs Insights, filtered to rejected traffic on the database port:
fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and action = "REJECT"
| sort @timestamp desc
| limit 20
And who changed what:
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=RevokeSecurityGroupIngress \
--start-time "$(date -u -d '-6 hours' +%FT%TZ)"
Rejected connections from the app subnet, starting a few minutes after a
RevokeSecurityGroupIngress call.
Root cause
Someone was cleaning up old rules that referenced IP ranges. One of them was the app subnet's range, which looked like an old one. The change was done in the console, so nothing reviewed it.
The fix
Put the rule back, then rewrote it to reference the app's security group instead of a CIDR range, so it describes intent ("the app can reach the database") and survives subnet changes. Security groups for production moved into code with review.
What I learnt
Network rules written as IP ranges are hard to read and easy to delete by mistake. Rules that say what they're for get deleted less.
What I do now
- Reference security groups, not CIDRs, for service-to-service rules.
- Manage production network rules in code with review.
- Keep flow logs on, so "is traffic being dropped?" has an answer in minutes.
if production network changes happen in the console → page me