Skip to content
rajesh.dhar
2020 · platform engineer, devops · aks, kubernetes, helm

Pods OOMKilled because nobody set limits

One service's memory leak kept getting other pods killed on the same nodes. Nothing had requests or limits set, so the scheduler had no idea.

1 min readby Rajesh Dhar
storykubernetesakshelm

The page

Pods for unrelated services kept restarting on the same few nodes, several times an hour.

Triage

# restarting pods across the cluster
kubectl get pods -A | grep -E "OOMKilled|CrashLoopBackOff|Error"

# why the last container died
kubectl describe pod <pod> -n <ns> | grep -A5 "Last State"

# who's using the memory right now
kubectl top pods -A --sort-by=memory | head -15
kubectl top nodes

# what resources the pod asked for (empty means nothing)
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources}'

Last State: Terminated, Reason: OOMKilled on the victims. kubectl top showed one service using far more memory than anything else, and its resources field was empty. So were most others.

Root cause

One service had a slow memory leak. With no requests or limits anywhere, the scheduler packed pods onto nodes with no idea how much memory they'd need, and when a node ran out the kernel killed whatever it picked, usually not the leaking pod.

The fix

Set requests and limits on every chart through Helm values, starting from what kubectl top showed over a week:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"

With a limit, the leaking service got killed itself, which pointed straight at it, and the team fixed the leak. A LimitRange in each namespace now gives defaults to anything that forgets.

What I learnt

Without requests and limits, one bad pod's problem becomes everyone's. Limits put the blast radius back where it belongs.

What I do now

  • Requests and limits on every workload, set from measured usage.
  • A namespace LimitRange so nothing ships without defaults.
  • Alert on restarts per pod, not just on nodes.

stage 2 sets this up properly from the start → see the engagements

Related

← all posts