An OOMKilled container means a memory boundary got crossed. Which boundary, and why, is the actual debugging problem: the container's own limit, the node running dry, or a slow leak eating either one. There's also a look-alike, where the runtime hits its own ceiling and OOMKilled never appears at all. Here's how to read the signals and fix the cause rather than the symptom.
What is an OOMKilled error in Kubernetes?
OOMKilled is the status Kubernetes reports when the Linux kernel's out-of-memory (OOM) killer terminates a process inside your container. The kernel sends SIGKILL, the process dies immediately, and the container's last state records Reason: OOMKilled with exit code 137. That exit code turns up in other situations too, which the next section covers.
Since Kubernetes 1.28 on cgroups v2, the kubelet normally sets memory.oom.group, so an OOM event kills every process in the container together. On older versions, the OOM killer could pick a single child process and leave the container running degraded; 1.32 added a kubelet setting, singleProcessOOMKill, for clusters that want the single-process behaviour back.
Does exit code 137 always mean OOMKilled?
No. Exit code 137 means the process died from SIGKILL (128 plus signal 9), and the OOM killer is only one sender. The same code shows up when:
- A liveness probe fails, the kubelet sends SIGTERM, and the container ignores it until
terminationGracePeriodSecondsexpires - A pod is deleted or a node is drained, and the process sits through its grace period the same way
- Something kills the process manually, or a controller does it during a rollout or scale-down
The Reason field is what disambiguates. OOMKilled means the kernel's OOM killer. Error with exit code 137 means a SIGKILL from somewhere else, and the events often say where. Checking the reason first saves you from tuning memory limits to fix a probe problem.
What is the impact of OOMKilled errors?
OOMKilled errors cost you on two fronts at once. Underprovision and containers die mid-request under normal load: in-flight work is lost, clients retry, and the surviving pods absorb traffic they weren't sized for. Overprovision to make the problem go away and you reserve memory nothing uses, across every replica, on every node.
The kill itself is abrupt. SIGKILL gives the process no chance to flush buffers, close connections, or finish writes, so stateful workloads can lose data and downstream services see connections drop without warning.
Both failure modes trace back to the same gap: nobody measured what the workload actually needs before setting resource requests and limits. The rest of this page is about closing that gap.
What are the three types of OOM in Kubernetes?
Memory failures come in three shapes. Two normally produce the OOMKilled status; the third looks related and shows up somewhere else entirely.
1. Container OOM
The container exceeded its own resources.limits.memory, and the kernel killed a process inside that container's cgroup. It's isolated: one container, one limit, one kill, repeating on the same workload.
The signature: Reason: OOMKilled, exit code 137, a climbing restart count, and CrashLoopBackOff when it recurs quickly. Common causes: a limit set too low, a genuine leak, a load or batch spike, or a workload whose steady-state footprint was never measured.
2. Node OOM
The node itself runs low on allocatable memory. Before the kernel gets involved, the kubelet evicts pods once available memory drops below its eviction threshold (100Mi by default, configurable). Eviction order isn't a straight QoS ladder: the kubelet ranks pods by whether usage exceeds requests, then by priority, then by how far usage exceeds requests. BestEffort pods tend to go first because any usage at all exceeds their nonexistent requests.
The signature is evicted pods citing memory pressure, a MemoryPressure condition on the node, and a mix of pods dying around the same time, including containers sitting comfortably under their own limits. Common causes: summed limits well above node capacity, a noisy-neighbor pod, or missing requests letting the scheduler overpack the node.
3. Runtime OOM, the look-alike
The runtime hits its own memory ceiling before the cgroup does. The JVM throws OutOfMemoryError, Node.js aborts with "JavaScript heap out of memory", and the container exits with Reason: Error or keeps limping along degraded. OOMKilled never appears, which is exactly why this case gets misdiagnosed: the evidence lives in the application logs rather than the pod status.
Which failure you get depends on where the runtime's ceiling sits relative to the container limit. Heap ceiling below the limit, and the runtime throws first: this case. Heap ceiling above the limit, or a runtime that isn't container-aware, and the cgroup kills first: container OOM. The heap is also only part of what the cgroup counts. Native allocations, thread stacks, metaspace, and buffers all add up, so a container can still get OOMKilled with its heap comfortably under the limit.
For the two genuine OOMKilled types, the tell is the blast radius. Container OOM hits one workload repeatedly; node OOM kills a mix of pods at once, alongside eviction events.
How do resource requests and limits work together with QoS classes?
Memory requests are what the scheduler accounts for when placing a pod: it only lands on a node with enough unrequested capacity to cover the sum of its containers' requests, and requests set too high across a fleet leave pods stuck in Pending. They're bookkeeping rather than a fenced-off block of RAM. For CPU, requests also set the container's proportional share under contention.
Limits are the ceiling, and the two resources enforce it differently. CPU is compressible: exceed the limit and the container gets throttled, container_cpu_cfs_throttled_periods_total climbs in Prometheus, and nothing dies. CPU is compressible: at the limit the container gets throttled, container_cpu_cfs_throttled_periods_total climbs in Prometheus, and nothing dies. Memory is a hard boundary enforced reactively: when the kernel can't reclaim enough to keep the cgroup within its limit, the OOM killer fires. CPU pressure surfaces as latency; memory pressure surfaces as dead containers.
Limits also apply per container in the traditional model: a sidecar that exceeds its own limit dies alone rather than pushing the main container over anything shared. Kubernetes 1.34 adds beta pod-level requests and limits, where containers share headroom under an aggregate ceiling, but unless you've set that field, per-container is what you have.
QoS classes fall out of how you set resource requests and limits, each pairing a condition with a behaviour:
- Guaranteed: every container has CPU and memory requests and limits, with each request equal to its limit. Under node pressure it's last in line, with the lowest OOM score adjustment (-997).
- Burstable: not Guaranteed, but at least one container sets a CPU or memory request or limit. Middle of the kill order, with an OOM score that scales with usage over memory requests.
- BestEffort: no requests or limits anywhere in the pod. Highest OOM score, and in practice first out the door under node pressure.
Guaranteed helps most under node pressure, but it doesn't stop a container exceeding its own limit, and the CPU half of the deal trades burst headroom for throttling. That's why some teams skip full Guaranteed and instead set memory requests equal to memory limits inside a Burstable pod, which captures most of the memory-side protection. Either way, the requests need to reflect measured usage rather than guesses.
How do you diagnose OOMKilled errors?
Start with kubectl get pods. A workload that keeps getting OOMKilled shows a climbing restart count, and once restarts stack up, a CrashLoopBackOff status while the kubelet backs off between attempts.
Then kubectl describe pod for the confirmation:
kubectl describe pod my-app-xyz123
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
That block separates the OOM killer from every other source of SIGKILL. The Events section adds timing, and the restart count tells you whether you're looking at a one-off spike or a pattern.
Check the pod logs from the run that died, since the current container's logs start after the kill:
kubectl logs my-app-xyz123 --previous
Memory pressure often shows in the tail: allocation failures, GC thrash, a burst of oversized requests. And if those logs show OutOfMemoryError or "JavaScript heap out of memory" while the pod reports Reason: Error instead of OOMKilled, you're in runtime OOM territory. An explicitly set heap won't follow a raised container limit, though a container-aware runtime with default sizing will.
Then work out which OOM you're dealing with:
kubectl top pod my-app-xyz123
kubectl describe node <node-name> | grep -A 5 Conditions
journalctl -k | grep -i "out of memory"
kubectl top gives a recent usage snapshot; compare it against the limits from kubectl describe pod. The node's conditions reveal MemoryPressure, and the kernel log, read on the affected node, names the process and cgroup the OOM killer chose, which usually settles container versus node OOM.
kubectl top is a point-in-time reading, though. The cause shows in the shape over time, so chart container_memory_working_set_bytes in Prometheus or a Grafana panel. Every pattern in the next section is a time-series shape, and none of them are visible in a single sample.
What causes unexpected memory consumption?
Memory leaks. Usage climbs steadily from startup until it hits the limit, the container restarts, and the cycle repeats. The restart resets the clock, which is how leaky services hide behind "it recovered on its own" for months.
Unbounded caches and queues. In-memory caches without eviction, or queues that buffer while a downstream consumer runs slow. These resemble leaks on a graph but plateau when input pressure drops.
Load-proportional spikes. Large request payloads, file processing, fan-out queries. The baseline fits the limit and the p99 doesn't, so kills correlate with traffic rather than uptime.
Runtimes sized off the node. The JVM is the classic case, failing in whichever direction the runtime OOM section describes: wherever the heap ceiling sits relative to the container limit, one of them loses. Container awareness (-XX:+UseContainerSupport) has been the default since JDK 10 and 8u191, but cgroups v2 support only arrived in JDK 15, backported to 11.0.16 and 8u372. An older JDK on a cgroups v2 node reads node memory, sizes its heap from that, and hits the container limit well before its own heap limit: container OOM with a runtime cause.
How do you fix OOMKilled errors?
Match the fix to the pattern:
- Steady climb to the limit: that's a leak. Profile and fix it; raising the limit only stretches the interval between kills.
- Traffic-correlated kills: raise the limit to observed peak plus headroom, or bound the work with payload caps, streaming instead of buffering, and cache eviction policies.
- Runtime misreading the container: set the heap explicitly with
-Xmxor move to a cgroups-v2-aware JDK, and leave room between heap and limit for off-heap memory. - Node OOM rather than container OOM: right-size memory requests fleet-wide so the scheduler stops overcommitting, and don't run anything important as BestEffort.
- Any pattern: alert before the kill. A Prometheus alert on working set above roughly 90% of the limit turns the next OOMKilled into a warning instead of a page.
For critical workloads, requests equal to limits removes the ambiguity. The pod either schedules with its memory reserved, or it doesn't schedule at all.
Why are memory issues hard to debug?
The restart is the trap. Auto-recovery means the service is back before anyone investigates, the root cause survives, and the kill returns at the next traffic peak, usually off-hours.
The signal is also split across systems. Usage history lives in Prometheus, the kill itself in kernel logs on the node, and the trigger in a deployment or traffic change somewhere upstream. kubectl shows you that it happened, rarely why.
Understanding what OOMKilled means takes a paragraph. Tying one specific kill to the change that caused it, in code, configuration, or load, takes correlation across all of those sources at once.
How Resolve AI approaches OOMKilled errors
An OOMKilled event is a symptom with a dozen candidate causes, whether in application code, resource configuration, traffic patterns, or the node underneath. Pasting the error into a general-purpose chatbot gets you the generic checklist: check your limits, look for leaks. It can't see your Grafana dashboards, your deploys, or your kernel logs.
Resolve AI investigates with that context attached. When a memory spike appears, it pulls the pod's usage history, recent deployments, traffic changes, and node-level pressure together, then traces the kill back to what changed. The same Kubernetes troubleshooting applies whether the answer is a leak introduced last Tuesday or a node quietly overcommitted for weeks.
That turns OOMKilled from a recurring page into a diagnosis: which boundary was crossed, what crossed it, and whether the fix belongs in a limit, a line of code, or the scheduler's inputs. Visibility across code, infrastructure, and telemetry is what makes that correlation fast instead of tedious.
