How do I diagnose out-of-memory (OOM) errors on CKS?

A memory kill on CKS is either one of the following:

Container OOM: Your container exceeded its own cgroup memory limit.
Node memory pressure: The Node ran out of physical memory.

To tell them apart, check the container’s last termination state. Replace [POD-NAME] with the Pod’s name and [NAMESPACE] with its namespace:

kubectl describe pod [POD-NAME] -n [NAMESPACE]

Reason: OOMKilled confirms a memory kill. To classify it, read the kernel OOM constraint from the Node’s kernel logs, if you have Node log access:

CONSTRAINT_MEMCG means a container limit you set too low
CONSTRAINT_NONE means Node-level pressure that may belong to CoreWeave.

OOM symptoms

A container restarts and kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137.
A Pod shows Evicted with a message about the Node being low on memory.
Kernel logs contain a line such as oom-kill:constraint=CONSTRAINT_MEMCG or Memory cgroup out of memory: Killed process ....

Container OOM versus Node memory pressure

These are different failures with different fixes. Use the tables to help classify which failure it is.

Container OOM

Signal	What you see
Kubernetes event	`OOMKilled`, exit code 137
Kernel log constraint	`constraint=CONSTRAINT_MEMCG`
Node `MemoryPressure` condition	`False` (the Node is fine)
Available Node memory	Ample, often hundreds of GB free
Fix	Raise the container memory limit, or reduce usage
Responsibility	Yours (limit set too low for the workload)

Container OOM is the most common case CoreWeave support sees and is almost always a workload configuration issue.

Node memory pressure

Signal	What you see
Kubernetes event	`Evicted` by the kubelet
Kernel log constraint	`constraint=CONSTRAINT_NONE`
Node `MemoryPressure` condition	`True`
Available Node memory	Low
Fix	Reduce workload memory on the Node, or add Nodes
Responsibility	Possibly CoreWeave

Diagnose the OOM

Follow these steps to identify the cause of the OOM.

Read the kernel constraint

The kernel OOM log line names the constraint that triggered the kill. This is the most reliable signal.

Kernel OOM log lines

# Container hit its own cgroup limit (your config to fix):
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=null,...,task=<process-name>,...

# The Node ran out of actual memory (node-level pressure):
oom-kill:constraint=CONSTRAINT_NONE,nodemask=null,...,task=<process-name>,...

The Memory cgroup out of memory line reports the killed process:

Example cgroup OOM line

Memory cgroup out of memory: Killed process 4183833 my-trainer \
  total-vm:7578312kB, anon-rss:2065712kB, file-rss:35328kB, \
  shmem-rss:0kB, UID:65532 pgtables:4396kB oom_score_adj:999

anon-rss is the resident memory the process used. If it’s close to your container’s memory limit, the container hit its own limit. total-vm is only virtual address space, not real memory use.The oom_memcg field in the oom-kill:constraint= line contains the Pod UID, with underscores instead of hyphens. Convert it before matching it to a Pod:

Matching a cgroup path to a Pod UID

Kernel:  ...kubepods-burstable-pod2d0d6db2_3cad_4289_a66c_b1938dc147e2.slice
Pod UID: 2d0d6db2-3cad-4289-a66c-b1938dc147e2

Find the Pod with a given UID. Replace [POD-UID] with the UID from the previous step:

kubectl get pods --all-namespaces \
  -o custom-columns=UID:.metadata.uid,NAMESPACE:.metadata.namespace,NAME:.metadata.name \
  | grep [POD-UID]

This command is safe to run anytime and it’s read-only.

Check the termination reason

Exit code 137 is 128 + 9 (SIGKILL). Rolling updates, Pod deletions, and Node drains also end in SIGKILL, so exit code 137 alone doesn’t prove an OOM.

# Read the container's last termination reason. Safe to run anytime; read-only.
kubectl describe pod [POD-NAME] -n [NAMESPACE] | grep -A5 "Last State:"

Termination reason OOMKilled: a real OOM.
Restart count increments: a real OOM. It stays at 0 when the Pod is recreated by a rolling update or deletion.
A graceful Stopping container event sequence: normal termination.

Alert on the OOMKilled reason, not on exit code 137, or every rolling update produces false OOM alerts.

Find the source of the OOM

The namespace where you notice symptoms is often not where the OOM started.

Find Pods with the highest restart counts cluster-wide. Safe; read-only.

kubectl get pods --all-namespaces \
  --sort-by='.status.containerStatuses[0].restartCount' | tail -20

For any suspect Pod, read its last termination state. Safe; read-only.

kubectl describe pod [POD-NAME] -n [NAMESPACE] | grep -A5 "Last State:"

Backlog-processing addons, such as policy report controllers, are common culprits: the backlog grows during CrashLoopBackOff, so each restart needs more memory than the last. Raise the limit well above the current value, let the backlog clear, then right-size it.A subprocess inside a multi-process Pod can be OOM-killed without the kubelet recording an OOMKilled event. If a Pod misbehaves with no OOMKilled event, check kernel logs if you have Node log access.

How to fix OOM problems

For container OOM (CONSTRAINT_MEMCG, MemoryPressure=False):

Raise the container’s memory limit, or reduce what the workload allocates. See What are the recommended resource requests and limits for GPU Pods?.
Watch for a missing unit suffix: memory: 16 means 16 bytes, memory: 16G means 16 gigabytes, and memory: 16Gi means about 17.2 gigabytes.

For Node memory pressure (CONSTRAINT_NONE, MemoryPressure=True):

Reduce the total memory your Pods use on the Node, or spread the workload across more Nodes.
Set requests to match actual usage so the scheduler doesn’t overpack the Node.
If there’s no apparent workload cause, contact support with the Node name and the OOM events.

When a CoreWeave-managed Pod is the one killed

If the killed process is node_exporter, the CoreWeave-managed monitoring agent in the cw-exporters namespace, the cause is usually still a workload that creates an unusually large number of processes, sockets, mounts, or file descriptors, which node_exporter enumerates on every scrape. This case looks like:

The killed process is node_exporter, not one of your processes.
Multiple Nodes are affected at about the same time, matching where the workload runs.
The Nodes show no MemoryPressure condition.

List cgroup OOM kill events cluster-wide:

kubectl get events --all-namespaces --field-selector reason=CGroupOOMKilling

Look for what changed recently, such as a new monitoring agent or a workload with many processes per Node, and reduce that count. For other CoreWeave-managed Pods killed with no workload cause, contact support with the Node name and the OOM events.

When the memory graphs mislead you

A Pod that OOMs within seconds of starting shows near-zero memory in dashboards, because it crashes before the metrics scrape records peak usage. When a Pod is in CrashLoopBackOff, trust the termination reason, the restart count, and the kernel log, not the memory graph.
In Grafana memory graphs, “used” memory includes page cache, which is not memory pressure: the kernel reclaims cache freely. Judge Node-level pressure by available memory, not used memory: a Node with hundreds of GB available isn’t memory-constrained even if the used graph looks full.
Available memory counts /dev/shm and other tmpfs memory as reclaimable cache, but the kernel can’t reclaim it. Workloads that use /dev/shm heavily can OOM a Node whose available memory looks adequate. To check, compare node_memory_Shmem_bytes to node_memory_Cached_bytes in Grafana: if shmem tracks cached, most of the “cache” can’t be reclaimed. Reduce shared-memory usage or leave real headroom.

Common confusions

Exit code 137 isn’t always an OOM. Read the termination reason.
“Connection refused” between your Pods can be an OOM symptom, not a network problem: an OOM-killed server process is no longer listening on its port. Check kubectl get events --field-selector involvedObject.name=[NODE-NAME] for SystemOOM events before investigating networking.
FailedKillPod events with DeadlineExceeded are a different failure: the container runtime couldn’t stop the container. If a Pod stays stuck in Terminating, contact support with the Pod and Node names.
A single Node that consistently OOMs on workloads that succeed on identical Nodes may have a hardware state issue. Contact support with the Node name.

Pod Inspector dashboard: OOM history and memory panels.
Node cordoning: why CoreWeave cordons Nodes and how to check the cordon reason.

Server Errors

​OOM symptoms

​Container OOM versus Node memory pressure

​Container OOM

​Node memory pressure

​Diagnose the OOM

​How to fix OOM problems

​When a CoreWeave-managed Pod is the one killed

​When the memory graphs mislead you

​Common confusions

​Related pages