> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# How do I diagnose out-of-memory (OOM) errors on CKS?

A memory kill on CKS is either one of the following:

* **Container OOM**: Your container exceeded its own cgroup memory limit.
* **Node memory pressure**: The Node ran out of physical memory.

To tell them apart, check the container's last termination state. Replace `[POD-NAME]` with the Pod's name and `[NAMESPACE]` with its namespace:

```bash theme={"system"}
kubectl describe pod [POD-NAME] -n [NAMESPACE]
```

`Reason: OOMKilled` confirms a memory kill. To classify it, read the kernel OOM
constraint from the Node's kernel logs, if you have Node log access:

* `CONSTRAINT_MEMCG` means a container limit you set too low
* `CONSTRAINT_NONE` means Node-level pressure that may belong to CoreWeave.

## OOM symptoms

* A container restarts and `kubectl describe pod` shows
  `Last State: Terminated, Reason: OOMKilled, Exit Code: 137`.
* A Pod shows `Evicted` with a message about the Node being low on memory.
* Kernel logs contain a line such as `oom-kill:constraint=CONSTRAINT_MEMCG` or
  `Memory cgroup out of memory: Killed process ...`.

## Container OOM versus Node memory pressure

These are different failures with different fixes. Use the tables to help classify
which failure it is.

### Container OOM

| Signal                          | What you see                                      |
| ------------------------------- | ------------------------------------------------- |
| Kubernetes event                | `OOMKilled`, exit code 137                        |
| Kernel log constraint           | `constraint=CONSTRAINT_MEMCG`                     |
| Node `MemoryPressure` condition | `False` (the Node is fine)                        |
| Available Node memory           | Ample, often hundreds of GB free                  |
| Fix                             | Raise the container memory limit, or reduce usage |
| Responsibility                  | Yours (limit set too low for the workload)        |

Container OOM is the most common case CoreWeave support sees and is almost
always a workload configuration issue.

### Node memory pressure

| Signal                          | What you see                                     |
| ------------------------------- | ------------------------------------------------ |
| Kubernetes event                | `Evicted` by the kubelet                         |
| Kernel log constraint           | `constraint=CONSTRAINT_NONE`                     |
| Node `MemoryPressure` condition | `True`                                           |
| Available Node memory           | Low                                              |
| Fix                             | Reduce workload memory on the Node, or add Nodes |
| Responsibility                  | Possibly CoreWeave                               |

## Diagnose the OOM

Follow these steps to identify the cause of the OOM.

<Steps>
  <Step title="Read the kernel constraint">
    The kernel OOM log line names the constraint that triggered the kill. This is the
    most reliable signal.

    ```text title="Kernel OOM log lines" theme={"system"}
    # Container hit its own cgroup limit (your config to fix):
    oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=null,...,task=<process-name>,...

    # The Node ran out of actual memory (node-level pressure):
    oom-kill:constraint=CONSTRAINT_NONE,nodemask=null,...,task=<process-name>,...
    ```

    The `Memory cgroup out of memory` line reports the killed process:

    ```text title="Example cgroup OOM line" theme={"system"}
    Memory cgroup out of memory: Killed process 4183833 my-trainer \
      total-vm:7578312kB, anon-rss:2065712kB, file-rss:35328kB, \
      shmem-rss:0kB, UID:65532 pgtables:4396kB oom_score_adj:999
    ```

    `anon-rss` is the resident memory the process used. If it's close to your
    container's memory limit, the container hit its own limit. `total-vm` is only
    virtual address space, not real memory use.

    The `oom_memcg` field in the `oom-kill:constraint=` line contains the Pod UID,
    with underscores instead of hyphens. Convert it before matching it to a Pod:

    ```text title="Matching a cgroup path to a Pod UID" theme={"system"}
    Kernel:  ...kubepods-burstable-pod2d0d6db2_3cad_4289_a66c_b1938dc147e2.slice
    Pod UID: 2d0d6db2-3cad-4289-a66c-b1938dc147e2
    ```

    Find the Pod with a given UID. Replace `[POD-UID]` with the UID from the previous step:

    ```bash theme={"system"}
    kubectl get pods --all-namespaces \
      -o custom-columns=UID:.metadata.uid,NAMESPACE:.metadata.namespace,NAME:.metadata.name \
      | grep [POD-UID]
    ```

    This command is safe to run anytime and it's read-only.
  </Step>

  <Step title="Check the termination reason">
    Exit code 137 is `128 + 9 (SIGKILL)`. Rolling updates, Pod deletions, and Node
    drains also end in SIGKILL, so exit code 137 alone doesn't prove an OOM.

    ```bash theme={"system"}
    # Read the container's last termination reason. Safe to run anytime; read-only.
    kubectl describe pod [POD-NAME] -n [NAMESPACE] | grep -A5 "Last State:"
    ```

    * Termination reason `OOMKilled`: a real OOM.
    * Restart count increments: a real OOM. It stays at 0 when the Pod is recreated
      by a rolling update or deletion.
    * A graceful `Stopping container` event sequence: normal termination.

    Alert on the `OOMKilled` reason, not on exit code 137, or every rolling update
    produces false OOM alerts.
  </Step>

  <Step title="Find the source of the OOM">
    The namespace where you notice symptoms is often not where the OOM started.

    1. Find Pods with the highest restart counts cluster-wide. Safe; read-only.
       ```bash theme={"system"}
       kubectl get pods --all-namespaces \
         --sort-by='.status.containerStatuses[0].restartCount' | tail -20
       ```

    2. For any suspect Pod, read its last termination state. Safe; read-only.
       ```bash theme={"system"}
       kubectl describe pod [POD-NAME] -n [NAMESPACE] | grep -A5 "Last State:"
       ```

    Backlog-processing addons, such as policy report controllers, are common
    culprits: the backlog grows during `CrashLoopBackOff`, so each restart needs
    more memory than the last. Raise the limit well above the current value, let
    the backlog clear, then right-size it.

    A subprocess inside a multi-process Pod can be OOM-killed without the kubelet
    recording an `OOMKilled` event. If a Pod misbehaves with no `OOMKilled` event,
    check kernel logs if you have Node log access.
  </Step>
</Steps>

## How to fix OOM problems

For **container OOM** (`CONSTRAINT_MEMCG`, `MemoryPressure=False`):

* Raise the container's memory limit, or reduce what the workload allocates. See
  [What are the recommended resource requests and limits for GPU Pods?](/support/cks/articles/what-are-the-recommended-resource-requests-and-limits-for-gpu-pods).
* Watch for a missing unit suffix: `memory: 16` means 16 bytes, `memory: 16G`
  means 16 gigabytes, and `memory: 16Gi` means about 17.2 gigabytes.

For **Node memory pressure** (`CONSTRAINT_NONE`, `MemoryPressure=True`):

* Reduce the total memory your Pods use on the Node, or spread the workload
  across more Nodes.
* Set requests to match actual usage so the scheduler doesn't overpack the Node.
* If there's no apparent workload cause, [contact support](/support) with the
  Node name and the OOM events.

## When a CoreWeave-managed Pod is the one killed

If the killed process is `node_exporter`, the CoreWeave-managed monitoring agent
in the `cw-exporters` namespace, the cause is usually still a workload that creates an
unusually large number of processes, sockets, mounts, or file descriptors, which
`node_exporter` enumerates on every scrape. This case looks like:

* The killed process is `node_exporter`, not one of your processes.
* Multiple Nodes are affected at about the same time, matching where the
  workload runs.
* The Nodes show no `MemoryPressure` condition.

List cgroup OOM kill events cluster-wide:

```bash theme={"system"}
kubectl get events --all-namespaces --field-selector reason=CGroupOOMKilling
```

Look for what changed recently, such as a new monitoring agent or a workload
with many processes per Node, and reduce that count. For other CoreWeave-managed
Pods killed with no workload cause, [contact support](/support) with the Node
name and the OOM events.

## When the memory graphs mislead you

* A Pod that OOMs within seconds of starting shows near-zero memory in
  dashboards, because it crashes before the metrics scrape records peak usage.
  When a Pod is in `CrashLoopBackOff`, trust the termination reason, the
  restart count, and the kernel log, not the memory graph.
* In Grafana memory graphs, "used" memory includes page cache, which is not
  memory pressure: the kernel reclaims cache freely. Judge Node-level pressure
  by available memory, not used memory: a Node with hundreds of GB available
  isn't memory-constrained even if the used graph looks full.
* Available memory counts `/dev/shm` and other tmpfs memory as reclaimable
  cache, but the kernel can't reclaim it. Workloads that use `/dev/shm` heavily
  can OOM a Node whose available memory looks adequate. To check, compare
  `node_memory_Shmem_bytes` to `node_memory_Cached_bytes` in Grafana: if shmem
  tracks cached, most of the "cache" can't be reclaimed. Reduce shared-memory
  usage or leave real headroom.

## Common confusions

* Exit code 137 isn't always an OOM. Read the termination reason.
* "Connection refused" between your Pods can be an OOM symptom, not a network
  problem: an OOM-killed server process is no longer listening on its port.
  Check `kubectl get events --field-selector involvedObject.name=[NODE-NAME]`
  for `SystemOOM` events before investigating networking.
* `FailedKillPod` events with `DeadlineExceeded` are a different failure: the
  container runtime couldn't stop the container. If a Pod stays stuck in
  `Terminating`, [contact support](/support) with the Pod and Node names.
* A single Node that consistently OOMs on workloads that succeed on identical
  Nodes may have a hardware state issue. [Contact support](/support) with the
  Node name.

## Related pages

* [Pod Inspector dashboard](/observability/managed-grafana/cks/pod-inspector): OOM history and memory panels.
* [Node cordoning](/products/cks/nodes/cordon): why CoreWeave cordons Nodes and how to check the cordon reason.

<Badge stroke shape="pill" color="blue" size="md">[Server Errors](/support/cks/tags/server-errors)</Badge>