> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Diagnose unexplained memory pressure on a SUNK node

> Tell reclaimable page cache from pinned shmem and system overhead when pod metrics look fine but the node OOMs

When a SUNK compute node exhausts its memory, the kernel terminates the largest offending process to reclaim it. This is an out-of-memory (OOM) kill, and Slurm reports the job as failed. The metrics afterward may appear contradictory: the node's Kubernetes Pod metrics show modest memory use, and the node's Grafana panel shows ample available memory in the moments before the OOM event.

This page explains how to diagnose the source of memory pressure on a node by checking node-level memory instead of pod metrics. Checking the right metrics helps you determine whether memory pressure comes from reclaimable page cache or pinned shmem and tmpfs. If necessary, cluster administrators can change `MemSpecLimit`, `ConstrainRAMSpace`, or the cache-dropper sidecar.

## Read node memory, not pod metrics

Slurm's cgroup plugin creates its own cgroup hierarchy on the node, separate from the Kubernetes pod's cgroup. Training job memory is charged to the Slurm hierarchy, not the pod hierarchy. This separation has two consequences:

* cAdvisor and Kubernetes pod metrics show only the `slurmd` daemon's small footprint, not the memory jobs use.
* A pod memory limit on the `slurmd` container doesn't constrain the Slurm jobs themselves.

To diagnose the source of memory pressure, read node-level memory (`/proc/meminfo` or `node_memory_*` metrics), not pod or container metrics. For the CPU side of the same split and the Slurm metrics the Syncer exports instead, see [Why do slurmd Pod metrics show almost no CPU or memory usage?](/support/sunk/articles/why-do-slurmd-pod-metrics-show-almost-no-cpu-or-memory-usage).

The next section shows how to compare `Shmem` and `Cached` in `/proc/meminfo` to distinguish reclaimable page cache from pinned shmem and tmpfs.

### Compare `Shmem` and `Cached`

In `/proc/meminfo`, the `Shmem` field (shared memory and tmpfs, including `/dev/shm`) is counted inside `Cached`. The kernel's `MemAvailable` estimate treats cached memory as reclaimable. Shmem and tmpfs pages aren't reclaimable under pressure: unlike file-backed page cache, they have no backing file to evict to.

<Note>
  Don't compare `Shmem` against `Unevictable`. The kernel's unevictable list holds ramfs pages, tmpfs mounted with `noswap`, `SHM_LOCK`ed segments, and `mlock()`ed memory, so an ordinary `/dev/shm` leaves `Unevictable` near zero even when tens of gibibytes of shmem are pinned.
</Note>

Use the ratio of `Shmem` to `Cached` to tell reclaimable cache from pinned shmem.

1. On the affected node, compare `Shmem` and `Cached` by running the commands below:

   ```bash theme={"system"}
   # Compare Shmem and Cached. Read-only.
   grep -E 'MemTotal|MemAvailable|^Cached|^Shmem' /proc/meminfo

   # Tmpfs shared memory in use. Read-only.
   df -h /dev/shm
   ```

2. In the output, look for the following node metrics:

   | `/proc/meminfo` field | Prometheus metric                |
   | --------------------- | -------------------------------- |
   | `Shmem`               | `node_memory_Shmem_bytes`        |
   | `Cached`              | `node_memory_Cached_bytes`       |
   | `MemAvailable`        | `node_memory_MemAvailable_bytes` |

3. Compare the two values and find your case in the table:

   | What you see                            | Source of the pressure                                                        | What frees it                                                                                                                                                                                                                                                                                                                      |
   | --------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | `Shmem` is a small fraction of `Cached` | Reclaimable, file-backed page cache left between exclusive jobs.              | The [cache-dropper sidecar](/products/sunk/optimize_workloads/cache-dropper) can free it. [`compute.cacheDropper.enabled`](/products/sunk/reference/slurm-parameters#compute--cacheDropper--enabled) defaults to `true`; a job triggers a drop by creating `/run/enroot/drop_caches`. The cache-dropper can't free shmem or tmpfs. |
   | `Shmem` accounts for most of `Cached`   | Pinned shmem and tmpfs. A high `MemAvailable` reading isn't trustworthy here. | Nothing on the node frees it: neither the cache-dropper nor a manual `drop_caches`. Reduce shmem use in the job, or ask an administrator to scope per-job temp space and raise reserved system memory (see the following section).                                                                                                 |

## Ask an administrator

Bring these items to the cluster administrator. Job authors can't set them.

**`MemSpecLimit`** reserves memory, in mebibytes, for system use so Slurm doesn't allocate it to jobs. Leave it unset and Slurm reserves nothing beyond what the node already holds back. It takes effect only when memory is a consumable resource, which the SUNK default `SelectTypeParameters=CR_CPU_MEMORY` satisfies.

On SUNK, CoreWeave already holds back memory for the operating system, kubelet, and DaemonSets, and Slurm advertises a reduced `RealMemory` to match. `MemSpecLimit` covers what that baseline misses: on-node sidecars and shmem peaks. See [Avoid stranded GPUs](/products/sunk/run_workloads/avoid-stranded-gpus) for how `RealMemory` relates to what a job can request.

<Warning>
  Don't put `MemSpecLimit` in `slurmConfig`. That block renders into global `slurm.conf` scope, where Slurm rejects the key with `_parse_next_key: Parsing error at unrecognized key: MemSpecLimit`. On SUNK that parse failure restarts every Slurm pod and disrupts running jobs. `MemSpecLimit` is a per-node parameter and belongs on the node configuration. The same applies to `CoreSpecCount` and other per-node keys.
</Warning>

The CoreWeave AI Object Storage proxy sidecar, when present, can consume 30 to 50 GiB under heavy object-storage workloads. That usage is invisible to cAdvisor and to Slurm cgroup accounting on cgroup v1. Account for it in `MemSpecLimit` on nodes that run object-storage-heavy jobs.

**`ConstrainRAMSpace=yes`** in `cgroup.conf` is what makes Slurm enforce per-job memory limits. SUNK sets it by default under [`slurmConfig.cgroupConfig`](/products/sunk/reference/slurm-parameters#slurmConfig). Two conditions make it effective:

* `SelectTypeParameters` must track memory. SUNK defaults to `CR_CPU_MEMORY`, so check this only when someone has overridden it. Without memory as a consumable resource, cgroup memory enforcement may not work even with `ConstrainRAMSpace=yes`.
* On cgroup v1, `ConstrainRAMSpace=yes` is the only thing bounding a Slurm job, because pod limits don't constrain Slurm jobs. On cgroup v2, Slurm still manages its own memory cgroup slice, so `ConstrainRAMSpace=yes` is still required for Slurm-side enforcement.

Even with `ConstrainRAMSpace=yes`, heavy `/dev/shm` use can still cause a node-level out-of-memory event, because shmem allocations live in the host-level Slurm cgroup rather than the per-task cgroup that `ConstrainRAMSpace` bounds. For full cgroup configuration, see [Manage resource binding with task plugins](/products/sunk/optimize_workloads/task-plugins).

```bash theme={"system"}
# cgroup2fs means cgroup v2; tmpfs means cgroup v1. Read-only.
stat -fc %T /sys/fs/cgroup/
```

**`JobContainerType=job_container/tmpfs`** scopes `/dev/shm` and temp directories per job so they're cleaned up automatically. Set `JobContainerType=job_container/tmpfs` and add `Contain` to `PrologFlags`; Slurm requires `PrologFlags=Contain` for this plugin. The SUNK default [`PrologFlags`](/products/sunk/reference/slurm-parameters#slurmConfig) is `Alloc,Serial`, so this is a change to make, not a default to rely on.

## When to file a support ticket

Open a ticket when a node runs out of memory and you've confirmed the pressure isn't reclaimable cache and have accounted for shmem and system overhead in `MemSpecLimit`. Include `/proc/meminfo` output, the cgroup version, and the job's memory pattern (especially `/dev/shm` use).

## Related pages

* [Manage memory with cache-dropper](/products/sunk/optimize_workloads/cache-dropper): trigger the page cache dropper between jobs.
* [Manage resource binding with task plugins](/products/sunk/optimize_workloads/task-plugins): cgroup configuration and constraints.
* [Avoid stranded GPUs](/products/sunk/run_workloads/avoid-stranded-gpus): how `RealMemory` and `DefMemPerCPU` bound what a job can request.
* [Why do slurmd Pod metrics show almost no CPU or memory usage?](/support/sunk/articles/why-do-slurmd-pod-metrics-show-almost-no-cpu-or-memory-usage): the same cgroup split, and which metrics to read instead.
* [Share storage across Slurm nodes](/products/sunk/manage_sunk/shared-storage): shared directory permissions and home directory hygiene.


## Related topics

- [Manage memory with cache-dropper](/products/sunk/optimize_workloads/cache-dropper.md)