Diagnose unexplained memory pressure on a SUNK node

When a SUNK compute node exhausts its memory, the kernel terminates the largest offending process to reclaim it. This is an out-of-memory (OOM) kill, and Slurm reports the job as failed. The metrics afterward may appear contradictory: the node’s Kubernetes Pod metrics show modest memory use, and the node’s Grafana panel shows ample available memory in the moments before the OOM event. This page explains how to diagnose the source of memory pressure on a node by checking node-level memory instead of pod metrics. Checking the right metrics helps you determine whether memory pressure comes from reclaimable page cache or pinned shmem and tmpfs. If necessary, cluster administrators can change MemSpecLimit, ConstrainRAMSpace, or the cache-dropper sidecar.

Read node memory, not pod metrics

Slurm’s cgroup plugin creates its own cgroup hierarchy on the node, separate from the Kubernetes pod’s cgroup. Training job memory is charged to the Slurm hierarchy, not the pod hierarchy. This separation has two consequences:

cAdvisor and Kubernetes pod metrics show only the slurmd daemon’s small footprint, not the memory jobs use.
A pod memory limit on the slurmd container doesn’t constrain the Slurm jobs themselves.

To diagnose the source of memory pressure, read node-level memory (/proc/meminfo or node_memory_* metrics), not pod or container metrics. For the CPU side of the same split and the Slurm metrics the Syncer exports instead, see Why do slurmd Pod metrics show almost no CPU or memory usage?. The next section shows how to compare Shmem and Cached in /proc/meminfo to distinguish reclaimable page cache from pinned shmem and tmpfs.

Compare `Shmem` and `Cached`

In /proc/meminfo, the Shmem field (shared memory and tmpfs, including /dev/shm) is counted inside Cached. The kernel’s MemAvailable estimate treats cached memory as reclaimable. Shmem and tmpfs pages aren’t reclaimable under pressure: unlike file-backed page cache, they have no backing file to evict to.

Don’t compare Shmem against Unevictable. The kernel’s unevictable list holds ramfs pages, tmpfs mounted with noswap, SHM_LOCKed segments, and mlock()ed memory, so an ordinary /dev/shm leaves Unevictable near zero even when tens of gibibytes of shmem are pinned.

Use the ratio of Shmem to Cached to tell reclaimable cache from pinned shmem.

On the affected node, compare Shmem and Cached by running the commands below:

# Compare Shmem and Cached. Read-only.
grep -E 'MemTotal|MemAvailable|^Cached|^Shmem' /proc/meminfo

# Tmpfs shared memory in use. Read-only.
df -h /dev/shm

In the output, look for the following node metrics:
/proc/meminfo field Prometheus metric
Shmem node_memory_Shmem_bytes
Cached node_memory_Cached_bytes
MemAvailable node_memory_MemAvailable_bytes

`/proc/meminfo` field	Prometheus metric
`Shmem`	`node_memory_Shmem_bytes`
`Cached`	`node_memory_Cached_bytes`
`MemAvailable`	`node_memory_MemAvailable_bytes`

Compare the two values and find your case in the table:

What you see	Source of the pressure	What frees it
`Shmem` is a small fraction of `Cached`	Reclaimable, file-backed page cache left between exclusive jobs.	The cache-dropper sidecar can free it. `compute.cacheDropper.enabled` defaults to `true`; a job triggers a drop by creating `/run/enroot/drop_caches`. The cache-dropper can’t free shmem or tmpfs.
`Shmem` accounts for most of `Cached`	Pinned shmem and tmpfs. A high `MemAvailable` reading isn’t trustworthy here.	Nothing on the node frees it: neither the cache-dropper nor a manual `drop_caches`. Reduce shmem use in the job, or ask an administrator to scope per-job temp space and raise reserved system memory (see the following section).

Ask an administrator

Bring these items to the cluster administrator. Job authors can’t set them. MemSpecLimit reserves memory, in mebibytes, for system use so Slurm doesn’t allocate it to jobs. Leave it unset and Slurm reserves nothing beyond what the node already holds back. It takes effect only when memory is a consumable resource, which the SUNK default SelectTypeParameters=CR_CPU_MEMORY satisfies. On SUNK, CoreWeave already holds back memory for the operating system, kubelet, and DaemonSets, and Slurm advertises a reduced RealMemory to match. MemSpecLimit covers what that baseline misses: on-node sidecars and shmem peaks. See Avoid stranded GPUs for how RealMemory relates to what a job can request.

Don’t put MemSpecLimit in slurmConfig. That block renders into global slurm.conf scope, where Slurm rejects the key with _parse_next_key: Parsing error at unrecognized key: MemSpecLimit. On SUNK that parse failure restarts every Slurm pod and disrupts running jobs. MemSpecLimit is a per-node parameter and belongs on the node configuration. The same applies to CoreSpecCount and other per-node keys.

The CoreWeave AI Object Storage proxy sidecar, when present, can consume 30 to 50 GiB under heavy object-storage workloads. That usage is invisible to cAdvisor and to Slurm cgroup accounting on cgroup v1. Account for it in MemSpecLimit on nodes that run object-storage-heavy jobs. ConstrainRAMSpace=yes in cgroup.conf is what makes Slurm enforce per-job memory limits. SUNK sets it by default under slurmConfig.cgroupConfig. Two conditions make it effective:

SelectTypeParameters must track memory. SUNK defaults to CR_CPU_MEMORY, so check this only when someone has overridden it. Without memory as a consumable resource, cgroup memory enforcement may not work even with ConstrainRAMSpace=yes.
On cgroup v1, ConstrainRAMSpace=yes is the only thing bounding a Slurm job, because pod limits don’t constrain Slurm jobs. On cgroup v2, Slurm still manages its own memory cgroup slice, so ConstrainRAMSpace=yes is still required for Slurm-side enforcement.

Even with ConstrainRAMSpace=yes, heavy /dev/shm use can still cause a node-level out-of-memory event, because shmem allocations live in the host-level Slurm cgroup rather than the per-task cgroup that ConstrainRAMSpace bounds. For full cgroup configuration, see Manage resource binding with task plugins.

# cgroup2fs means cgroup v2; tmpfs means cgroup v1. Read-only.
stat -fc %T /sys/fs/cgroup/

JobContainerType=job_container/tmpfs scopes /dev/shm and temp directories per job so they’re cleaned up automatically. Set JobContainerType=job_container/tmpfs and add Contain to PrologFlags; Slurm requires PrologFlags=Contain for this plugin. The SUNK default PrologFlags is Alloc,Serial, so this is a change to make, not a default to rely on.

When to file a support ticket

Open a ticket when a node runs out of memory and you’ve confirmed the pressure isn’t reclaimable cache and have accounted for shmem and system overhead in MemSpecLimit. Include /proc/meminfo output, the cgroup version, and the job’s memory pattern (especially /dev/shm use).

Manage memory with cache-dropper: trigger the page cache dropper between jobs.
Manage resource binding with task plugins: cgroup configuration and constraints.
Avoid stranded GPUs: how RealMemory and DefMemPerCPU bound what a job can request.
Why do slurmd Pod metrics show almost no CPU or memory usage?: the same cgroup split, and which metrics to read instead.
Share storage across Slurm nodes: shared directory permissions and home directory hygiene.

​Read node memory, not pod metrics

​Compare Shmem and Cached

​Ask an administrator

​When to file a support ticket

​Related pages

Read node memory, not pod metrics

Compare `Shmem` and `Cached`

Ask an administrator

When to file a support ticket

Related pages