This page explains what backs the /tmp directory on SUNK Slurm nodes and how to keep temporary files on Node-local NVMe instead of RAM. It’s for SUNK administrators and users who run Slurm jobs, especially containerized jobs that write large amounts of temporary data.
CoreWeave SUNK runs Slurm on Kubernetes for batch and burst workloads. Each Slurm node runs in a Kubernetes Pod on a CoreWeave Node, so it shares the same Node-local NVMe storage described in About local storage. The backing for /tmp, however, depends on whether the job runs inside a container.
What backs the temporary directory
On a Slurm node, the directory that backs /tmp depends on whether the job runs inside an enroot or Pyxis container:
| Job environment | What backs /tmp |
|---|
Bare srun (no container) | The Node’s local NVMe RAID array (/dev/md127). |
Inside an enroot or Pyxis container (srun --container-image) | tmpfs (RAM). The container root filesystem is an overlay on the same NVMe array, but /tmp is a separate tmpfs mount. |
tmpfs is sized to a fraction of the Node’s memory. For example, on a Node with 2 TiB of RAM, /tmp is roughly 1 TiB. Because tmpfs is RAM, writing temporary data to /tmp in a containerized job consumes system memory rather than disk.
Keep temporary files on NVMe
When /tmp is tmpfs, a job that writes large temporary files there consumes RAM. The job can fail with a “no space left on device” (ENOSPC) error even though Node and volume disk-usage metrics look clean, because the data is filling memory rather than the NVMe array. Libraries that cache to a temporary directory (for example, some object-storage clients) are a common cause.
To keep temporary files on NVMe, set TMPDIR to an NVMe-backed path in your job script. Most programs and libraries honor TMPDIR when they create temporary files:
export TMPDIR=/var/nvme/storage/${SLURM_JOB_ID}
mkdir -p "$TMPDIR"
Setting TMPDIR with #SBATCH --export=TMPDIR=/tmp doesn’t help, because inside an enroot container /tmp is still tmpfs. Point TMPDIR at an NVMe-backed path instead.
Where enroot stores container data
enroot extracts and stores container data under the path set by ENROOT_DATA_PATH. On SUNK, this defaults to /opt/sunk/tmp/enroot-data/user-$(id -u), an ephemeral directory backed by Node-local storage (XFS). This keeps container extraction on local NVMe rather than on shared network storage.
Verify what backs a path
To confirm what backs /tmp (or any path) on a Node, check the mount source and filesystem type:
findmnt -no SOURCE,FSTYPE /tmp
tmpfs source means the path is RAM-backed. A device such as /dev/md127 means it’s on the Node-local NVMe RAID array. You can also use df -h /tmp to see the backing device and available space.
