> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Node-local storage and temporary files on Slurm nodes

> What backs /tmp on SUNK Slurm nodes and how to keep temporary files on NVMe instead of RAM

This page explains what backs the `/tmp` directory on SUNK Slurm nodes and how to keep temporary files on Node-local NVMe instead of RAM. It's for SUNK administrators and users who run Slurm jobs, especially containerized jobs that write large amounts of temporary data.

CoreWeave SUNK runs Slurm on Kubernetes for batch and burst workloads. Each Slurm node runs in a Kubernetes Pod on a CoreWeave Node, so it shares the same Node-local NVMe storage described in [About local storage](/products/storage/local-storage). The backing for `/tmp`, however, depends on whether the job runs inside a container.

## What backs the temporary directory

On a Slurm node, the directory that backs `/tmp` depends on whether the job runs inside an [enroot](https://github.com/NVIDIA/enroot) or Pyxis container:

| Job environment                                                | What backs `/tmp`                                                                                                          |
| :------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------- |
| Bare `srun` (no container)                                     | The Node's local NVMe RAID array (`/dev/md127`).                                                                           |
| Inside an enroot or Pyxis container (`srun --container-image`) | `tmpfs` (RAM). The container root filesystem is an overlay on the same NVMe array, but `/tmp` is a separate `tmpfs` mount. |

Inside a container, `tmpfs` is sized to a fraction of the Node's memory. For example, on a Node with 2 TiB of RAM, `/tmp` is roughly 1 TiB. Because `tmpfs` is RAM, writing temporary data to `/tmp` in a containerized job consumes system memory rather than disk.

## Keep temporary files on NVMe

When `/tmp` is `tmpfs`, a job that writes large temporary files there consumes RAM. The job can fail with a "no space left on device" (`ENOSPC`) error even though Node and volume disk-usage metrics look clean, because the data is filling memory rather than the NVMe array. Libraries that cache to a temporary directory (for example, some object-storage clients) are a common cause.

To keep temporary files on NVMe, set `TMPDIR` to an NVMe-backed path in your job script. Most programs and libraries honor `TMPDIR` when they create temporary files:

```bash theme={"system"}
export TMPDIR=/var/nvme/storage/${SLURM_JOB_ID}
mkdir -p "$TMPDIR"
```

<Warning>
  Setting `TMPDIR` with `#SBATCH --export=TMPDIR=/tmp` doesn't help, because inside an enroot container `/tmp` is still `tmpfs`. Point `TMPDIR` at an NVMe-backed path instead.
</Warning>

## Where enroot stores container data

enroot extracts and stores container data under the path set by `ENROOT_DATA_PATH`. On SUNK, this defaults to `/opt/sunk/tmp/enroot-data/user-$(id -u)`, an ephemeral directory backed by Node-local storage (XFS). This keeps container extraction on local NVMe rather than on shared network storage.

## Verify what backs a path

To confirm what backs `/tmp` (or any path) on a Node, check the mount source and filesystem type:

```bash theme={"system"}
findmnt -no SOURCE,FSTYPE /tmp
```

A `tmpfs` source means the path is RAM-backed. A device such as `/dev/md127` means it's on the Node-local NVMe RAID array. You can also use `df -h /tmp` to see the backing device and available space.

## Related

* [About local storage](/products/storage/local-storage): RAID layout, throughput, and benchmarking for Node-local NVMe storage.
* [Share storage across Slurm nodes](/products/sunk/manage_sunk/shared-storage): mount Persistent Volume Claims for data that must persist or be shared across nodes.
* [`emptyDir` volumes](/products/storage/local-storage#recommended-emptydir-volumes): NVMe-backed scratch space in a Kubernetes Pod, an alternative to `/tmp`.