> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why are Slurm commands not found inside a Pyxis container?

Custom container images don't include Slurm CLI tools. `sbatch`, `srun`, `squeue`, `sinfo`, `sacct`, and `scancel` work on the login node and in a bare `srun`. Inside `srun --container-image`, they return `command not found`. When you can, submit dependent jobs from the login node. When a job must call `sbatch` or `srun` based on results it produces at runtime, mount the host Slurm CLI into the container.

## Submit from the login node

Chain jobs with dependencies so the container never needs Slurm binaries:

```bash theme={"system"}
JOB_ID=$(sbatch --parsable train.sh)
sbatch --dependency=afterok:${JOB_ID} postprocess.sh
```

If the follow-on work is a set of similar tasks, use a [job array](https://slurm.schedmd.com/job_array.html) instead of submitting children from inside the container.

## Mount the host Slurm CLI

When a running job must call `sbatch` or `srun` based on runtime results, bind-mount the host Slurm binaries, configuration, Munge client library, and Munge socket. The following example is for an x86\_64 node. Replace `[USERNAME]`, `[IMAGE]`, and `[SCRIPT]` with your values.

```bash theme={"system"}
srun --no-container-entrypoint \
  --container-image=/mnt/home/[USERNAME]/[IMAGE].sqsh \
  --container-mounts=/mnt/home:/mnt/home,/etc/passwd:/etc/passwd,/usr/lib/slurm:/usr/lib/slurm,/etc/slurm:/etc/slurm,/usr/local/lib/slurm:/usr/local/lib/slurm,/run/munge:/run/munge,/usr/lib/x86_64-linux-gnu/libmunge.so.2:/usr/lib/x86_64-linux-gnu/libmunge.so.2,/usr/lib/x86_64-linux-gnu/libmunge.so.2.0.0:/usr/lib/x86_64-linux-gnu/libmunge.so.2.0.0,/usr/bin/sbatch:/usr/bin/sbatch,/usr/bin/srun:/usr/bin/srun,/usr/bin/squeue:/usr/bin/squeue,/usr/bin/sinfo:/usr/bin/sinfo,/usr/bin/sacct:/usr/bin/sacct,/usr/bin/scancel:/usr/bin/scancel \
  [SCRIPT]
```

GB200, GH200, and GB300 compute nodes are aarch64, where the Munge library is under `/usr/lib/aarch64-linux-gnu` instead. Confirm the paths on the node before you copy the list:

```bash theme={"system"}
which sbatch
ldd $(which sbatch) | grep munge
```

Inside the container, point Slurm at the mounted configuration and confirm the controller is reachable:

```bash theme={"system"}
export SLURM_CONF=/etc/slurm/slurm.conf
scontrol ping
sbatch --version
```

To launch another `--container-image` job from inside the container, also mount `/opt/sunk:/opt/sunk` and `/etc/enroot:/etc/enroot`.

## Install `slurm-client` in the image

A Slurm client works only when its version is inside the range the cluster's controller accepts. Slurm 24.11 and later accept commands from the current release and the three previous major releases. A client outside that window fails with a protocol version error even when the configuration and the Munge socket are mounted.

Distribution packages are usually too old to qualify. Ubuntu 24.04 ships `slurm-client` 23.11, Debian 12 ships 22.05, and Ubuntu 22.04 ships 21.08. Run `sbatch --version` on the login node to read the cluster version, then compare it against the package your base image installs.

When the versions line up, install the package in the image:

```dockerfile theme={"system"}
RUN apt-get update && apt-get install -y slurm-client libmunge2 \
  && rm -rf /var/lib/apt/lists/*
```

You still need to bind-mount `/etc/slurm`, `/etc/munge`, and `/run/munge` so the client can authenticate to the cluster. When the versions don't line up, bind-mount the host Slurm CLI instead.

## Related

* [How do I run containers with Pyxis and Enroot in Slurm?](/support/sunk/articles/how-do-i-run-containers-with-pyxis-and-enroot-in-slurm)
* [Submit a training job with PyTorch or TensorFlow](/products/sunk/tutorials/train-on-sunk/3-submit-a-training-job)
* [Connect to the Slurm login node](/products/sunk/access_sunk/connect-to-slurm-login-node)

***

<Badge stroke shape="pill" color="blue" size="md">[Workload Scheduling](/support/sunk/tags/workload-scheduling)</Badge>