Why are Slurm commands not found inside a Pyxis container?

Custom container images don’t include Slurm CLI tools. sbatch, srun, squeue, sinfo, sacct, and scancel work on the login node and in a bare srun. Inside srun --container-image, they return command not found. When you can, submit dependent jobs from the login node. When a job must call sbatch or srun based on results it produces at runtime, mount the host Slurm CLI into the container. Chain jobs with dependencies so the container never needs Slurm binaries:

JOB_ID=$(sbatch --parsable train.sh)
sbatch --dependency=afterok:${JOB_ID} postprocess.sh

If the follow-on work is a set of similar tasks, use a job array instead of submitting children from inside the container.

Mount the host Slurm CLI

When a running job must call sbatch or srun based on runtime results, bind-mount the host Slurm binaries, configuration, Munge client library, and Munge socket. The following example is for an x86_64 node. Replace [USERNAME], [IMAGE], and [SCRIPT] with your values.

srun --no-container-entrypoint \
  --container-image=/mnt/home/[USERNAME]/[IMAGE].sqsh \
  --container-mounts=/mnt/home:/mnt/home,/etc/passwd:/etc/passwd,/usr/lib/slurm:/usr/lib/slurm,/etc/slurm:/etc/slurm,/usr/local/lib/slurm:/usr/local/lib/slurm,/run/munge:/run/munge,/usr/lib/x86_64-linux-gnu/libmunge.so.2:/usr/lib/x86_64-linux-gnu/libmunge.so.2,/usr/lib/x86_64-linux-gnu/libmunge.so.2.0.0:/usr/lib/x86_64-linux-gnu/libmunge.so.2.0.0,/usr/bin/sbatch:/usr/bin/sbatch,/usr/bin/srun:/usr/bin/srun,/usr/bin/squeue:/usr/bin/squeue,/usr/bin/sinfo:/usr/bin/sinfo,/usr/bin/sacct:/usr/bin/sacct,/usr/bin/scancel:/usr/bin/scancel \
  [SCRIPT]

GB200, GH200, and GB300 compute nodes are aarch64, where the Munge library is under /usr/lib/aarch64-linux-gnu instead. Confirm the paths on the node before you copy the list:

which sbatch
ldd $(which sbatch) | grep munge

Inside the container, point Slurm at the mounted configuration and confirm the controller is reachable:

export SLURM_CONF=/etc/slurm/slurm.conf
scontrol ping
sbatch --version

To launch another --container-image job from inside the container, also mount /opt/sunk:/opt/sunk and /etc/enroot:/etc/enroot.

Install `slurm-client` in the image

A Slurm client works only when its version is inside the range the cluster’s controller accepts. Slurm 24.11 and later accept commands from the current release and the three previous major releases. A client outside that window fails with a protocol version error even when the configuration and the Munge socket are mounted. Distribution packages are usually too old to qualify. Ubuntu 24.04 ships slurm-client 23.11, Debian 12 ships 22.05, and Ubuntu 22.04 ships 21.08. Run sbatch --version on the login node to read the cluster version, then compare it against the package your base image installs. When the versions line up, install the package in the image:

RUN apt-get update && apt-get install -y slurm-client libmunge2 \
  && rm -rf /var/lib/apt/lists/*

You still need to bind-mount /etc/slurm, /etc/munge, and /run/munge so the client can authenticate to the cluster. When the versions don’t line up, bind-mount the host Slurm CLI instead.

Workload Scheduling

​Submit from the login node

​Mount the host Slurm CLI

​Install slurm-client in the image

​Related

Submit from the login node

Mount the host Slurm CLI

Install `slurm-client` in the image

Related