Why does Pyxis report File already exists after a job is preempted?

After a job is preempted or requeued, Pyxis can fail to start with ERROR File already exists if a leftover container directory from the previous attempt is still on the node. Give each attempt a unique --container-name, or add an Epilog script that removes leftover pyxis_* directories after the job ends.

Symptoms

The next start after preemption or requeue fails with errors similar to the following:

pyxis: ERROR File already exists: /opt/sunk/tmp/enroot-data/user-[UID]/pyxis_[JOB-ID].[STEP]
pyxis: couldn't start container
spank: required plugin spank_pyxis.so: task_init failed with rc=-1

These errors are specific to a reused leftover directory. A pyxis: container exited too soon failure on a fresh launch is a different problem.

Check for leftover directories

The enroot runtime extracts container data under ENROOT_DATA_PATH. On SUNK, that path defaults to /opt/sunk/tmp/enroot-data/user-$(id -u). For details, see Node-local storage and temporary files on Slurm nodes. The slurmstepd error output names the failing node. You can also read it from sacct, then open a shell on that node without starting a container:

sacct -j [JOB-ID] --format=JobID,NodeList,State
srun --nodelist=[NODE-NAME] --pty bash

From that shell, list leftover Pyxis directories:

ls -la /opt/sunk/tmp/enroot-data/user-$(id -u)/pyxis_*

If a directory for the same job ID is still present, the next start that reuses that name fails.

Use a unique container name

Pass a --container-name that changes when Slurm restarts the job. Replace [USERNAME] and [IMAGE] with your values.

srun --container-image=/mnt/home/[USERNAME]/[IMAGE].sqsh \
  --container-name=job_${SLURM_JOB_ID}_r${SLURM_RESTART_COUNT:-0} \
  --container-mounts=/mnt/home:/mnt/home

A unique name avoids colliding with the leftover directory. It can break workflows that attach to a fixed name from a nested srun inside the same job. If your workflow needs a fixed name, use the Epilog cleanup instead.

Drop `--kill-on-bad-exit`

--kill-on-bad-exit terminates the step as soon as any task exits non-zero. Slurm then sends SIGKILL right away instead of its usual SIGTERM and KillWait sequence, which leaves Pyxis less time to remove the container directory. Dropping the flag makes leftover directories less frequent. It doesn’t prevent them.

Clean leftover directories in the Epilog

SUNK administrators can remove leftover directories in a job Epilog, which runs after the job has fully stopped. Add a script under /etc/slurm/epilog.d/ with compute.managedEpilogConfig or a user-managed Epilog ConfigMap. For details, see Run Prolog and Epilog scripts on SUNK.

#!/bin/bash
# A non-zero exit drains the node, so this script always exits 0.
shopt -s nullglob
if [ -n "${SLURM_JOB_UID}" ] && [ -n "${SLURM_JOB_ID}" ]; then
  for dir in /opt/sunk/tmp/enroot-data/user-${SLURM_JOB_UID}/pyxis_${SLURM_JOB_ID}.*; do
    [ -d "$dir" ] || continue
    enroot remove -f "$(basename "$dir")" || rm -rf "$dir"
  done
fi
exit 0

An Epilog that exits non-zero drains the node, so end the script with exit 0 and guard every path it touches. Removing extracted container data can also delay node teardown when the leftover filesystem is large, which holds the node before the next job starts. Keep the script short, and log the job ID. For details, see Failure handling.

Workload Scheduling Server Errors

​Symptoms

​Check for leftover directories

​Use a unique container name

​Drop --kill-on-bad-exit

​Clean leftover directories in the Epilog

​Related

Symptoms

Check for leftover directories

Use a unique container name

Drop `--kill-on-bad-exit`

Clean leftover directories in the Epilog

Related