> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why does Pyxis report File already exists after a job is preempted?

After a job is preempted or requeued, Pyxis can fail to start with `ERROR File already exists` if a leftover container directory from the previous attempt is still on the node. Give each attempt a unique `--container-name`, or add an Epilog script that removes leftover `pyxis_*` directories after the job ends.

## Symptoms

The next start after preemption or requeue fails with errors similar to the following:

```text theme={"system"}
pyxis: ERROR File already exists: /opt/sunk/tmp/enroot-data/user-[UID]/pyxis_[JOB-ID].[STEP]
pyxis: couldn't start container
spank: required plugin spank_pyxis.so: task_init failed with rc=-1
```

These errors are specific to a reused leftover directory. A `pyxis: container exited too soon` failure on a fresh launch is a different problem.

## Check for leftover directories

The enroot runtime extracts container data under `ENROOT_DATA_PATH`. On SUNK, that path defaults to `/opt/sunk/tmp/enroot-data/user-$(id -u)`. For details, see [Node-local storage and temporary files on Slurm nodes](/products/sunk/manage_sunk/node-local-storage-and-tmp#where-enroot-stores-container-data).

The `slurmstepd` error output names the failing node. You can also read it from `sacct`, then open a shell on that node without starting a container:

```bash theme={"system"}
sacct -j [JOB-ID] --format=JobID,NodeList,State
srun --nodelist=[NODE-NAME] --pty bash
```

From that shell, list leftover Pyxis directories:

```bash theme={"system"}
ls -la /opt/sunk/tmp/enroot-data/user-$(id -u)/pyxis_*
```

If a directory for the same job ID is still present, the next start that reuses that name fails.

## Use a unique container name

Pass a `--container-name` that changes when Slurm restarts the job. Replace `[USERNAME]` and `[IMAGE]` with your values.

```bash theme={"system"}
srun --container-image=/mnt/home/[USERNAME]/[IMAGE].sqsh \
  --container-name=job_${SLURM_JOB_ID}_r${SLURM_RESTART_COUNT:-0} \
  --container-mounts=/mnt/home:/mnt/home
```

A unique name avoids colliding with the leftover directory. It can break workflows that attach to a fixed name from a nested `srun` inside the same job. If your workflow needs a fixed name, use the Epilog cleanup instead.

## Drop `--kill-on-bad-exit`

`--kill-on-bad-exit` terminates the step as soon as any task exits non-zero. Slurm then sends `SIGKILL` right away instead of its usual `SIGTERM` and `KillWait` sequence, which leaves Pyxis less time to remove the container directory. Dropping the flag makes leftover directories less frequent. It doesn't prevent them.

## Clean leftover directories in the Epilog

SUNK administrators can remove leftover directories in a job Epilog, which runs after the job has fully stopped. Add a script under `/etc/slurm/epilog.d/` with `compute.managedEpilogConfig` or a user-managed Epilog ConfigMap. For details, see [Run Prolog and Epilog scripts on SUNK](/products/sunk/run_workloads/prolog-epilog).

```bash theme={"system"}
#!/bin/bash
# A non-zero exit drains the node, so this script always exits 0.
shopt -s nullglob
if [ -n "${SLURM_JOB_UID}" ] && [ -n "${SLURM_JOB_ID}" ]; then
  for dir in /opt/sunk/tmp/enroot-data/user-${SLURM_JOB_UID}/pyxis_${SLURM_JOB_ID}.*; do
    [ -d "$dir" ] || continue
    enroot remove -f "$(basename "$dir")" || rm -rf "$dir"
  done
fi
exit 0
```

<Warning>
  An Epilog that exits non-zero drains the node, so end the script with `exit 0` and guard every path it touches. Removing extracted container data can also delay node teardown when the leftover filesystem is large, which holds the node before the next job starts. Keep the script short, and log the job ID. For details, see [Failure handling](/products/sunk/run_workloads/prolog-epilog#failure-handling).
</Warning>

## Related

* [Node-local storage and temporary files on Slurm nodes](/products/sunk/manage_sunk/node-local-storage-and-tmp)
* [Run Prolog and Epilog scripts on SUNK](/products/sunk/run_workloads/prolog-epilog)
* [How do I run containers with Pyxis and Enroot in Slurm?](/support/sunk/articles/how-do-i-run-containers-with-pyxis-and-enroot-in-slurm)

***

<Badge stroke shape="pill" color="blue" size="md">[Workload Scheduling](/support/sunk/tags/workload-scheduling)</Badge><Badge stroke shape="pill" color="blue" size="md">[Server Errors](/support/sunk/tags/server-errors)</Badge>


## Related topics

- [Server Errors](/support/sunk/tags/server-errors.md)