> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Handle Slurm job failures

> Observe job status, retry failed Slurm jobs, and resume training and experiment tracking

[SUNK Standard](https://docs.coreweave.com/products/sunk.md)

[SUNK Self-Service](https://docs.coreweave.com/products/sunk/about-self-service-sunk.md)

Recover a failed Slurm batch job on SUNK by requeuing the existing job or submitting a corrected batch script:

* Requeue the job to preserve the Slurm job ID and restart the submitted script.
* Resubmit the job to create a new job ID and use the script you submit.

With each approach, your training application loads a checkpoint to recover progress.

## Prerequisites

Before you begin, verify the following:

* You can access a [Slurm login node](https://docs.coreweave.com/products/sunk/access_sunk/connect-to-slurm-login-node.md) as the job owner.
* You have the failed job's ID, batch script, and application logs.
* To recover training progress, you have a valid checkpoint that every node the job might use can access, and your training code loads it. See [Checkpoint and restart Slurm jobs after a node failure](https://docs.coreweave.com/products/sunk/run_workloads/checkpoint-and-restart.md).
* To resume a Weights & Biases run, the `wandb` library is installed and authenticated in the training environment. See [Install the `wandb` library and log in](https://docs.coreweave.com/products/wandb/quickstart#install-the-wandb-library-and-log-in).

## Inspect a failed job

Before starting another attempt, check whether Slurm has already requeued the job. An earlier failed attempt can appear in accounting while the same job ID is pending or running again.

**1. Inspect the job history**

From a login node, inspect the job's history. Replace `JOB_ID` with your job ID:

```bash theme={"system"}
sacct -j JOB_ID --duplicates --format=JobID,JobName,State,ExitCode,Submit,Start,End,NodeList
```

`--duplicates` includes earlier records for a requeued job. Use the `Submit` and `Start` fields to distinguish attempts. Inspect both the job row and the step rows. A batch script that exits successfully can hide a failed application step.

**2. Check the current queue**

Check the current queue:

```bash theme={"system"}
squeue -j JOB_ID
```

If the job is already `PENDING` or `RUNNING` after a requeue, monitor that attempt instead of submitting another copy. For pending reasons and state definitions, see [Monitor Slurm job states](https://docs.coreweave.com/products/sunk/manage_sunk/slurm-job-states.md).

**3. Identify the failure**

Read the application error log and identify the failure before retrying:

| Failure | What to check |
| - | - |
| `NODE_FAIL` | A node failed or became unresponsive. Check whether Slurm already requeued the job and whether the application can load its checkpoint. |
| `FAILED` | Inspect the step exit code and application log. Fix persistent errors, such as an invalid path or application bug, before retrying. |
| `OUT_OF_MEMORY` or `TIMEOUT` | Review memory use or the requested time limit. Update the workload or resource request before resubmitting. |
| Prolog failure | Slurm requeues the job and drains the affected node. Check the [Prolog failure behavior](https://docs.coreweave.com/products/sunk/run_workloads/prolog-epilog.md#failure-handling), including whether the job is held. |

Use the [Slurm Job Metrics dashboard](https://docs.coreweave.com/observability/managed-grafana/sunk/slurm-job-metrics.md) to correlate the job's state timeline with node alerts. For hardware recovery, see [Fault tolerance in SUNK](https://docs.coreweave.com/products/sunk/discover_sunk/fault-tolerance.md). For application exits caused by signals, see [Training job exit codes](https://docs.coreweave.com/support/sunk/articles/what-do-training-job-exit-codes-mean).

## Choose requeue or resubmission

| Choice | Use it when | Job identity and script |
| - | - | - |
| Requeue with `scontrol requeue` | The submitted job is still suitable and you want another attempt after a recoverable failure. | Keeps the job ID and restarts the batch script Slurm stored at submission. Changes to the local batch script don't change that stored copy. |
| Resubmit with `sbatch` | You need to submit a corrected batch script or different resource requests, or Slurm can no longer requeue the old job. | Creates a new job ID and reads the batch script again. Keep the old ID for failure investigation. |

Requeue the job when the existing submission is suitable, because attempts remain associated with one job ID. Neither choice resumes the application automatically: the batch script starts from the beginning, and the application must restore its saved state. External files referenced by the script aren't copied by `sbatch`. Keep the intended code and checkpoint available.

## Requeue a batch job

Only batch jobs submitted with `sbatch` can be requeued. The job must be eligible for requeue and still be known to the Slurm controller. A job is eligible if you submitted it with `--requeue`, or if you submitted it without `--no-requeue` on a cluster that keeps Slurm's default `JobRequeue=1` setting. To check both conditions, run `scontrol show job JOB_ID` and look for `Requeue=1` in the output. A record in `sacct` alone doesn't establish that the job can be requeued. If requeue is turned off, add `--requeue` to the corrected batch script and resubmit it. If the controller no longer has the job, resubmit it.

**1. Check the original script**

Confirm that the original script loads the intended checkpoint and can safely repeat setup steps. If you need to change the batch script, use [resubmission](#resubmit-a-corrected-job).

If the job wasn't submitted with `--open-mode=append`, save copies of its current output and error logs before requeuing. The restart can truncate these files.

**2. Requeue the job**

After confirming that a retry isn't already pending or running, requeue the job:

```bash theme={"system"}
scontrol requeue JOB_ID
```

> [!WARNING]
>
> When you requeue a running job, Slurm terminates its current allocation. Don't use this command as a status check.

**3. Verify the restarted job**

Check `squeue -j JOB_ID` for the new attempt. When resources are available, the job runs under the same ID. Confirm in the application log that training loaded the expected checkpoint, then check that training steps or metrics advance.

For retry history, use the `sacct --duplicates` command in [Inspect a failed job](#inspect-a-failed-job). For a job still known to the controller, `scontrol show job JOB_ID` also reports the `Restarts` count. See the Slurm [`requeue` reference](https://slurm.schedmd.com/scontrol.html#OPT_requeue).

## Resubmit a corrected job

To submit a corrected job, follow these steps:

**1. Fix the failure**

Fix the failure in the application, batch script, or resource request. Ensure the previous job isn't pending or running before submitting another copy.

**2. Select the previous checkpoint**

Point the corrected script at the checkpoint from the previous attempt. A new submission has a new `SLURM_JOB_ID` and no requeue history. A checkpoint path derived only from the new ID or a check of `SLURM_RESTART_COUNT` doesn't find the old run's state. Use your application's explicit checkpoint path or stable experiment identifier.

**3. Submit the corrected script**

Submit the corrected script. Replace `BATCH_SCRIPT` with its path:

```bash theme={"system"}
sbatch BATCH_SCRIPT
```

Save the new job ID returned by `sbatch`, alongside the old ID.

**4. Verify the new job**

Check the new ID with `squeue` and `sacct`. Confirm in the application log that the intended checkpoint loaded and that training progresses. If the same error repeats, investigate it before starting more attempts.

## Automate retries for recoverable failures

For node failures, add `#SBATCH --requeue` to make the job's eligibility explicit. Add `#SBATCH --open-mode=append` to preserve output across attempts. Follow the [checkpoint and restart guide](https://docs.coreweave.com/products/sunk/run_workloads/checkpoint-and-restart.md#make-your-job-explicitly-requeueable) for setup. `--requeue` alone doesn't retry arbitrary application errors.

For application failures, choose exit codes that specifically mean a retry is appropriate. Your cluster administrator can configure [`RequeueExit` or `RequeueExitHold`](https://docs.coreweave.com/products/sunk/discover_sunk/fault-tolerance#requeue-on-application-exit-codes). Slurm evaluates the batch script's exit code, so preserve the launcher's failure status instead of letting a later successful command hide it.

Alternatively, the batch script can request its own requeue. The following excerpt retries exit code `42` up to three times. These are example policy values. Use `42` only if your launch command returns it for a recoverable failure. Choose a retry limit for your workload.

When the script stops retrying, it exits with the launch command's exit code, so accounting records the original failure. Don't use a retry code that's listed in the cluster's `RequeueExit` or `RequeueExitHold` settings, because Slurm requeues a job that exits with a listed code regardless of the script's retry limit. For multi-process jobs, test the status returned by `srun`. A failed task or a signal that terminates another task can affect the final exit code.

Replace the `srun` command with your checkpoint-aware training launch command. Place this excerpt after your `#SBATCH` directives and environment setup:

```bash title="Batch-script retry excerpt" theme={"system"}
MAX_RESTARTS=3
RETRY_EXIT_CODE=42

if [ "${SLURM_RESTART_COUNT:-0}" -gt "$MAX_RESTARTS" ]; then
    echo "Restart limit reached; inspect the failure before retrying." >&2
    exit 1
fi

exit_code=0
srun --kill-on-bad-exit=1 python3 train.py || exit_code=$?

# Success and nonretryable failures keep the launch command's exit code.
if [ "$exit_code" -ne "$RETRY_EXIT_CODE" ]; then
    exit "$exit_code"
fi

if [ "${SLURM_RESTART_COUNT:-0}" -ge "$MAX_RESTARTS" ]; then
    echo "Restart limit reached; inspect the failure before retrying." >&2
    exit "$exit_code"
fi

if ! scontrol requeue "$SLURM_JOB_ID"; then
    echo "Requeue request failed." >&2
    exit "$exit_code"
fi

exit 0
```

The `|| exit_code=$?` list captures a failed launch even when the surrounding script uses `set -e`. `SLURM_RESTART_COUNT` counts all requeues, including node-failure restarts. This policy allows the initial attempt and up to three restarted attempts. Coordinate the script policy with any cluster-wide exit-code policy so another rule doesn't keep requeuing terminal failures.

For distributed jobs, `--kill-on-bad-exit=1` stops the remaining tasks in the step when one fails. If you use Pyxis, also review the [container cleanup considerations](https://docs.coreweave.com/products/sunk/discover_sunk/fault-tolerance.md#end-a-step-when-one-rank-fails). For time limits and preemption, see [Handle Slurm signals for graceful shutdown](https://docs.coreweave.com/products/sunk/run_workloads/handle-slurm-signals.md). A hard node failure might leave no opportunity for an exit handler, so save checkpoints periodically.

## Resume a Weights & Biases run

Treat the Slurm job ID, training checkpoint, and Weights & Biases run ID as separate identifiers. To continue one experiment across retries, persist the run ID with the experiment's checkpoint configuration and reuse it after either requeue or resubmission. Don't generate a new run ID on every batch-script start.

In the process responsible for logging, initialize Weights & Biases with that ID and `resume="allow"`. Replace the placeholders with your entity, project, and persistent run ID:

```python theme={"system"}
import wandb

run = wandb.init(
    entity="ENTITY",
    project="PROJECT",
    id="RUN_ID",
    resume="allow",
)
```

This initialization excerpt creates the run if it doesn't exist and resumes it if it does. When the run must already exist, use `resume="must"`. Keep the same entity and project, and don't let independent processes write to the same run ID concurrently. See [Resume a run](https://docs.coreweave.com/products/wandb/runs/resuming.md) and [Log distributed training experiments](https://docs.coreweave.com/products/wandb/track/log/distributed-training.md).

A resumed Weights & Biases run continues metric logging. Your training code must separately restore the model, optimizer, and training position from the checkpoint. Resumption continues after the last recorded history step. The history step is separate from your application's training position.

Weights & Biases ignores explicit step values older than its current logging step. Resumption doesn't overwrite previously logged steps. See the [`Run.log()` reference](https://docs.coreweave.com/products/wandb/ref/python/experiments/run#method-runlog). For options when your checkpoint predates the logged history, see [Fork a run](https://docs.coreweave.com/products/wandb/runs/forking.md), including its availability requirements.

After restart, verify that metrics appear under the intended run ID and that the application reports the expected checkpoint. If you use W\&B Sweeps, Slurm requeue and sweep-run requeue are separate mechanisms. Follow [Signal handling and sweep runs](https://docs.coreweave.com/products/wandb/sweeps/signal-handling-sweep-runs.md) for `mark_preempting()`, signal forwarding, and exit status. Don't mark every application failure as preemption.
