Skip to main content
SUNK Standard SUNK Self-Service Recover a failed Slurm batch job on SUNK by requeuing the existing job or submitting a corrected batch script:
  • Requeue the job to preserve the Slurm job ID and restart the submitted script.
  • Resubmit the job to create a new job ID and use the script you submit.
With each approach, your training application loads a checkpoint to recover progress.

Prerequisites

Before you begin, verify the following:

Inspect a failed job

Before starting another attempt, check whether Slurm has already requeued the job. An earlier failed attempt can appear in accounting while the same job ID is pending or running again.
1

Inspect the job history

From a login node, inspect the job’s history. Replace JOB_ID with your job ID:
--duplicates includes earlier records for a requeued job. Use the Submit and Start fields to distinguish attempts. Inspect both the job row and the step rows. A batch script that exits successfully can hide a failed application step.
2

Check the current queue

Check the current queue:
If the job is already PENDING or RUNNING after a requeue, monitor that attempt instead of submitting another copy. For pending reasons and state definitions, see Monitor Slurm job states.
3

Identify the failure

Read the application error log and identify the failure before retrying:Use the Slurm Job Metrics dashboard to correlate the job’s state timeline with node alerts. For hardware recovery, see Fault tolerance in SUNK. For application exits caused by signals, see Training job exit codes.

Choose requeue or resubmission

Requeue the job when the existing submission is suitable, because attempts remain associated with one job ID. Neither choice resumes the application automatically: the batch script starts from the beginning, and the application must restore its saved state. External files referenced by the script aren’t copied by sbatch. Keep the intended code and checkpoint available.

Requeue a batch job

Only batch jobs submitted with sbatch can be requeued. The job must be eligible for requeue and still be known to the Slurm controller. A job is eligible if you submitted it with --requeue, or if you submitted it without --no-requeue on a cluster that keeps Slurm’s default JobRequeue=1 setting. To check both conditions, run scontrol show job JOB_ID and look for Requeue=1 in the output. A record in sacct alone doesn’t establish that the job can be requeued. If requeue is turned off, add --requeue to the corrected batch script and resubmit it. If the controller no longer has the job, resubmit it.
1

Check the original script

Confirm that the original script loads the intended checkpoint and can safely repeat setup steps. If you need to change the batch script, use resubmission.If the job wasn’t submitted with --open-mode=append, save copies of its current output and error logs before requeuing. The restart can truncate these files.
2

Requeue the job

After confirming that a retry isn’t already pending or running, requeue the job:
When you requeue a running job, Slurm terminates its current allocation. Don’t use this command as a status check.
3

Verify the restarted job

Check squeue -j JOB_ID for the new attempt. When resources are available, the job runs under the same ID. Confirm in the application log that training loaded the expected checkpoint, then check that training steps or metrics advance.
For retry history, use the sacct --duplicates command in Inspect a failed job. For a job still known to the controller, scontrol show job JOB_ID also reports the Restarts count. See the Slurm requeue reference.

Resubmit a corrected job

To submit a corrected job, follow these steps:
1

Fix the failure

Fix the failure in the application, batch script, or resource request. Ensure the previous job isn’t pending or running before submitting another copy.
2

Select the previous checkpoint

Point the corrected script at the checkpoint from the previous attempt. A new submission has a new SLURM_JOB_ID and no requeue history. A checkpoint path derived only from the new ID or a check of SLURM_RESTART_COUNT doesn’t find the old run’s state. Use your application’s explicit checkpoint path or stable experiment identifier.
3

Submit the corrected script

Submit the corrected script. Replace BATCH_SCRIPT with its path:
Save the new job ID returned by sbatch, alongside the old ID.
4

Verify the new job

Check the new ID with squeue and sacct. Confirm in the application log that the intended checkpoint loaded and that training progresses. If the same error repeats, investigate it before starting more attempts.

Automate retries for recoverable failures

For node failures, add #SBATCH --requeue to make the job’s eligibility explicit. Add #SBATCH --open-mode=append to preserve output across attempts. Follow the checkpoint and restart guide for setup. --requeue alone doesn’t retry arbitrary application errors. For application failures, choose exit codes that specifically mean a retry is appropriate. Your cluster administrator can configure RequeueExit or RequeueExitHold. Slurm evaluates the batch script’s exit code, so preserve the launcher’s failure status instead of letting a later successful command hide it. Alternatively, the batch script can request its own requeue. The following excerpt retries exit code 42 up to three times. These are example policy values. Use 42 only if your launch command returns it for a recoverable failure. Choose a retry limit for your workload. When the script stops retrying, it exits with the launch command’s exit code, so accounting records the original failure. Don’t use a retry code that’s listed in the cluster’s RequeueExit or RequeueExitHold settings, because Slurm requeues a job that exits with a listed code regardless of the script’s retry limit. For multi-process jobs, test the status returned by srun. A failed task or a signal that terminates another task can affect the final exit code. Replace the srun command with your checkpoint-aware training launch command. Place this excerpt after your #SBATCH directives and environment setup:
Batch-script retry excerpt
The || exit_code=$? list captures a failed launch even when the surrounding script uses set -e. SLURM_RESTART_COUNT counts all requeues, including node-failure restarts. This policy allows the initial attempt and up to three restarted attempts. Coordinate the script policy with any cluster-wide exit-code policy so another rule doesn’t keep requeuing terminal failures. For distributed jobs, --kill-on-bad-exit=1 stops the remaining tasks in the step when one fails. If you use Pyxis, also review the container cleanup considerations. For time limits and preemption, see Handle Slurm signals for graceful shutdown. A hard node failure might leave no opportunity for an exit handler, so save checkpoints periodically.

Resume a Weights & Biases run

Treat the Slurm job ID, training checkpoint, and Weights & Biases run ID as separate identifiers. To continue one experiment across retries, persist the run ID with the experiment’s checkpoint configuration and reuse it after either requeue or resubmission. Don’t generate a new run ID on every batch-script start. In the process responsible for logging, initialize Weights & Biases with that ID and resume="allow". Replace the placeholders with your entity, project, and persistent run ID:
This initialization excerpt creates the run if it doesn’t exist and resumes it if it does. When the run must already exist, use resume="must". Keep the same entity and project, and don’t let independent processes write to the same run ID concurrently. See Resume a run and Log distributed training experiments. A resumed Weights & Biases run continues metric logging. Your training code must separately restore the model, optimizer, and training position from the checkpoint. Resumption continues after the last recorded history step. The history step is separate from your application’s training position. Weights & Biases ignores explicit step values older than its current logging step. Resumption doesn’t overwrite previously logged steps. See the Run.log() reference. For options when your checkpoint predates the logged history, see Fork a run, including its availability requirements. After restart, verify that metrics appear under the intended run ID and that the application reports the expected checkpoint. If you use W&B Sweeps, Slurm requeue and sweep-run requeue are separate mechanisms. Follow Signal handling and sweep runs for mark_preempting(), signal forwarding, and exit status. Don’t mark every application failure as preemption.
Last modified on November 20, 2026