- Requeue the job to preserve the Slurm job ID and restart the submitted script.
- Resubmit the job to create a new job ID and use the script you submit.
Prerequisites
Before you begin, verify the following:- You can access a Slurm login node as the job owner.
- You have the failed job’s ID, batch script, and application logs.
- To recover training progress, you have a valid checkpoint that every node the job might use can access, and your training code loads it. See Checkpoint and restart Slurm jobs after a node failure.
- To resume a Weights & Biases run, the
wandblibrary is installed and authenticated in the training environment. See Install thewandblibrary and log in.
Inspect a failed job
Before starting another attempt, check whether Slurm has already requeued the job. An earlier failed attempt can appear in accounting while the same job ID is pending or running again.1
Inspect the job history
From a login node, inspect the job’s history. Replace
JOB_ID with your job ID:--duplicates includes earlier records for a requeued job. Use the Submit and Start fields to distinguish attempts. Inspect both the job row and the step rows. A batch script that exits successfully can hide a failed application step.2
Check the current queue
Check the current queue:If the job is already
PENDING or RUNNING after a requeue, monitor that attempt instead of submitting another copy. For pending reasons and state definitions, see Monitor Slurm job states.3
Identify the failure
Read the application error log and identify the failure before retrying:
Use the Slurm Job Metrics dashboard to correlate the job’s state timeline with node alerts. For hardware recovery, see Fault tolerance in SUNK. For application exits caused by signals, see Training job exit codes.
Choose requeue or resubmission
Requeue the job when the existing submission is suitable, because attempts remain associated with one job ID. Neither choice resumes the application automatically: the batch script starts from the beginning, and the application must restore its saved state. External files referenced by the script aren’t copied by
sbatch. Keep the intended code and checkpoint available.
Requeue a batch job
Only batch jobs submitted withsbatch can be requeued. The job must be eligible for requeue and still be known to the Slurm controller. A job is eligible if you submitted it with --requeue, or if you submitted it without --no-requeue on a cluster that keeps Slurm’s default JobRequeue=1 setting. To check both conditions, run scontrol show job JOB_ID and look for Requeue=1 in the output. A record in sacct alone doesn’t establish that the job can be requeued. If requeue is turned off, add --requeue to the corrected batch script and resubmit it. If the controller no longer has the job, resubmit it.
1
Check the original script
Confirm that the original script loads the intended checkpoint and can safely repeat setup steps. If you need to change the batch script, use resubmission.If the job wasn’t submitted with
--open-mode=append, save copies of its current output and error logs before requeuing. The restart can truncate these files.2
Requeue the job
After confirming that a retry isn’t already pending or running, requeue the job:
3
Verify the restarted job
Check
squeue -j JOB_ID for the new attempt. When resources are available, the job runs under the same ID. Confirm in the application log that training loaded the expected checkpoint, then check that training steps or metrics advance.sacct --duplicates command in Inspect a failed job. For a job still known to the controller, scontrol show job JOB_ID also reports the Restarts count. See the Slurm requeue reference.
Resubmit a corrected job
To submit a corrected job, follow these steps:1
Fix the failure
Fix the failure in the application, batch script, or resource request. Ensure the previous job isn’t pending or running before submitting another copy.
2
Select the previous checkpoint
Point the corrected script at the checkpoint from the previous attempt. A new submission has a new
SLURM_JOB_ID and no requeue history. A checkpoint path derived only from the new ID or a check of SLURM_RESTART_COUNT doesn’t find the old run’s state. Use your application’s explicit checkpoint path or stable experiment identifier.3
Submit the corrected script
Submit the corrected script. Replace Save the new job ID returned by
BATCH_SCRIPT with its path:sbatch, alongside the old ID.4
Verify the new job
Check the new ID with
squeue and sacct. Confirm in the application log that the intended checkpoint loaded and that training progresses. If the same error repeats, investigate it before starting more attempts.Automate retries for recoverable failures
For node failures, add#SBATCH --requeue to make the job’s eligibility explicit. Add #SBATCH --open-mode=append to preserve output across attempts. Follow the checkpoint and restart guide for setup. --requeue alone doesn’t retry arbitrary application errors.
For application failures, choose exit codes that specifically mean a retry is appropriate. Your cluster administrator can configure RequeueExit or RequeueExitHold. Slurm evaluates the batch script’s exit code, so preserve the launcher’s failure status instead of letting a later successful command hide it.
Alternatively, the batch script can request its own requeue. The following excerpt retries exit code 42 up to three times. These are example policy values. Use 42 only if your launch command returns it for a recoverable failure. Choose a retry limit for your workload.
When the script stops retrying, it exits with the launch command’s exit code, so accounting records the original failure. Don’t use a retry code that’s listed in the cluster’s RequeueExit or RequeueExitHold settings, because Slurm requeues a job that exits with a listed code regardless of the script’s retry limit. For multi-process jobs, test the status returned by srun. A failed task or a signal that terminates another task can affect the final exit code.
Replace the srun command with your checkpoint-aware training launch command. Place this excerpt after your #SBATCH directives and environment setup:
Batch-script retry excerpt
|| exit_code=$? list captures a failed launch even when the surrounding script uses set -e. SLURM_RESTART_COUNT counts all requeues, including node-failure restarts. This policy allows the initial attempt and up to three restarted attempts. Coordinate the script policy with any cluster-wide exit-code policy so another rule doesn’t keep requeuing terminal failures.
For distributed jobs, --kill-on-bad-exit=1 stops the remaining tasks in the step when one fails. If you use Pyxis, also review the container cleanup considerations. For time limits and preemption, see Handle Slurm signals for graceful shutdown. A hard node failure might leave no opportunity for an exit handler, so save checkpoints periodically.
Resume a Weights & Biases run
Treat the Slurm job ID, training checkpoint, and Weights & Biases run ID as separate identifiers. To continue one experiment across retries, persist the run ID with the experiment’s checkpoint configuration and reuse it after either requeue or resubmission. Don’t generate a new run ID on every batch-script start. In the process responsible for logging, initialize Weights & Biases with that ID andresume="allow". Replace the placeholders with your entity, project, and persistent run ID:
resume="must". Keep the same entity and project, and don’t let independent processes write to the same run ID concurrently. See Resume a run and Log distributed training experiments.
A resumed Weights & Biases run continues metric logging. Your training code must separately restore the model, optimizer, and training position from the checkpoint. Resumption continues after the last recorded history step. The history step is separate from your application’s training position.
Weights & Biases ignores explicit step values older than its current logging step. Resumption doesn’t overwrite previously logged steps. See the Run.log() reference. For options when your checkpoint predates the logged history, see Fork a run, including its availability requirements.
After restart, verify that metrics appear under the intended run ID and that the application reports the expected checkpoint. If you use W&B Sweeps, Slurm requeue and sweep-run requeue are separate mechanisms. Follow Signal handling and sweep runs for mark_preempting(), signal forwarding, and exit status. Don’t mark every application failure as preemption.