Skip to main content
When the node running a Slurm job on SUNK fails, for example from a hardware fault, Slurm requeues the job and restarts the batch script from the beginning. Without a checkpoint to resume from, this means losing all progress made before the failure. This guide shows you how to make a Slurm job checkpoint-aware. It becomes explicit about being requeued, able to detect that it was restarted, and able to resume from its last saved state instead of starting over. This guide covers hardware-triggered restarts: a job that requeues because the node running it failed. For user-initiated or scheduler-initiated stops, such as scancel, a job hitting its time limit, or preemption, see Handle Slurm signals for graceful shutdown.

​
How SUNK handles a node failure

When a node running a Slurm job stops responding or fails a health check, Slurm marks that node DOWN and terminates the job with a NODE_FAIL state. SUNK doesn’t override Slurm’s cluster-wide JobRequeue setting, which defaults to 1. This means jobs are eligible for requeue after a node failure without any special configuration. Slurm returns the job to the queue and, once resources are available, restarts the batch script from the beginning using the same job ID. The job’s output and error files, and its accounting record, continue under that same ID. This automatic requeue is documented Slurm behavior, not a CoreWeave customization. A cluster administrator can require jobs to opt in explicitly by setting JobRequeue=0. If requeue behavior on your cluster doesn’t match what’s described here, check that setting with your cluster administrator.

​
Prerequisites

Before you start, you must have the following:
  • A training script that already checkpoints its own state, for example by periodically calling torch.save(). This guide assumes your framework can save and load a checkpoint. It doesn’t cover how to implement checkpointing itself.
  • A writable, shared checkpoint location that every node in the job can reach, such as CoreWeave AI Object Storage.
  • A working SUNK cluster where you can submit training jobs.

​
Make your job explicitly requeueable

Add --requeue to your sbatch script to make the job’s requeue eligibility explicit, regardless of the cluster’s JobRequeue default:
Pair it with --open-mode=append so a requeued run appends to the existing output and error files instead of truncating them. Slurm’s default open mode is truncate, so without this flag a node failure erases your job’s logs from before the failure:
This flag governs the output and error files only. Any training progress made since the last checkpoint is lost to the failure itself, which is what the checkpoint interval controls.
--no-requeue disables all automatic recovery for a job, including recovery from a node failure. Don’t set it on a long-running training job unless you want a node failure to end the job permanently.

​
Detect a requeue and resume from checkpoint

Slurm sets SLURM_RESTART_COUNT in the batch script’s environment whenever it restarts a job, whether from a node failure or an explicit requeue. The variable is unset on the job’s first run. After a requeue, it holds the number of times the job has restarted. Check this variable at the start of your job script to decide whether to resume from a checkpoint or start fresh:
train.sbatch
Your training code then determines which checkpoint to load. Point it at a stable identifier, such as the job ID ($SLURM_JOB_ID) or a fixed “latest” path, rather than a value derived from wall-clock time. A checkpoint path built from the current date, for example, silently points at a new location after midnight. A job that restarts across a day boundary then can’t find the checkpoint it wrote before the restart. When more than one process in a job restores from the same checkpoint, have a single process, conventionally rank 0, read it and broadcast the result to the other ranks. Letting every rank read the same object at once collapses read throughput. Rank-0-read-then-broadcast is the standard pattern for distributed training at scale, not a workaround.

​
Choose a checkpoint interval

Balance checkpoint frequency against two costs: the time and I/O each checkpoint write costs the job, and the training progress you lose if a node fails between checkpoints. Time a single checkpoint write for your job, decide how much recomputation you can tolerate after a failure, and set the interval so the write cost stays a small fraction of the work it protects. Large multi-node jobs, such as those running on GB200 NVL72-powered instances, write bigger checkpoints, so the write cost weighs more heavily. Design long-running jobs to treat a node failure as a normal, recoverable event rather than an exception. On NVL72 racks specifically, if more than two nodes become unavailable, CoreWeave cordons and drains the entire rack. See Deploy NVL72-powered instances as full racks for that policy.

​
Store checkpoints in CoreWeave AI Object Storage

Object Storage, accessed through Local Object Transport Accelerator (LOTA), is the recommended location to store checkpoints. A requeued job can land on a different set of nodes than the original run, so checkpoints need to be reachable from every node in the job:
  • Object Storage is durable and reachable from any node in the cluster. This means a checkpoint written by one set of nodes is available to a different set of nodes after a requeue.
  • LOTA caches recently accessed objects on local node disks. Re-reading a large checkpoint after a restart doesn’t require a full round trip to the storage backend every time.
Avoid node-local storage, such as /tmp or NVMe scratch space, for checkpoints. A failed node’s local storage isn’t recoverable, and a requeued job isn’t guaranteed to return to the same node. A shared filesystem PVC also works, but sizing it to hold checkpoints for large models adds an operational cost that Object Storage avoids.

​
Example: sbatch script with checkpoint and resume

The following example combines the patterns from this guide into a complete template. Replace [BUCKET-NAME] with your Object Storage bucket name and [INTERVAL-MINUTES] with the checkpoint interval you chose, then adapt the train.py flags to match your own training script.
train.sbatch
This example assumes train.py accepts --checkpoint-uri, --checkpoint-interval-minutes, and --resume-from, and implements the matching save and load logic. Submit this script the same way you’d submit any other job. If a node failure interrupts the run, Slurm requeues it automatically. The restart check at the top of the script then resumes training from the last checkpoint in Object Storage instead of starting over.
Last modified on September 14, 2026