SunkCluster resource. Upgrading a cluster means moving those versions forward. This page describes the version fields that determine what a cluster runs and what to expect when you upgrade a cluster, including how the rollout treats running jobs.
The supported workflow for starting an upgrade and for rolling back is being finalized. In the meantime, coordinate upgrade timing with your CoreWeave Solutions Architect, especially for production clusters with running jobs. This page will be updated with the step-by-step procedure.
Version fields
Four fields onspec determine the versions a cluster runs:
sunkVersion and slurmVersion are a matched pair, so you must change them together using a row from the version mapping table. For the full field definitions, see the SunkCluster reference.
What changes during an upgrade
TheslurmVersion, ubuntuVersion, and cudaVersion fields determine the compute node image, and sunkVersion sets the version of the SUNK operator that manages the cluster. When a cluster moves to a new version, the operator regenerates the underlying cluster configuration and the compute NodeSets roll to the new image. Because NodeSets use a rolling update strategy, nodes are replaced gradually rather than all at once.
Rolling a compute node does not interrupt the Slurm jobs already running on it. NodeSets roll using safe deletion: the Syncer drains the Slurm node first, and the Pod is only replaced after the node is both drained and idle. The rollout replaces not-ready and idle Pods first, then waits for the remaining Pods to finish their work. As a result, a long-running job can hold its node at the old version until the job completes.
SUNK can requeue the jobs on a draining node so they restart elsewhere, which keeps an upgrade inside a bounded window instead of waiting on long jobs. This matters because a draining node accepts no new jobs, so the cluster’s schedulable capacity shrinks while the rollout progresses. You can scope requeueing by time, partition, or Quality of Service. See Automatic requeueing during rolling upgrades for how it works.
Upgrading the Slurm control plane briefly restarts the control plane workloads. Submitted and queued jobs are preserved in slurmctld’s saved state, which lives on a dedicated persistent volume, so they survive the restart, but the cluster is momentarily unavailable for new scheduling decisions while the controller comes back up.
To track an upgrade in progress, watch the SunkCluster status conditions the same way you do during creation. The aggregate Ready condition returns to True once the rolled NodeSets and the SlurmCluster report ready again. See the SunkCluster reference for the conditions to watch.
Related
For more information, see the following pages:- SUNK to Slurm version mapping for supported
sunkVersionandslurmVersioncombinations. SunkClusterreference for the version field definitions.- NodeSet for how a rolling upgrade orders and drains Pods.
- Automatic requeueing during rolling upgrades for bounding how long a rollout waits on running jobs.