> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade a SUNK cluster

> The version fields that control a self-service SUNK cluster and what to expect when you upgrade the cluster

A self-service SUNK cluster runs a specific set of software versions, declared on the `SunkCluster` resource. Upgrading a cluster means moving those versions forward. This page describes the version fields that determine what a cluster runs and what to expect when you upgrade a cluster, including how the rollout treats running jobs.

<Note>
  The supported workflow for starting an upgrade and for rolling back is being finalized. In the meantime, coordinate upgrade timing with your CoreWeave Solutions Architect, especially for production clusters with running jobs. This page will be updated with the step-by-step procedure.
</Note>

## Version fields

Four fields on `spec` determine the versions a cluster runs:

| Field           | Controls                                     | Notes                                                                                                                                                                       |
| --------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sunkVersion`   | The SUNK release version.                    | Optional. Defaults to `v8.0.0`. Must be paired with a compatible `slurmVersion`. See [SUNK to Slurm version mapping](/products/sunk/reference/sunk-slurm-versions).         |
| `slurmVersion`  | The Slurm version.                           | Optional. Defaults to `v25.11.6-coreweave.2`, the Slurm build shipped with the default SUNK release. Must match the version paired with `sunkVersion` in the mapping table. |
| `cudaVersion`   | The CUDA version installed on compute nodes. | Optional. Defaults to `13.0`. Allowed values: `13.0`, `13.1`.                                                                                                               |
| `ubuntuVersion` | The Ubuntu base image version for nodes.     | Optional. Defaults to `24.04`. Allowed values: `22.04`, `24.04`.                                                                                                            |

`sunkVersion` and `slurmVersion` are a matched pair, so you must change them together using a row from the [version mapping table](/products/sunk/reference/sunk-slurm-versions). For the full field definitions, see the [`SunkCluster` reference](/products/sunk/reference/sunkcluster-reference#top-level-spec-fields).

## What changes during an upgrade

The `slurmVersion`, `ubuntuVersion`, and `cudaVersion` fields determine the compute node image, and `sunkVersion` sets the version of the SUNK operator that manages the cluster. When a cluster moves to a new version, the operator regenerates the underlying cluster configuration and the compute `NodeSets` roll to the new image. Because `NodeSets` use a rolling update strategy, nodes are replaced gradually rather than all at once.

Rolling a compute node does not interrupt the Slurm jobs already running on it. `NodeSets` roll using [safe deletion](/products/sunk/discover_sunk/nodeset#safe-deletion): the Syncer drains the Slurm node first, and the Pod is only replaced after the node is both drained and idle. The rollout replaces not-ready and idle Pods first, then waits for the remaining Pods to finish their work. As a result, a long-running job can hold its node at the old version until the job completes.

SUNK can requeue the jobs on a draining node so they restart elsewhere, which keeps an upgrade inside a bounded window instead of waiting on long jobs. This matters because a draining node accepts no new jobs, so the cluster's schedulable capacity shrinks while the rollout progresses. You can scope requeueing by time, partition, or Quality of Service. See [Automatic requeueing during rolling upgrades](/changelog/release-notes/sunk-v6-9-0#automatic-requeueing-during-rolling-upgrades) for how it works.

Upgrading the Slurm control plane briefly restarts the control plane workloads. Submitted and queued jobs are preserved in `slurmctld`'s saved state, which lives on a dedicated persistent volume, so they survive the restart, but the cluster is momentarily unavailable for new scheduling decisions while the controller comes back up.

To track an upgrade in progress, watch the `SunkCluster` status conditions the same way you do during creation. The aggregate `Ready` condition returns to `True` once the rolled `NodeSets` and the `SlurmCluster` report ready again. See the [`SunkCluster` reference](/products/sunk/reference/sunkcluster-reference#status-conditions) for the conditions to watch.

## Related

For more information, see the following pages:

* [SUNK to Slurm version mapping](/products/sunk/reference/sunk-slurm-versions) for supported `sunkVersion` and `slurmVersion` combinations.
* [`SunkCluster` reference](/products/sunk/reference/sunkcluster-reference#top-level-spec-fields) for the version field definitions.
* [NodeSet](/products/sunk/discover_sunk/nodeset#rolling-upgrade) for how a rolling upgrade orders and drains Pods.
* [Automatic requeueing during rolling upgrades](/changelog/release-notes/sunk-v6-9-0#automatic-requeueing-during-rolling-upgrades) for bounding how long a rollout waits on running jobs.