> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# October 23, 2026 - SUNK v8.2 release

> SUNK v8.2 gang-schedules Jobs and LeaderWorkerSets and moves to topology.yaml

## Overview

SUNK v8.2.0 was released on October 23, 2026. It ships with Slurm `25.11.8-coreweave.2` by default, up from `25.11.8-coreweave.1` in SUNK 8.1. See [SUNK to Slurm version mapping](https://docs.coreweave.com/products/sunk/reference/sunk-slurm-versions.md).

The main additions in 8.2 are gang scheduling for Kubernetes Jobs and LeaderWorkerSets through the SUNK Pod Scheduler, and a move from a generated `topology.conf` to a generated `topology.yaml`. The release also limits out-of-memory (OOM) kills in login and compute Pods to a single process, adds zsh to the login image, and changes several defaults.

Before you upgrade, review [Changes to check before you upgrade](#changes-to-check-before-you-upgrade).

## Upgrade to 8.2

**SUNK Standard**

Before you upgrade, check your `slurm` chart values for the settings in [Changes to check before you upgrade](#changes-to-check-before-you-upgrade).

Upgrade the `sunk` chart to 8.2.0 first, then the `slurm` chart. The 8.2 scheduler drops the `--slurm-kill-wait` flag that the 8.1 operator passes to it.

> [!WARNING]
>
> If you upgrade the `slurm` chart before the `sunk` chart, the new scheduler Pod restarts repeatedly until the operator is upgraded, while the old scheduler Pod keeps running.

## Changes to check before you upgrade

| Change | What to check |
| - | - |
| Per-user login Pods are off by default | In the `slurm` chart, `userPods` on an entry in `login.groups` now defaults to `false`, and `groupPods` still defaults to `true`. A group that doesn't set `userPods` gets only its shared group login Pod after the upgrade, and its members' per-user login Pods are removed. To keep per-user login Pods, set `userPods: true` on each group that needs them. |
| SUNK generates `topology.yaml` instead of `topology.conf` | The operator writes a `topology.yaml` with `tree`, `block`, and `flat` entries to the same `CLUSTER_NAME-topology-conf` ConfigMap and removes the `topology.conf` key. Slurm ignores `TopologyPlugin` when `topology.yaml` exists, so the operator uses `slurmConfig.TopologyPlugin` to pick the default topology. Set it to `topology/tree`, `topology/block`, or `topology/flat`. An unset value, `topology/none`, or `topology/default` selects `topology/flat`. Any other value fails reconciliation, and the `SlurmClusterAvailable` condition reports the error. |
| You supply your own topology file | If you set `compute.generateTopology: false` and supply your own `topology.conf`, convert it to [`topology.yaml`](https://slurm.schedmd.com/topology.yaml.html) and supply it through `controller.etcConfigMap`. If a ConfigMap in `controller.etcConfigMap` contains `topology.conf` or `topology.yaml`, the operator no longer generates a topology, even when `compute.generateTopology` is `true`. |
| Leaf switch names change | While SUNK generates the topology, each node now carries its own NVLink block and tree switches: new Pods declare them at registration through the `sunk.coreweave.com/topology` Pod annotation, and the syncer sets them on existing nodes. Leaf switch names change from `LG-<fabric>-<leaf>` to `LG-<fabric>-<superpod>-<leaf>`. Update any script or dashboard that matches the old names. `slurmctld` restarts once during the upgrade. |
| `SYS_RESOURCE` is added to the default capabilities | The `slurmd` container and the login `sshd` container now get `SYS_RESOURCE` by default, which the new OOM handling requires. If you set `capabilities` in `compute.securityContext` or `login.common.pod.securityContext`, your list replaces the defaults, so add `SYS_RESOURCE` to it. Login and compute Pods roll once. |
| `securityContext` overrides apply in full | `controller.securityContext` and `compute.securityContext`, and `munge.securityContext` on the `SlurmCluster` resource, now apply every field you set over the operator defaults. Before 8.2, only `privileged`, `runAsUser`, `runAsGroup`, `runAsNonRoot`, and `capabilities` applied, so a field that was set but ignored starts to apply when Pods roll. When Pyxis is enabled, `compute.securityContext.appArmorProfile` must match `compute.pyxis.appArmorProfile`, which defaults to `localhost/enroot`. |
| An empty `ssh` or `cacheDropper` section means enabled | An empty `compute.ssh` or `compute.cacheDropper` section, or a `compute.cacheDropper` section that sets only `resources`, now enables the feature. To turn either one off, set `enabled: false`. For `cacheDropper`, this also removes a sidecar that's already running. `compute.cacheDropper.resources` is now applied. See [Manage memory with cache-dropper](https://docs.coreweave.com/products/sunk/optimize_workloads/cache-dropper.md). |
| Empty `slurmConfig` values remove operator defaults | In `slurmConfig`, `cgroupConfig`, and `accounting.config`, a key set to `""` or `[]` now removes the operator's default from the rendered configuration. Before 8.2, the chart dropped these values, so the default applied. A `null` value still keeps the default. Check for keys left as `""` that you expect to keep their default. |
| Syncer and scheduler configuration is validated | `scheduler.config` and `syncer.config` now have a schema on the `SlurmCluster` resource. The API server rejects values of the wrong type, negative durations, `poolSize` and `maxConcurrentReconciles` values below 1, and an invalid `nodesetUpdateJobPreemption` `method` even when preemption is off. Unknown keys are dropped. Check these sections for values written as strings, such as `"false"`, before you upgrade. |
| Cluster affinity applies to more Pods | `affinity` on the `SlurmCluster` now applies to the scheduler and the JWT secret Job. The managed MySQL Pods now get the full `affinity` and `tolerations`. Before 8.2, they kept only the required node-selector terms, and MOCO Pods got no tolerations. If you set `tolerations` or use preferred or Pod affinity terms, the MySQL Pod can restart once. With a single replica, `slurmdbd` is unavailable until it returns. Make sure the MySQL Pod can satisfy your affinity rules. |
| GPU Straggler Detection defaults to 1.1.0 | `compute.gpusd.version` now defaults to `1.1.0`. With 1.1.0 or later, the `slurmd` container also exposes TCP port `10500`, in addition to ports `10400` through `10407`. To keep only the 1.0 ports, set `compute.gpusd.version: 1.0.0`. See [Straggler Detection](https://docs.coreweave.com/products/sunk/discover_sunk/straggler-detection.md). |
| The scheduler uses the cluster's `KillWait` | The SUNK Pod Scheduler now reads `KillWait` from the cluster's `slurm.conf` instead of always using 30 seconds. If you override `KillWait` in `slurmConfig`, check that each Pod's `terminationGracePeriodSeconds` still passes admission. A `KillWait` that isn't a whole number from `0` to `65535` stops the scheduler's reconciliation, and the running scheduler is left unchanged. |

## New in 8.2

* **Gang scheduling for Jobs and LeaderWorkerSets**: The SUNK Pod Scheduler schedules the Pods of a Kubernetes Job, or of each LeaderWorkerSet replica group, together as one Slurm job. As a result, a workload starts only when all of its Pods can run. No annotation is needed. The scheduler automatically cancels orphaned placeholder jobs, which `scheduler.config.scheduler.orphanJobSweep` controls. See [Run workloads with gang scheduling](https://docs.coreweave.com/products/sunk/run_workloads/gang-scheduling.md).
* **Login Pod topology spread**: Login Pods now spread across nodes by default, with `maxSkew: 1` on `kubernetes.io/hostname`. Set `login.topologySpreadConstraints` to replace the default, or set it to `[]` to turn it off. The change takes effect when a login Pod is recreated.
* **zsh on login Pods**: The login image now includes zsh. SUNK's aliases, `ulimit` and `umask` settings, and the `drain` and `undrain` helpers load in both bash and zsh. To make zsh a user's shell, set `nsscache.defaultShell: /bin/zsh`, or set the user's `sunkLoginShell` attribute (SCIM) or `loginShell` attribute (LDAP). Slurm command completion remains bash-only. See [Set the default shell](https://docs.coreweave.com/products/sunk/deploy_sunk/configure-slurm-individual-login.md#set-the-default-shell).
* **OOM kill containment**: When a login or `slurmd` container runs out of memory, the kernel now kills a single process instead of the whole container, and SSH sessions are killed first. On compute nodes, jobs are killed before `slurmd`, so the container doesn't restart. Compute containment requires cgroup v2.
* **NodeSet conditions and rendered spec**: NodeSets report a `PodSpecValid` condition when the API server rejects a compute Pod, and a `ComputeConfigValid` condition when the `SlurmCluster` compute settings are invalid. The operator no longer patches NodeSet `spec`. It writes the effective Pod template to the read-only `sunk.coreweave.com/rendered` annotation instead, so GitOps tools stop reporting drift on NodeSets. Manual edits to operator-derived fields in `spec`, such as the `slurmd` image, are now ignored instead of reverted.
* **Clearer disk drain reasons**: When the health check drains a node because `/` or `/tmp` is full, the drain reason starts with `NHC: check_cw_fs_used:` and lists the space that `/tmp`, `/coredumps`, and `/opt/sunk` use, largest first. The thresholds are unchanged.
* **Inode lock fixer on by default**: New `SlurmCluster` resources enable the inode lock fixer for the managed database (`database.mysqlCluster.inodeLockFixer` in the `slurm` chart). Existing clusters keep their current setting.

## Fixes

* A running job submitted with more than one QoS keeps the QoS it started with when `slurmctld` restarts or reconfigures. Before 8.2, it could switch to the highest-priority QoS.
* SUNK now sets `RealMemory` from the `slurmd` container's memory limit, minus reserved memory, instead of the node type's hardware memory. Full-node jobs on NodeSets whose memory limit is lower than the node type's memory now schedule.
* SUNK can now set a node stuck in `COMPLETING` to `DOWN`. Before 8.2, its cleanup controller failed with `slurm_update error: Invalid user id`.
* The `reconfigure-watch`, `sync-users-from-system`, and `nsscache-update` sidecars exit cleanly on shutdown. Before 8.2, they were killed at the end of the grace period, which delayed `slurmctld` teardown.
* When `compute.ssh.enabled` is `false`, the operator deletes the compute `sshd` s6 ConfigMaps.
* Argo CD no longer reports a permanent diff on the `sunk` controller manager Deployment.