October 23, 2026 - SUNK v8.2 release

Overview

SUNK v8.2.0 was released on October 23, 2026. It ships with Slurm 25.11.8-coreweave.2 by default, up from 25.11.8-coreweave.1 in SUNK 8.1. See SUNK to Slurm version mapping. The main additions in 8.2 are gang scheduling for Kubernetes Jobs and LeaderWorkerSets through the SUNK Pod Scheduler, and a move from a generated topology.conf to a generated topology.yaml. The release also limits out-of-memory (OOM) kills in login and compute Pods to a single process, adds zsh to the login image, and changes several defaults. Before you upgrade, review Changes to check before you upgrade.

Upgrade to 8.2

SUNK Standard

Before you upgrade, check your slurm chart values for the settings in Changes to check before you upgrade.Upgrade the sunk chart to 8.2.0 first, then the slurm chart. The 8.2 scheduler drops the --slurm-kill-wait flag that the 8.1 operator passes to it.

If you upgrade the slurm chart before the sunk chart, the new scheduler Pod restarts repeatedly until the operator is upgraded, while the old scheduler Pod keeps running.

Changes to check before you upgrade

Change	What to check
Per-user login Pods are off by default	In the `slurm` chart, `userPods` on an entry in `login.groups` now defaults to `false`, and `groupPods` still defaults to `true`. A group that doesn’t set `userPods` gets only its shared group login Pod after the upgrade, and its members’ per-user login Pods are removed. To keep per-user login Pods, set `userPods: true` on each group that needs them.
SUNK generates `topology.yaml` instead of `topology.conf`	The operator writes a `topology.yaml` with `tree`, `block`, and `flat` entries to the same `CLUSTER_NAME-topology-conf` ConfigMap and removes the `topology.conf` key. Slurm ignores `TopologyPlugin` when `topology.yaml` exists, so the operator uses `slurmConfig.TopologyPlugin` to pick the default topology. Set it to `topology/tree`, `topology/block`, or `topology/flat`. An unset value, `topology/none`, or `topology/default` selects `topology/flat`. Any other value fails reconciliation, and the `SlurmClusterAvailable` condition reports the error.
You supply your own topology file	If you set `compute.generateTopology: false` and supply your own `topology.conf`, convert it to `topology.yaml` and supply it through `controller.etcConfigMap`. If a ConfigMap in `controller.etcConfigMap` contains `topology.conf` or `topology.yaml`, the operator no longer generates a topology, even when `compute.generateTopology` is `true`.
Leaf switch names change	While SUNK generates the topology, each node now carries its own NVLink block and tree switches: new Pods declare them at registration through the `sunk.coreweave.com/topology` Pod annotation, and the syncer sets them on existing nodes. Leaf switch names change from `LG-<fabric>-<leaf>` to `LG-<fabric>-<superpod>-<leaf>`. Update any script or dashboard that matches the old names. `slurmctld` restarts once during the upgrade.
`SYS_RESOURCE` is added to the default capabilities	The `slurmd` container and the login `sshd` container now get `SYS_RESOURCE` by default, which the new OOM handling requires. If you set `capabilities` in `compute.securityContext` or `login.common.pod.securityContext`, your list replaces the defaults, so add `SYS_RESOURCE` to it. Login and compute Pods roll once.
`securityContext` overrides apply in full	`controller.securityContext` and `compute.securityContext`, and `munge.securityContext` on the `SlurmCluster` resource, now apply every field you set over the operator defaults. Before 8.2, only `privileged`, `runAsUser`, `runAsGroup`, `runAsNonRoot`, and `capabilities` applied, so a field that was set but ignored starts to apply when Pods roll. When Pyxis is enabled, `compute.securityContext.appArmorProfile` must match `compute.pyxis.appArmorProfile`, which defaults to `localhost/enroot`.
An empty `ssh` or `cacheDropper` section means enabled	An empty `compute.ssh` or `compute.cacheDropper` section, or a `compute.cacheDropper` section that sets only `resources`, now enables the feature. To turn either one off, set `enabled: false`. For `cacheDropper`, this also removes a sidecar that’s already running. `compute.cacheDropper.resources` is now applied. See Manage memory with cache-dropper.
Empty `slurmConfig` values remove operator defaults	In `slurmConfig`, `cgroupConfig`, and `accounting.config`, a key set to `""` or `[]` now removes the operator’s default from the rendered configuration. Before 8.2, the chart dropped these values, so the default applied. A `null` value still keeps the default. Check for keys left as `""` that you expect to keep their default.
Syncer and scheduler configuration is validated	`scheduler.config` and `syncer.config` now have a schema on the `SlurmCluster` resource. The API server rejects values of the wrong type, negative durations, `poolSize` and `maxConcurrentReconciles` values below 1, and an invalid `nodesetUpdateJobPreemption` `method` even when preemption is off. Unknown keys are dropped. Check these sections for values written as strings, such as `"false"`, before you upgrade.
Cluster affinity applies to more Pods	`affinity` on the `SlurmCluster` now applies to the scheduler and the JWT secret Job. The managed MySQL Pods now get the full `affinity` and `tolerations`. Before 8.2, they kept only the required node-selector terms, and MOCO Pods got no tolerations. If you set `tolerations` or use preferred or Pod affinity terms, the MySQL Pod can restart once. With a single replica, `slurmdbd` is unavailable until it returns. Make sure the MySQL Pod can satisfy your affinity rules.
GPU Straggler Detection defaults to 1.1.0	`compute.gpusd.version` now defaults to `1.1.0`. With 1.1.0 or later, the `slurmd` container also exposes TCP port `10500`, in addition to ports `10400` through `10407`. To keep only the 1.0 ports, set `compute.gpusd.version: 1.0.0`. See Straggler Detection.
The scheduler uses the cluster’s `KillWait`	The SUNK Pod Scheduler now reads `KillWait` from the cluster’s `slurm.conf` instead of always using 30 seconds. If you override `KillWait` in `slurmConfig`, check that each Pod’s `terminationGracePeriodSeconds` still passes admission. A `KillWait` that isn’t a whole number from `0` to `65535` stops the scheduler’s reconciliation, and the running scheduler is left unchanged.

New in 8.2

Gang scheduling for Jobs and LeaderWorkerSets: The SUNK Pod Scheduler schedules the Pods of a Kubernetes Job, or of each LeaderWorkerSet replica group, together as one Slurm job. As a result, a workload starts only when all of its Pods can run. No annotation is needed. The scheduler automatically cancels orphaned placeholder jobs, which scheduler.config.scheduler.orphanJobSweep controls. See Run workloads with gang scheduling.
Login Pod topology spread: Login Pods now spread across nodes by default, with maxSkew: 1 on kubernetes.io/hostname. Set login.topologySpreadConstraints to replace the default, or set it to [] to turn it off. The change takes effect when a login Pod is recreated.
zsh on login Pods: The login image now includes zsh. SUNK’s aliases, ulimit and umask settings, and the drain and undrain helpers load in both bash and zsh. To make zsh a user’s shell, set nsscache.defaultShell: /bin/zsh, or set the user’s sunkLoginShell attribute (SCIM) or loginShell attribute (LDAP). Slurm command completion remains bash-only. See Set the default shell.
OOM kill containment: When a login or slurmd container runs out of memory, the kernel now kills a single process instead of the whole container, and SSH sessions are killed first. On compute nodes, jobs are killed before slurmd, so the container doesn’t restart. Compute containment requires cgroup v2.
NodeSet conditions and rendered spec: NodeSets report a PodSpecValid condition when the API server rejects a compute Pod, and a ComputeConfigValid condition when the SlurmCluster compute settings are invalid. The operator no longer patches NodeSet spec. It writes the effective Pod template to the read-only sunk.coreweave.com/rendered annotation instead, so GitOps tools stop reporting drift on NodeSets. Manual edits to operator-derived fields in spec, such as the slurmd image, are now ignored instead of reverted.
Clearer disk drain reasons: When the health check drains a node because / or /tmp is full, the drain reason starts with NHC: check_cw_fs_used: and lists the space that /tmp, /coredumps, and /opt/sunk use, largest first. The thresholds are unchanged.
Inode lock fixer on by default: New SlurmCluster resources enable the inode lock fixer for the managed database (database.mysqlCluster.inodeLockFixer in the slurm chart). Existing clusters keep their current setting.

Fixes

A running job submitted with more than one QoS keeps the QoS it started with when slurmctld restarts or reconfigures. Before 8.2, it could switch to the highest-priority QoS.
SUNK now sets RealMemory from the slurmd container’s memory limit, minus reserved memory, instead of the node type’s hardware memory. Full-node jobs on NodeSets whose memory limit is lower than the node type’s memory now schedule.
SUNK can now set a node stuck in COMPLETING to DOWN. Before 8.2, its cleanup controller failed with slurm_update error: Invalid user id.
The reconfigure-watch, sync-users-from-system, and nsscache-update sidecars exit cleanly on shutdown. Before 8.2, they were killed at the end of the grace period, which delayed slurmctld teardown.
When compute.ssh.enabled is false, the operator deletes the compute sshd s6 ConfigMaps.
Argo CD no longer reports a permanent diff on the sunk controller manager Deployment.

Infrastructure

Forge

October 23, 2026 - SUNK v8.2 release

Overview

Upgrade to 8.2

Changes to check before you upgrade

New in 8.2

Fixes

​Overview

​Upgrade to 8.2

​Changes to check before you upgrade

​New in 8.2

​Fixes

Overview

Upgrade to 8.2

Changes to check before you upgrade

New in 8.2

Fixes