Skip to main content

Overview

SUNK v8.0.0 is a re-architecture, not a routine version bump. This page describes the configuration and behavior changes in 8.0 so that you can update your Helm values before you upgrade from a 7.x release. In 7.x, the Helm chart was the cluster. The chart shipped the defaults and rendered slurmctld, slurmdbd, the login pods, and everything else directly. In 8.0, the slurm chart renders a single SlurmCluster resource, and the SUNK operator builds the cluster from it. You still configure the cluster through Helm values, and most of your keys survive with the same names and the same override semantics. What changed is where a value comes from when you don’t set it. The operator supplies it now, instead of the chart’s values.yaml. This release also removes SSSD directory integration, consolidates four database mechanisms into one, replaces shared login replicas with per-user and per-group login pods, and deletes the slurm-login chart.

Action required

Upgrading from any 7.x release isn’t a standard helm upgrade. The chart no longer templates the stateful objects that hold your cluster state and accounting data, so Helm treats them as removed and deletes them, taking the accounting database and cluster state with them.Do not run helm upgrade --reuse-values. It carries your 7.x values forward without the new chart defaults, which can silently select the wrong database backend.Your 7.x cluster must already be on the MOCO database backend before you upgrade. SUNK 8.0 doesn’t include the Bitnami MySQL to MOCO migration, so run that migration while you’re still on 7.x.This page covers the configuration changes in 8.0. It isn’t a complete upgrade guide, and your cluster may have upgrade considerations beyond the ones listed here. Contact CoreWeave Support to plan your upgrade before you begin.
The following sections describe the Slurm version upgrade, the defaults that change if you do nothing, the identity, accounting, and login re-architectures, and the full list of new, renamed, and removed values.

Upgrade to Slurm 25.11

SUNK v8.0.0 upgrades Slurm from 25.05 to 25.11, a major version bump that brings upstream bug fixes and performance improvements. The Slurm protocol version default moves from 25.05 to 25.11 alongside it. If you need to stage the protocol change separately from the version upgrade, pin the protocol version explicitly with slurmCluster.spec.protocolVersion, which accepts 25_05 or 25_11.

Defaults that change if you do nothing

SUNK 8.0 drops a set of chart defaults and replaces them with operator defaults that differ. None of these produce an error. You get different behavior, and which ones reach you depends on your setup.
The cgroup v1 plugin is deprecated and won’t be supported in future Slurm versions, so SUNK now defaults to cgroup/v2. The cgroup/v1 plugin is still supported, subject to upstream Slurm support and support in the broader ecosystem, including Linux distributions, the kernel, and systemd.

Identity moves from SSSD to nsscache

SUNK 7.x supported SSSD, configured across directoryService, an sssdContainer sidecar, and a userLookupContainer. All of it is removed, along with directoryCache from the deleted slurm-login chart. Identity comes from nsscache in 8.0. The operator runs it, refreshes it on an interval, and publishes the results into per-map Secrets that login and compute pods consume. Two sources are available:
  • SCIM, the default. Set nsscache.existingSecret to a Secret holding the nsscache-scim-auth-token key, and nsscache.groups to select which groups SCIM fetches.
  • LDAP. Set source: ldap under nsscache.nsscacheConfig along with your directory settings.
nsscache.defaultShell and nsscache.overrideHomeDirectory set the login shell and home path for SCIM-sourced accounts. The home path defaults to /mnt/home/%u, so check that against where your home directories live. Two smaller changes come with it:
  • The operator generates nsswitch.conf, and nsscache.nsswitchConfig is removed.
  • nsscache.cronJobSchedule becomes nsscache.fetchInterval, an integer number of seconds defaulting to 60, with an operator-managed Deployment in place of the CronJob.
nsscache.enabled, nsscache.existingSecret, nsscache.sudoGroups, and nsscache.slurmUserProvisioning all carry over unchanged. nsscache.nsscacheConfig now requires every inner value to be a string, so quote numbers and booleans, and convert 7.x lists into the comma-separated strings that nsscache.conf expects.

Accounting consolidates on one database backend

SUNK 7.x gave you four overlapping ways to attach a database: the Bitnami MySQL subchart, MOCO, accounting.externalDB, and accounting.external. SUNK 8.0 replaces all four with one selector, accounting.database.backend, and one set of connection fields. Write backend out explicitly even when you want the default, because both the chart template and the CRD fall back to cwdb when the value is empty. Everything else about slurmdbd keeps its meaning. accounting.config still configures slurmdbd.conf, and replicas, image, resources, security context, and probes all remain overridable. Only their defaults moved into the operator. Your MOCO connection and authentication settings move to accounting.database, and MOCO infrastructure tuning moves to database.mysqlCluster. Accounting also changes in the following ways:
  • The Bitnami to MOCO migration Job (moco.migration) exists only in 7.x. Run it before you upgrade. SUNK 8.0 ships only a manual moco-migrator.sh script in the custom scripts ConfigMap.
  • External slurmdbd (accounting.external) has no 8.0 equivalent. Only an external database is supported.

Login capacity is provisioned per user and per group

SUNK 7.x gave you two kinds of login pods, under two names, built by two different charts. Shared login pods came from the slurm chart as one StatefulSet of login.replicas interchangeable pods behind <release>-login. Individual login pods came from the slurm-login subchart, off by default, one StatefulSet per user, sized by login.individualResources. SUNK 8.0 keeps both ideas, renames them, and moves both into the operator. Shared becomes group pods, and individual becomes user pods. They’re now configured identically: login.common sets the base, login.userPods and login.groupPods override per kind, and each entry in login.groups selects which kinds that group gets, with membership resolved from nsscache. Resource names change. Per-user StatefulSets and Services are <cluster>-login-<username>-<hash>, and per-group ones are <cluster>-login-<group>. The headless <cluster>-login-all-users Service keeps its 7.x name and shape. The old <release>-login StatefulSet, its Service, and the per-replica <release>-login-<n> Services have no 8.0 counterpart.
Check the login defaults before you list groups. A group entry that doesn’t say otherwise gets both per-user and per-group pods, because userPods and groupPods both default to true. Every login entity gets its own Service defaulting to LoadBalancer, and each pod requests 4 CPU and 8Gi.That last default applies to per-user pods too. In 7.x those ran on the much smaller individualResources of 500m and 300Mi, so the per-user CPU request grows roughly eight times and the memory request roughly 27 times unless you override login.userPods.pod.resources. A group of N users on defaults means N+1 StatefulSets and N+1 external load balancers.
Service annotations and labels support %s and %h substitution for the entity name, and login.common.service.individual overrides them per user or group. login.s6 keeps its name but gains validation. Script-based oneshots need timeoutUp, longruns need a timeout, and each service must set exactly one of script or packages. The same rules apply to compute.s6.

Slurm configuration defaults

Slurm configuration changes little in 8.0. The following table lists the keys whose defaults differ:

RPC rate limiting is on by default

slurmctld now throttles RPCs, so review any tooling that issues them at volume. For more information about tuning the limits, see the Slurm configuration documentation.

Other behavior changes

The following changes affect areas not covered in the preceding sections.
  • Hooks API configuration moved to the syncer. Both syncer.hooksAPI and scheduler.hooksAPI.waitForPodDeletionInterval become syncer.config.hooksAPI. If your 7.x syncer and scheduler settings disagreed, they now converge on a single value, so pick the one you want.
  • Low-profile compute requests default on. The new compute.lowProfileRequests overlays low CPU and memory requests on slurmd containers so Kubernetes can co-schedule alongside them, and it’s forced on when the scheduler is enabled. This changes how compute nodes pack.
  • Cleanup runs every 300 seconds instead of every 60. cleanupCompleting.timeoutSeconds is now a flat 2400 rather than twice KillWait, so the timeout no longer tracks your KillWait.
  • The slurmrestd workload and Service are renamed from <release>-rest to <release>-restd. Repoint every client, dashboard, and script that talks to the old name.
  • Scheduler scrape configuration can no longer be turned off. The operator creates it whenever the VMPodScrape CRD is installed, and the sunk chart’s scheduler section is gone. Operator-created VMPodScrape objects replace the chart-rendered monitoring templates.
  • The SUNK admission webhooks and their cert-manager serving certificate are gone. CRD validation rules and declarative defaults replace them, at parity with what the 7.x webhook covered for NodeSets.
  • The operator provisions the JWT and cluster secrets. secretJob is removed. jwt.existingSecret still works unchanged.
Workload sizing barely moved. slurmctld, slurmdbd, the syncer, slurmrestd, munge, group login pods, cleanup-completing, and the database all keep their 7.x requests and limits, including the 32Gi slurmctld state volume. nsscache rises to 500m and 200Mi requested with a 1Gi limit, up from 200m and 100Mi with a 500Mi limit. Per-user login pods are the exception, covered in Login capacity is provisioned per user and per group.

New in 8.0

These values have no 7.x equivalent.

Renamed and moved values

Set the new key and delete the old one.

Values that changed shape

Same key, different type. Check these manually, because a wrong type here doesn’t always produce a visible error.

Removed values

Delete these. Where something replaces the behavior, it’s named.
Last modified on August 27, 2026