Cluster lifecycle and reconciliation

A self-service SUNK cluster is reconciled, not scripted. You declare the cluster you want in a SunkCluster resource, and the operators continuously work to make the running cluster match that declaration. This page explains the reconciliation model, the order in which a cluster comes up, and how to read the status conditions that report progress.

The reconciliation model

Reconciliation is the continuous loop that drives a cluster’s actual state toward the desired state you declare. The SunkCluster resource is the desired state. Two operators reconcile it: the out-of-band operator turns your SunkCluster into managed NodePools and a generated SlurmCluster resource, and the in-cluster operator turns that SlurmCluster into the running Slurm control plane. Because the loop is continuous, reconciliation governs the whole lifecycle, not just creation. When you change a mutable field, such as scaling a node count, the operators reconcile the difference. When a component fails, the operators work to restore it. The status conditions described later report the result of each loop.

Creation ordering

A cluster comes up in dependency order: each stage depends on the one before it. The first stage is a prerequisite you provide; the rest are reconciled for you after you apply the SunkCluster.

Stage	What it produces	Provided by
CKS cluster	The Kubernetes cluster that hosts SUNK, with the SUNK operator installed.	You, as a prerequisite
`SunkCluster` resource	The declared cluster specification, applied to a namespace of your choice.	You (Console or CR)
`NodePools`	The managed pools that provision the physical Nodes for compute and the control plane.	Out-of-band operator
`NodeSets`	The Slurm node definitions backed by the Nodes from the `NodePools`.	Reconciliation
`SlurmCluster`	The Slurm control plane: `slurmctld`, accounting, the syncer, the scheduler, and the login pods your users connect to over SSH.	In-cluster operator

Until a stage’s dependency is satisfied, the stage can’t complete. For example, NodeSets can’t report ready Pods until their NodePools have provisioned Nodes, and the SlurmCluster can’t bring up slurmctld until the control-plane Nodes exist. This dependency chain is why the status conditions described next become ready in roughly the same order.

Node provisioning is usually the longest stage. A new cluster typically takes around 40 minutes to become ready, and most of that time is the NodePools reaching their target Node count. Once Nodes are available, the Slurm control plane and login pods come up quickly.

Track progress with status conditions

The SunkCluster reports its progress through status.conditions. Inspect them at any time. Replace [NAMESPACE] with the namespace you applied the SunkCluster to:

kubectl get sunkcluster -n [NAMESPACE] -o yaml

The aggregate Ready condition is the single signal for whether the cluster is fully reconciled. It’s True only when every dependent condition is True. While the cluster is coming up for the first time, Ready reports reason: Bootstrapping and a message that lists the dependent conditions still pending, such as Waiting for conditions: NodePoolsAvailable, NodeSetsAvailable, SlurmClusterAvailable. Each dependent condition tracks one stage of the creation ordering, so the condition that is not yet Ready tells you where the cluster is:

NodePoolsAvailable covers the NodePools stage.
NodeSetsAvailable covers the NodeSets stage.
SlurmClusterAvailable aggregates the SlurmCluster, including slurmctld, accounting, login pods, the syncer, and the scheduler.

Each condition carries a reason, one of Ready (satisfied), InProgress (working), Bootstrapping (set on the aggregate Ready during first creation), or Error (failed). The message adds detail and usually names the specific resources still pending, such as NodePools not at target: [a192, gb200] or NodeSets not ready: [a192 (0/36 ready)]. When SlurmClusterAvailable is not True, inspect the underlying SlurmCluster to find which subcomponent is pending or failing. Use the same namespace as the SunkCluster:

kubectl get slurmcluster -n [NAMESPACE] -o yaml

For the complete set of conditions, including the SlurmCluster subcomponent conditions and the lastReadyTime field, see the SunkCluster reference.

Bootstrapping compared to a regression

The status distinguishes a cluster that has never been ready from one that was ready and then regressed. The lastReadyTime field records the most recent time Ready became True:

If lastReadyTime is unset and Ready reports reason: Bootstrapping, the cluster is still coming up for the first time.
If lastReadyTime is set but Ready is now False, the cluster was previously healthy and a component has regressed. The dependent condition in reason: Error or reason: InProgress indicates which one.

Ongoing reconciliation

After a cluster is ready, the operators keep reconciling it against the SunkCluster resource. Editing a mutable field, such as a node count, prompts the operators to reconcile the change: scaling a count up or down adjusts the underlying NodePool and NodeSet. Immutable fields, such as a node entry’s name and instanceType, are fixed after creation. For which fields are mutable and the effect of changing version fields, see the SunkCluster reference. If a cluster is stuck with Ready: False, the most common cause is insufficient quota holding NodePoolsAvailable in reason: InProgress. See Troubleshoot for common failure modes and fixes.

​The reconciliation model

​Creation ordering

​Track progress with status conditions

​Bootstrapping compared to a regression

​Ongoing reconciliation

The reconciliation model

Creation ordering

Track progress with status conditions

Bootstrapping compared to a regression

Ongoing reconciliation