Bind NodeSets to Node Pools

A SUNK NodeSet places one slurmd Pod on each Kubernetes Node it selects. The NodeSet’s affinity, tolerations, and resource requests determine which Nodes it can select. A CoreWeave Kubernetes Service (CKS) Node Pool controls the other side of that decision. It applies the labels and taints in its spec to every Node it delivers. Those two sides have to agree. If a Node Pool taints its Nodes and the NodeSet doesn’t tolerate the taint, the NodeSet treats those Nodes as infeasible and the Node Pool sits idle. If the NodeSet has no affinity at all, its Pods spread across whatever production Nodes they fit on, including Nodes you meant to reserve for something else. This guide covers both approaches to binding the two together, how to make a NodeSet track Node Pool scale automatically, and how to verify the result. By the end, the NodeSet’s slurmd Pods run only on Nodes from the Node Pool you picked for them.

In SUNK, Slurm nodes run in Kubernetes Pods. These aren’t the same as Kubernetes Nodes, which are the worker machines that run the Pods. To distinguish between the two, this documentation capitalizes Kubernetes Nodes, while Slurm nodes aren’t capitalized.

How NodeSet scheduling uses Node Pool metadata

A NodeSet evaluates every Kubernetes Node in the cluster and counts the ones it could place a Pod on as feasible. Three things narrow that count:

Affinity. The NodeSet reads only affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution. It ignores preferred affinity and every other affinity section, so a preference has no effect on feasibility.
Tolerations. A tainted Node is infeasible unless the NodeSet tolerates each of its taints.
Resource requests. A Node with too little allocatable CPU, memory, or accelerator capacity is infeasible. For that reason, a stray non-Slurm Pod on a Node can make that Node infeasible for the NodeSet.

Two fields on the Node Pool spec supply the metadata to match against. nodeLabels applies labels to every Node in the pool, and nodeTaints applies taints. CKS applies both when it delivers a Node, so Nodes added by a later scale-up carry them without any manual step. For the full Node Pool spec, see Create a Node Pool. You can bind a NodeSet to a Node Pool in two ways:

Target a Node Pool by name uses a label CKS maintains on every Node. This approach requires no changes to the Node Pool, and it’s the right choice when the correct hardware is the only requirement.
Dedicate a Node Pool to a NodeSet adds a custom label and a matching taint to the Node Pool spec. Choose this when the Nodes must also repel Pods that aren’t part of the NodeSet.

Prerequisites

This guide applies to a SUNK cluster deployed from the Slurm Helm chart, where you own the compute.nodes values. On a managed self-service cluster created from a SunkCluster resource, the operator creates the Node Pools and NodeSets and keeps their counts in sync, so you don’t bind them yourself. Before you begin, make sure you have the following:

A CKS cluster with at least one Node Pool.
A SUNK cluster deployed from the Slurm Helm chart, and write access to its values.
kubectl configured against the cluster, with an API Access Token. Editing or patching a Node Pool needs the CKS Admin role. A token with only CKS Viewer can read Node Pools but can’t change them. See IAM access policies.

Target a Node Pool by name

CKS labels every Node with the name of the Node Pool that delivered it, using the key compute.coreweave.com/node-pool. Matching that label in a NodeSet’s affinity binds the NodeSet to the pool without editing the Node Pool spec. Replace [NODE-POOL-NAME] with the name of your Node Pool:

Bind a NodeSet to one Node Pool

compute:
  nodes:
    cpu-batch:
      enabled: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: compute.coreweave.com/node-pool
                    operator: In
                    values:
                      - [NODE-POOL-NAME]

You can match this label even though CKS rejects user-provided labels in the coreweave.com and coreweave.cloud namespaces. That restriction governs labels you create. CKS sets this one itself, and you can read it in a selector. When capacity for one instance type is split across several Node Pools, list every pool the NodeSet should use. A NodeSet that names only one pool leaves the other pool’s Nodes infeasible, which looks like missing capacity. Replace [NODE-POOL-NAME-A] and [NODE-POOL-NAME-B] with your Node Pool names:

Bind a NodeSet to several Node Pools

compute:
  nodes:
    cpu-batch:
      enabled: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: compute.coreweave.com/node-pool
                    operator: In
                    values:
                      - [NODE-POOL-NAME-A]
                      - [NODE-POOL-NAME-B]

This approach steers the NodeSet onto the right Nodes, but it doesn’t reserve them. Any other Pod that fits and tolerates the Node’s taints can still schedule there. To reserve the Nodes, add a taint as described in the following section.

Dedicate a Node Pool to a NodeSet

A custom label and a matching taint on the Node Pool do two jobs at once. The label gives the NodeSet something to select, and the taint keeps every Pod that doesn’t tolerate it off those Nodes. This matters most for CPU-only Slurm nodes. A CPU slurmd Pod requests only a fraction of the Node, so unrelated CPU workloads can fit alongside it. When a Slurm job then starts on that Node, SUNK’s sunk.coreweave.com/lock taint evicts them. A taint on the pool prevents them from landing there at all. If you run GPU Pods with small CPU and memory requests, the same reasoning applies to GPU Nodes. This takes an edit on each side. The following sections cover the Node Pool spec first, then the NodeSet fields that match it.

Label and taint the Node Pool

Add nodeLabels and nodeTaints to the Node Pool spec, using the same key for both so a single value controls selection and repulsion:

slurm-cpu-nodepool.yaml

apiVersion: compute.coreweave.com/v1alpha1
kind: NodePool
metadata:
  name: slurm-cpu
spec:
  computeClass: default
  autoscaling: false
  instanceType: cd-gp-a192-genoa
  maxNodes: 0
  minNodes: 0
  targetNodes: 8
  nodeLabels:
    sunk.example.com/node: "true"
  nodeTaints:
    - key: sunk.example.com/node
      value: "true"
      effect: NoSchedule

Apply it:

kubectl apply -f slurm-cpu-nodepool.yaml

Pick a label prefix in a domain you control. CKS reserves the coreweave.com and coreweave.cloud namespaces for its own labels. A Node Pool that sets a key like sunk.coreweave.com/node is still accepted, but the label never reaches the Nodes, and the only signal is a CWNodePoolMetadataSanitized warning Event on the Node Pool. Replace sunk.example.com throughout these examples with your own domain. Set labels and taints on the Node Pool rather than on individual Nodes. A label set with kubectl label node disappears when CKS replaces that Node. Dedicating a Node manually also means cordoning it, draining it, labeling and tainting it, then uncordoning it, once per Node. CKS applies Node Pool metadata when it delivers a Node, reconciles later edits to nodeLabels and nodeTaints onto the Nodes already in the pool, and reapplies the metadata when a Node is replaced.

A Node Pool can’t transfer a Node to another Node Pool. Moving capacity between pools means scaling one pool down and the other up. Scaling down destroys the Nodes in that pool, and scaling up provisions fresh Nodes. Plan pool boundaries around groupings you expect to keep.

Match the label and taint in the NodeSet

The NodeSet requires an affinity for the label and a toleration for the taint. Both fields sit on the node definition under compute.nodes, and both pass straight through to the slurmd Pod template:

Match both sides of the Node Pool metadata

compute:
  nodes:
    cpu-batch:
      enabled: true
      tolerations:
        - key: sunk.example.com/node
          operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: sunk.example.com/node
                    operator: Exists
      gresGpu: null
      resources:
        limits:
          memory: [MEMORY-LIMIT]
        requests:
          cpu: "[CPU-REQUEST]"
          memory: [MEMORY-REQUEST]

Size resources to the instance type the Node Pool provisions, leaving headroom below the Node’s total capacity for DaemonSets and system overhead. A slurmd Pod whose requests exceed a Node’s allocatable capacity makes that Node infeasible, which produces the same empty Node Pool as a label mismatch. For per-Node capacity, see the CPU instance and GPU instance specifications. operator: Exists matches any value for the key, so you don’t have to keep the toleration’s value in sync with the taint’s. When one label key carries several meaningful values, use operator: In with an explicit values list. The gresGpu: null line matters on a CPU-only definition. node.gresGpu sets the Slurm GPU generic resource for the node. Setting it to null clears any value the definition would otherwise inherit from a layer, so Slurm doesn’t advertise GPUs on a CPU node. With both sides in place, the NodeSet places slurmd Pods only on the Node Pool’s Nodes.

Set scheduling constraints with affinity, never with nodeSelector. A nodeSelector on a node definition doesn’t reach the generated Pod template, so the NodeSet schedules as though no constraint existed. The configuration looks valid and the failure is silent. A NodeSet reporting more Nodes than its target Node Pool contains is the usual symptom.

Let the NodeSet follow Node Pool scale

If you leave replicas unset, the NodeSet sets its desired count to the number of feasible Nodes. Once affinity and tolerations bind the NodeSet to one Node Pool, scaling that pool is the only action needed. This matters because Node Pool size and NodeSet size are otherwise independent. The Node Pool controls how many Kubernetes Nodes exist, and the NodeSet controls how many of them run a slurmd Pod. Scaling the pool while replicas stays fixed adds Nodes that never run a slurmd Pod, which is a common surprise after a scale-up.

Track Node Pool size automatically

compute:
  nodes:
    cpu-batch:
      enabled: true
      # replicas omitted: desired count tracks the feasible Node count
      tolerations:
        - key: sunk.example.com/node
          operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: sunk.example.com/node
                    operator: Exists

To scale, change targetNodes on the Node Pool. Replace [NODE-POOL-NAME] with your Node Pool name. The patch body is JSON, so replace [N] and its brackets with a bare number:

kubectl patch nodepool [NODE-POOL-NAME] --type=merge -p '{"spec":{"targetNodes":[N]}}'

This applies to Node Pools that set targetNodes. A rack-based Node Pool, such as GB200 or GB300, scales with targetRacks instead, since the two fields are mutually exclusive. An older pattern sets replicas to a deliberately large number, such as 1000, to the same effect. The desired count exceeds the feasible count, so the feasible count is the binding constraint. Omitting replicas expresses the same intent without an arbitrary number to maintain. To cap a NodeSet below the size of its Node Pool, set an explicit replicas value. You might do this to reserve Nodes for Kubernetes workloads instead of Slurm. replicas sets the desired count, and the feasible Node count still limits how many Pods the NodeSet places, so the effective size is whichever of the two is lower. When several NodeSets target the same Node Pool, factor the affinity and tolerations into a node definition that other definitions include by name. A definition used as a shared layer must omit enabled. Setting enabled: true on it deploys it as a NodeSet of its own, and every other definition that references it by name silently stops picking up its rules, since an enabled definition is no longer available to be included as a layer:

A reusable matching layer

compute:
  nodes:
    # Shared layer. No `enabled` key, so this is never deployed on its own.
    slurm-cpu:
      tolerations:
        - key: sunk.example.com/node
          operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: sunk.example.com/node
                    operator: Exists
      gresGpu: null

    cpu-a192-genoa:
      enabled: true
      definitions:
        - slurm-cpu
      staticFeatures:
        - cpu
        - amd
        - epyc
      resources:
        limits:
          memory: [MEMORY-LIMIT]
        requests:
          cpu: "[CPU-REQUEST]"
          memory: [MEMORY-REQUEST]

Layers combine rather than overwrite, which is what makes this composition work:

Tolerations accumulate. SUNK adds each layer’s tolerations to the list, so a NodeSet tolerates the taints from every layer it includes plus its own.
Match expressions merge by key and combine with a logical AND. SUNK merges the match expressions from each layer into a single nodeSelectorTerm, keyed on the expression’s key. Expressions on different keys both survive the merge and apply together. A NodeSet that includes a layer selecting sunk.example.com/node and also sets its own expression on gpu.nvidia.com/class requires both. Expressions on the same key don’t cleanly replace the inherited one. The merge unions the two values lists, and it can leave your own expression in place next to the merged one. Kubernetes requires every expression in the term, so the result is narrower than either one alone. Set a given key in one layer only.
Every NodeSet inherits a production Node requirement. SUNK’s base definition contributes a match expression requiring node.coreweave.cloud/state to be production. Because expressions on other keys merge alongside it, your affinity narrows that selection rather than replacing it. Don’t write your own expression on node.coreweave.cloud/state. A same-key expression can’t cleanly override the inherited one, so the result rarely matches what you intended. For what the other states mean, see Node lifecycle labels.

SUNK resolves layers in the order listed under definitions and applies the NodeSet’s own values last, so a NodeSet can override any value it inherits.

Only the first entry under nodeSelectorTerms takes part in this merge. Kubernetes treats multiple terms as a logical OR, but SUNK drops a layer’s second and later terms when it merges them. Keep each definition to a single nodeSelectorTerm and express alternatives with operator: In and multiple values.

For more on composing node definitions, see Custom node definitions.

Verify the binding

Deploy the Helm release, then compare what the Node Pool delivered against what the NodeSet can use.

Confirm the Node Pool reached its target

Replace [NODE-POOL-NAME] with your Node Pool name:

kubectl get nodepool [NODE-POOL-NAME]

CURRENT should equal TARGET. If it doesn’t, the gap is a Node Pool problem rather than a matching problem. See Node Pool status.

Check that the Nodes carry the expected metadata

kubectl get nodes -l sunk.example.com/node=true

An empty result means the labels aren’t reaching the Nodes. Confirm nodeLabels is set on the Node Pool spec, then check the Node Pool’s Events:

kubectl describe nodepool [NODE-POOL-NAME]

A CWNodePoolMetadataSanitized warning means CKS dropped the label because its key sits in a reserved namespace. Pick a key in a domain you control and reapply. For every Event a Node Pool can fire, see Node Pool reference.

Compare NodeSet feasibility against the Node Pool

Replace [NAMESPACE] with your Slurm namespace:

kubectl get nodeset -n [NAMESPACE]

Example output

NAME               DESIRED   FEASIBLE   CURRENT   READY   UP-TO-DATE   RUNNING   DRAIN   MISSCHEDULED   UNAVAILABLE   AGE
slurm-cpu-batch    8         8          8         8       8            0         0       0              0             12m

FEASIBLE equal to the Node Pool’s CURRENT means the matching is correct. FEASIBLE lower than CURRENT means some Nodes in the pool don’t satisfy the NodeSet’s affinity, tolerations, or resource requests. For what each column means, see NodeSet status.

Confirm Pod placement

SUNK labels each slurmd Pod with app.kubernetes.io/instance, set to the NodeSet name. Replace [NODESET-NAME] with the name from the previous step:

kubectl get pods -n [NAMESPACE] -l app.kubernetes.io/instance=[NODESET-NAME] -o wide

Every Pod should be on a Node from the intended Node Pool. A Pod on any other Node means the constraint isn’t reaching the Pod template, usually because the node definition sets it with nodeSelector instead of affinity.

Confirm Slurm sees the nodes

From a Slurm login node:

sinfo

The node count for the partition should match the NodeSet’s READY count. See Connect to a Slurm login node.

Troubleshoot

The following table maps the mismatch you see to its usual cause.

Symptom	Likely cause	Fix
`FEASIBLE` is `0` and the Node Pool is healthy	The NodeSet’s affinity doesn’t match any Node label, or it doesn’t tolerate the Node Pool’s taint.	Compare the NodeSet’s `affinity` and `tolerations` keys against `kubectl describe node` output for one Node in the pool.
`FEASIBLE` is short by exactly the Node count of another Node Pool	The NodeSet names one pool, but the instance type’s capacity spans several.	Add the other pool to `values` with `operator: In`, or match a label both pools share.
`DESIRED` is lower than `FEASIBLE` after a Node Pool scale-up	`replicas` caps the NodeSet below the Node Pool size.	Omit `replicas`, or raise it.
`FEASIBLE` is one or two below the Node Pool’s `CURRENT`	A non-Slurm Pod occupies those Nodes and consumes the resources the `slurmd` Pod requests.	Add a taint to the Node Pool, or fix the other workload’s affinity to exclude these Nodes. Drain the Node to evict the Pod already there.
The NodeSet uses more Nodes than the target Node Pool contains	The node definition sets the constraint with `nodeSelector`, which doesn’t reach the Pod template.	Move the constraint into an `affinity` block with `requiredDuringSchedulingIgnoredDuringExecution`.
`MISSCHEDULED` is above `0`	Pods sit on Nodes that no longer match the NodeSet’s selector, usually after a label or affinity change.	Confirm the intended Nodes, then let the NodeSet replace the Pods.
Nodes lose their labels after a Node Pool scale event	The labels were applied to individual Nodes instead of the Node Pool spec.	Move the labels and taints into the Node Pool’s `nodeLabels` and `nodeTaints`.
A label added to the Node Pool never appears on any Node	The label key is in the reserved `coreweave.com` or `coreweave.cloud` namespace, so CKS drops it.	Check for a `CWNodePoolMetadataSanitized` Event on the Node Pool, then move the label to a domain you control.

For deeper NodeSet triage, see NodeSet status. For Node Pool conditions, capacity, and quota, see Node Pool status.

Create a Node Pool for the full Node Pool spec, including nodeLabels and nodeTaints.
Configure compute nodes for every field available on a node definition.
Workload scheduling on CKS for reserved label namespaces, CKS taints, and the SUNK lock taint.
NodeSet for how the NodeSet controller selects Nodes and reports status.
Node Pool status for diagnosing a Node Pool that hasn’t reached its target.

​How NodeSet scheduling uses Node Pool metadata

​Prerequisites

​Target a Node Pool by name

​Dedicate a Node Pool to a NodeSet

​Label and taint the Node Pool

​Match the label and taint in the NodeSet

​Let the NodeSet follow Node Pool scale

​Share matching rules across NodeSets

​Verify the binding

​Troubleshoot

​Related pages

How NodeSet scheduling uses Node Pool metadata

Prerequisites

Target a Node Pool by name

Dedicate a Node Pool to a NodeSet

Label and taint the Node Pool

Match the label and taint in the NodeSet

Let the NodeSet follow Node Pool scale

Share matching rules across NodeSets

Verify the binding

Troubleshoot

Related pages