Workload scheduling on CKS

In , namespaces, labels, taints, and eviction policies organize Nodes. These features control where CoreWeave schedules its core managed services and ensure that customer workloads always run on healthy production Nodes. This page is a reference for cluster operators and workload owners who need to understand how CKS schedules Pods, which labels and taints CoreWeave reserves, how to target Nodes with a selector or affinity rule, and how to choose an eviction strategy that matches a workload’s tolerance for interruption. For cluster-level scheduler configuration, see Scheduling profiles. For third-party schedulers, see Custom schedulers and third-party integrations. For runtime-specific Node selection and isolation requirements, see Container runtimes on CKS.

CKS namespaces

CKS has two types of namespaces:

User namespaces carry your Org ID label. You have full control over user namespaces. You can create, change, and delete them.
Control Plane namespaces host critical services that run within the cluster. CoreWeave creates these namespaces. Don’t alter or delete them. CoreWeave automates workloads within these managed namespaces and doesn’t bill customers for jobs that run in Control Plane namespaces.

CKS applies the label ns.coreweave.cloud/org=control-plane to cw-* prefix namespaces, which host the CKS control plane components. To view these namespaces in a CKS cluster:

kubectl get namespaces --selector=ns.coreweave.cloud/org=control-plane

For the components that run in each namespace, see Cluster components.

Node type selection labels

All CoreWeave GPU and CPU Nodes feature Instance IDs. To ensure consistency, every Node within a Node Pool carries an Instance ID through the instance-type label. For example:

node.kubernetes.io/instance-type=instance-type-example

For a list of all Instance IDs, see the GPU Instance and CPU Instance details.

Some customers may use the node.kubernetes.io/type label. CoreWeave updated this label to reference the new Instance ID.

Node lifecycle labels

CKS uses labels to identify a Node’s state in the Node lifecycle. These labels ensure that CKS always schedules customer workloads on healthy production Nodes.

node.coreweave.cloud/state: Identifies the Node’s lifecycle state, such as production, zap, or triage.
node.coreweave.cloud/reserved: Identifies the workload type running on the Node:
- If /reserved is a customer Org ID and /state=production, it’s for user workloads.
- If /state is not production, then /reserved matches /state.
node.coreweave.cloud/reserved-desired: Overrides /reserved. If it doesn’t match /reserved, the marked Node is pending and transitions Reservations automatically.

User-provided labels

Create custom Node labels for scheduling or organization, but never in the *.coreweave.cloud or *.coreweave.com namespaces. CKS reserves those for its own labels. A Node Pool that sets one in nodeLabels is still accepted, but the label never reaches the Nodes. CKS drops it and fires a CWNodePoolMetadataSanitized warning Event on the Node Pool.

metadata:
  labels:
    foo.coreweave.cloud/bar: "true"   # Not allowed
    foo.coreweave.com/bar: "true"     # Not allowed

Target Nodes

To run a Pod on a specific instance type, rack, or Node, use nodeSelector or nodeAffinity. Both keep the Kubernetes scheduler involved, so preemption, priority, topology spread, and Kueue integration all work. Note that a Pod whose schedulerName is the SUNK Pod Scheduler has its placement decided by Slurm instead: Node affinity still applies, but Pod affinity, Pod anti-affinity, and topology spread constraints have no effect. Pods that use the default scheduler are unaffected. A nodeSelector matches a single value for each label and covers most cases. Replace the bracketed placeholder in each of the following examples with your target value. To target an instance type, match the instance-type label described in Node type selection labels:

Target an instance type

spec:
  nodeSelector:
    node.kubernetes.io/instance-type: [INSTANCE-ID]

To keep a job on Nodes that share an NVLink domain, match the NVLink domain label. On NVL72-powered Nodes, every Node in a rack shares the same globally unique value:

Target an NVLink domain

spec:
  nodeSelector:
    ds.coreweave.com/nvlink.domain: [NVLINK-DOMAIN]

To place Pods in the same physical rack, match the rack number. Quote the value, because Kubernetes label values are strings:

Target a rack

spec:
  nodeSelector:
    node.coreweave.cloud/rack: "[RACK-NUMBER]"

To reach one specific Node, match its hostname. This is the supported alternative to .spec.nodeName:

Target a single Node

spec:
  nodeSelector:
    kubernetes.io/hostname: [NODE-NAME]

For more flexible matching, use nodeAffinity with operators such as In, NotIn, and Exists. Affinity accepts multiple values for the same key, which helps when a Node Pool spans more than one rack or NVLink domain:

Match multiple NVLink domains with nodeAffinity

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: ds.coreweave.com/nvlink.domain
            operator: In
            values:
            - [NVLINK-DOMAIN-A]
            - [NVLINK-DOMAIN-B]

To target a specific InfiniBand or RoCE fabric, superpod, or leafgroup, see InfiniBand and RoCE labels.

Avoid `.spec.nodeName` for Node targeting

Never use .spec.nodeName to pin a Pod to a Node. Setting .spec.nodeName bypasses the Kubernetes scheduler entirely. The Pod already looks scheduled, so the kubelet on the named Node admits it directly. As a result, the Pod gets no preemption, no priority ordering, no affinity or topology spread, and no Kueue or Topology-Aware Scheduling. If the Node lacks free resources, the kubelet rejects the Pod during admission instead of triggering preemption. The Pod ends in Failed with a reason such as OutOfcpu, OutOfmemory, or OutOfnvidia.com/gpu. Because the scheduler never saw the Pod, kubectl describe pod reports no FailedScheduling event to explain the outcome. CoreWeave runs short HPC Verification Pods on idle Nodes at a priority lower than customer workloads. The default-scheduler preempts them automatically, so you usually don’t notice them. Third-party schedulers don’t preempt them by default. A Pod pinned with .spec.nodeName can’t trigger that preemption, so it can fail to schedule on a Node that looks free. Use nodeSelector or nodeAffinity instead. See Why is my Pod stuck in pending on CKS?.

Pod interruption and eviction policies

CKS supports three eviction strategies for Pods: non-interruptible, interruptible, and gracefully interruptible. These strategies determine how CKS handles Pods during Node maintenance, reboots, or scale-down events.

Summary of eviction strategies

Strategy	Pod label	Description	Note
Non-interruptible	None (default)	CKS waits for these Pods to stop before deferred maintenance, safe reboots, and Node Pool scale-down with the default `IdleOnly` strategy. Use for critical training jobs and single-instance stateful apps.	Immediate actions, `PreferIdle` scale-down, the Cluster Autoscaler, and prefill timeout expiry can still interrupt these Pods.
Interruptible	`qos.coreweave.cloud/interruptable`	CKS terminates Pods and proceeds with the Node action without waiting for their full `terminationGracePeriodSeconds`. Use for stateless workloads that you can restart without data loss.	Misspelled as `interruptable` for historical reasons. In the `qos.coreweave.cloud` namespace.
Gracefully interruptible	`qos.coreweave.com/graceful-interruptible`	Reboots and deferred maintenance wait until the Pod terminates or its `terminationGracePeriodSeconds` expires. Node Pool removal has a separate drain deadline. Use only for stateful applications that block rebooting until the application terminates, such as databases and local storage. For all other workloads, use interruptible.	Spelled correctly. In the `qos.coreweave.com` namespace.

Choose one interruption strategy per Pod; don’t combine eviction labels. For Pods using only qos.coreweave.cloud/interruptable: "true", CKS excludes the Pod from the CWActive Node condition. If a Pod also has qos.coreweave.com/graceful-interruptible: "true", the gracefully interruptible behavior takes precedence for CWActive. DaemonSet Pods remain exempt. Gracefully interruptible Pods can remain counted during termination, even after their termination grace period expires. Check CWActive to confirm when the Node becomes idle. The following sections describe each strategy in detail.

Non-interruptible

Workloads that you don’t want interrupted, such as critical training jobs and single-instance stateful apps, shouldn’t set either of the two labels in the preceding table. If neither label is set, the Pod is considered non-interruptible. CKS treats Nodes with non-interruptible Pods as active, which is the default behavior for Pods. CKS waits for these Pods to stop before proceeding with deferred Node maintenance or safe reboots. When you lower targetNodes on a Node Pool that uses the default IdleOnly scale-down strategy, CKS also waits for these Pods before it removes a Node. The PreferIdle strategy and the Cluster Autoscaler can remove a Node that still runs non-interruptible Pods. A Node in the prefill flow can also start draining when its idle timeout expires, even if these Pods are still running. Immediate state transitions, including administrator-requested immediate reboots, can still interrupt non-interruptible Pods. Extreme events such as complete Node failure or data center power loss can also interrupt these Pods. Leaving both interruption labels unset doesn’t protect a Pod from an immediate reboot.

Long-running GPU Pods

A Pod that holds GPU resources (nvidia.com/gpu) indefinitely without an interruptible label has two consequences:

Deferred maintenance can stall: CKS ordinarily waits for non-interruptible workloads to finish before starting discretionary maintenance. A long-running Pod can keep a pending transition waiting. For a Node in the prefill flow, CKS can start draining it after the prefill timeout expires, even if workloads are still running. Critical hardware failures can still trigger immediate action.
Prevents HPC Verification tests: CoreWeave runs hourly HPC Verification tests on Nodes where all GPUs are available. A long-running Pod that holds GPUs prevents these tests from scheduling on that Node for the duration of the Pod’s lifetime.

To allow deferred maintenance to proceed, apply the interruptible label to Pods that can tolerate interruption. For stateful Pods that need cleanup time, use graceful-interruptible instead. Once non-interruptible workloads finish, deferred maintenance can drain either kind of Pod. CKS proceeds without waiting for interruptible Pods to finish evicting. For gracefully interruptible Pods, it waits until the Pod terminates or its terminationGracePeriodSeconds expires after the deletion request. This bounded wait doesn’t exclude gracefully interruptible Pods from CWActive. To allow HPC Verification tests to run on a Node alongside a long-running GPU Pod, the Pod must not hold any GPUs. SUNK addresses this by introducing a separate sunk.coreweave.com/accelerator resource for slurmd Pods, leaving the Node’s GPUs available for HPC Verification. See SUNK-specific scheduling for details.

Interruptible

Workloads such as inference Pods or stateless applications that you can restart without data loss should use the interruptible strategy. CKS terminates these Pods and proceeds with the Node action (such as reboot or scale-down) without waiting for the Pod’s full terminationGracePeriodSeconds.

Never apply this label to distributed training jobs, single-instance databases, stateful services, or any workload that can’t tolerate sudden termination. Evicting these workloads may cause data loss or require costly restarts across multiple Nodes.

To choose this strategy, apply the label qos.coreweave.cloud/interruptable: "true" to your Pods. This label is in the qos.coreweave.cloud namespace.

metadata:
  labels:
    qos.coreweave.cloud/interruptable: "true"

Note that for historical reasons, the label is misspelled as interruptable.

Gracefully interruptible

Use the gracefully interruptible strategy only for stateful applications that block a reboot until the application terminates, such as databases and local storage. These workloads need to finish writing data or hand off state before a Node reboots or drains. For all other applications, including stateless workloads that can be restarted without data loss, use the interruptible strategy instead. Gracefully interruptible Pods delay reboots (including immediate reboots) and deferred maintenance until they terminate or their terminationGracePeriodSeconds expires. Node Pool removal has a separate drain deadline and can remove a Node before a longer Pod grace period expires; see Node idle status and maintenance. A deletion request starts the termination grace period. The Pod can continue to count toward CWActive while its object remains in the Kubernetes API, even after that grace period expires. Grace-period expiry alone doesn’t establish that the Node is idle; inspect CWActive. A Node running only gracefully interruptible Pods can therefore still show Active on dashboards that reflect CWActive. To choose this, apply the label qos.coreweave.com/graceful-interruptible: "true" to your Pods. This label is in the qos.coreweave.com namespace.

metadata:
  labels:
    qos.coreweave.com/graceful-interruptible: "true"

Key behaviors and limitations

The following sections describe behaviors and limitations to consider when using graceful-interruptible Pods.

NodePool scale-down

When choosing a new Node to drain for scale-down, the default IdleOnly strategy requires CWActive = False. A counted gracefully interruptible Pod can prevent selection even if the Node also hosts interruptible Pods or has unused GPUs. Adding interruptible Pods doesn’t make a Node idle while counted Pods remain. To make Nodes eligible, finish or move the workloads keeping them active, then confirm CWActive = False. If you need scale-down to interrupt active workloads, consider the PreferIdle strategy. The Cluster Autoscaler uses its own removal rules. Nodes already marked for removal or already draining follow their existing removal workflow.

Tolerations that prevent graceful eviction

Pods that tolerate either node.coreweave.cloud/evict=true:NoExecute or node.coreweave.cloud/reserved:NoExecute don’t go through the graceful-interruptible logic and may be evicted immediately. See Eviction taints.

Drain time differences

Reboots and deferred maintenance use a default drain timeout of 3 minutes, and honor terminationGracePeriodSeconds for graceful-interruptible Pods.
Node Pool removal has a separate 5-minute drain timeout. A longer Pod termination grace period doesn’t extend this deadline.
CKS-initiated scale-down doesn’t wait for DaemonSets unless they carry qos.coreweave.com/graceful-interruptible: "true" and don’t tolerate eviction taints.
For services that require long termination phases, explicitly set terminationGracePeriodSeconds and account for the deadline on each removal path.

Potential for scale-down stalls

If too few Nodes become idle, the Node Pool can remain above its target. Use the Node Pool health check commands to inspect CWActive and identify the Pods keeping each Node active. See Current is above target for diagnosis steps.

Risk of stuck Nodes

IdleOnly can wait for an eligible Node without starting a drain. Maintenance instead drains gracefully interruptible Pods after non-interruptible workloads finish, then waits for Pod termination or expiry of the termination grace period. This maintenance wait can finish while CWActive is still True. If a maintenance action remains stalled, inspect the Node conditions and events and contact Support with the Node name and the blocking messages.

Taints and tolerations

CKS uses taints to guard control-plane Nodes and enforce GPU and CPU scheduling. The following sections describe the eviction taints CKS applies to Nodes and the user-facing taints that route Pods to the correct hardware.

Eviction taints

CKS applies the following eviction taints to Nodes:

node.coreweave.cloud/evict=true:NoExecute
node.coreweave.cloud/reserved:NoExecute

Pods that tolerate these bypass graceful eviction and may be evicted immediately.

User taints

For custom taints on GPU Nodes, use NoSchedule to prevent new Pods without matching tolerations from scheduling. Use PreferNoSchedule when a scheduling preference is sufficient. Neither effect evicts Pods that are already running. Do not use NoExecute taints on GPU Nodes. HPC Verification Pods tolerate arbitrary NoSchedule and PreferNoSchedule taints, but not arbitrary NoExecute taints. A NoExecute taint can block automated verification on both idle Nodes and rebooted Nodes, which leaves them unverified. CoreWeave might replace Nodes that remain in a reboot state too long. This guidance applies to customer-managed taints, including taints configured on a Node Pool. Leave CoreWeave-managed taints unchanged. To request a test, see Verify a Node on demand. Pods without GPU requests automatically tolerate the CPU taint (is_cpu_compute:NoSchedule).

CPU taint

  - effect: NoSchedule
    key: is_cpu_compute

The GPU taint (is_gpu=true:PreferNoSchedule) prevents CPU-only Pods from scheduling on GPU Nodes unless necessary. A CPU-only Pod can still schedule on a GPU Node if no CPU Nodes are available.

GPU taint

  - effect: PreferNoSchedule
    key: is_gpu
    value: "true"

Use caution before adding tolerations to your Pods, so workloads continue to run on healthy Nodes.

SUNK-specific scheduling

The following sections describe scheduling behaviors specific to SUNK workloads on CKS. To reserve a Node Pool for a SUNK NodeSet, match the pool’s nodeLabels and nodeTaints to the NodeSet’s affinity and tolerations. See Bind NodeSets to Node Pools for SUNK Standard.

The SUNK `/lock` taint

To prevent contention with other Pods that request GPU access while long-running slurmd Pods are active, SUNK adds a new GPU resource to Kubernetes, sunk.coreweave.com/accelerator, in addition to the nvidia.com/gpu resource provided by NVIDIA’s plugin. Because the GPU has two different resource names, Kubernetes tracks the consumption separately, which lets slurmd Pods request the same underlying GPU as other Kubernetes Pods. However, this requires SUNK to manage GPU contention instead of the Kubernetes scheduler.

Use nvidia.com/gpu for GPU requests in all Pod specs, including Pods scheduled by the SUNK Pod Scheduler. Don’t request sunk.coreweave.com/accelerator directly. SUNK reserves this resource for its own internal Pods, such as slurmd, and converts nvidia.com/gpu requests into Slurm generic resource (GRES) allocations. A direct request for sunk.coreweave.com/accelerator bypasses this conversion and can cause incorrect GPU accounting or contention between Slurm jobs and Kubernetes Pods.

SUNK manages the contention with a taint called sunk.coreweave.com/lock. SUNK applies this taint to Nodes through a call to the SUNK Syncer during the Prolog phase.

SUNK's lock taint

  - effect: NoExecute
    key: sunk.coreweave.com/lock
    value: "true"

Prolog completion blocks until SUNK evicts every Pod that doesn’t tolerate the taint.

DaemonSets on SUNK Nodes

Kubernetes DaemonSets that run on SUNK Nodes must tolerate the sunk.coreweave.com/lock taint, as well as is_cpu_compute, is_gpu, and node.coreweave.cloud/reserved:

Example toleration

spec:
  tolerations:
  - key: sunk.coreweave.com/lock
    value: "true"
    operator: Equal

  - key: is_cpu_compute
    operator: Exists

  - key: is_gpu
    operator: Exists

  - key: node.coreweave.cloud/reserved
    operator: Exists

Scale down workloads

To scale down Pods in a specific order, use the Cloud Console or CKS API to adjust the cluster specification. For more information, see the Kubernetes guide about Pod deletion cost.

Troubleshoot Pod scheduling

If a GPU Pod stays Pending, see Why are my Pods not scheduling on GPU Nodes? for common causes and fixes.

Resource requests for GPU Pods

Size CPU and memory requests to roughly the per-GPU share of the instance, leaving headroom for DaemonSets and system overhead. Specify GPUs under limits as nvidia.com/gpu: [COUNT]. Kubernetes mirrors the value to requests automatically. For a worked whole-Node example, see Target specific GPUs or CPUs. For long-running training and inference, set CPU and memory requests equal to limits so the Pod gets the Guaranteed Quality-of-Service (QoS) class and is the last to be evicted under resource pressure. See How do Kubernetes QoS classes work?. For recommended ratios and how requests behave on GPU Pods, see What are the recommended resource requests and limits for GPU Pods? and How do CPU and memory requests work with GPU Pods?.

Create and manage clusters

Configure cluster access

Deploy workloads

Secure your cluster

Tutorials

Reference

Workload scheduling on CKS

CKS namespaces

Node type selection labels

Node lifecycle labels

User-provided labels

Target Nodes

Avoid `.spec.nodeName` for Node targeting

Pod interruption and eviction policies

Summary of eviction strategies

Non-interruptible

Long-running GPU Pods

Interruptible

Gracefully interruptible

Key behaviors and limitations

NodePool scale-down

Tolerations that prevent graceful eviction

Drain time differences

Potential for scale-down stalls

Risk of stuck Nodes

Taints and tolerations

Eviction taints

User taints

SUNK-specific scheduling

The SUNK `/lock` taint

DaemonSets on SUNK Nodes

Scale down workloads

Troubleshoot Pod scheduling

Resource requests for GPU Pods

​CKS namespaces

​Node type selection labels

​Node lifecycle labels

​User-provided labels

​Target Nodes

​Avoid .spec.nodeName for Node targeting

​Pod interruption and eviction policies

​Summary of eviction strategies

​Non-interruptible

​Long-running GPU Pods

​Interruptible

​Gracefully interruptible

​Key behaviors and limitations

​NodePool scale-down

​Tolerations that prevent graceful eviction

​Drain time differences

​Potential for scale-down stalls

​Risk of stuck Nodes

​Taints and tolerations

​Eviction taints

​User taints

​SUNK-specific scheduling

​The SUNK /lock taint

​DaemonSets on SUNK Nodes

​Scale down workloads

​Troubleshoot Pod scheduling

​Resource requests for GPU Pods

CKS namespaces

Node type selection labels

Node lifecycle labels

User-provided labels

Target Nodes

Avoid `.spec.nodeName` for Node targeting

Pod interruption and eviction policies

Summary of eviction strategies

Non-interruptible

Long-running GPU Pods

Interruptible

Gracefully interruptible

Key behaviors and limitations

NodePool scale-down

Tolerations that prevent graceful eviction

Drain time differences

Potential for scale-down stalls

Risk of stuck Nodes

Taints and tolerations

Eviction taints

User taints

SUNK-specific scheduling

The SUNK `/lock` taint

DaemonSets on SUNK Nodes

Scale down workloads

Troubleshoot Pod scheduling

Resource requests for GPU Pods