Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt

Use this file to discover all available pages before exploring further.

In , Nodes are organized using namespaces, labels, taints, and eviction policies. These features control where CoreWeave’s core managed services are scheduled and ensure that customer workloads always run on healthy production Nodes.

​
CKS namespaces

CKS has two types of namespaces:
  • User namespaces are labeled with your Org ID.
    • You have full control over user namespaces - you can create, change, and delete them.
  • Control Plane namespaces are created by CoreWeave to host critical services that run within the cluster.
    • Do not alter or delete these namespaces. CoreWeave workloads are automated within these managed namespaces. Jobs that run in Control Plane namespaces are not billed to the customer.
CKS applies the label ns.coreweave.cloud/org=control-plane to all Control Plane namespaces. To view these namespaces in a CKS cluster: 
kubectl get namespaces --selector=ns.coreweave.cloud/org=control-plane

​
Node type selection labels

All CoreWeave GPU and CPU Nodes feature Instance IDs. To ensure consistency, all Nodes within a Node Pool are tagged with an Instance ID using the instance-type label. For example: 
node.kubernetes.io/instance-type=instance-type-example
For a list of all Instance IDs, see the GPU Instance and CPU Instance details.
Some customers may be using the node.kubernetes.io/type label. This label has been updated to reference the new Instance ID.

​
Node lifecycle labels

CKS uses labels to identify a Node’s state in the Node lifecycle. These labels ensure that customer workloads are always scheduled on healthy production Nodes.
  • node.coreweave.cloud/state: Identifies the Node’s lifecycle state, such as production, zap, or triage.
  • node.coreweave.cloud/reserved: Identifies the workload type running on the Node:
    • If /reserved is a customer Org ID and /state=production, it’s for user workloads.
    • If /state is not production, then /reserved matches /state.
  • node.coreweave.cloud/reserved-desired: Overrides /reserved. If it doesn’t match /reserved, the Node marked is pending and will transition Reservations automatically.

​
User-provided labels

Customers may create custom Node labels for scheduling or organization, but never in the *.coreweave.cloud or *.coreweave.com namespaces. Attempts to do so are rejected by CKS. 
metadata:
  labels:
    foo.coreweave.cloud/bar: "true"   # Not allowed
    foo.coreweave.com/bar: "true"     # Not allowed

​
Pod interruption and eviction policies

CKS supports three eviction strategies for Pods: non-interruptible, interruptible, and gracefully interruptible. These strategies determine how CKS handles Pods during Node maintenance, reboots, or scale-down events.

​
Summary of eviction strategies

StrategyPod LabelDescriptionNote
Non-interruptibleNone (default)CKS will not proceed with maintenance or scale-down while these Pods are running. Use for critical training jobs and single-instance stateful apps.Default behavior for Nodes, ensuring stability and reliability.
Interruptibleqos.coreweave.cloud/interruptableCKS terminates Pods and proceeds with the Node action without waiting for their full terminationGracePeriodSeconds. Use for stateless workloads that can be restarted without data loss.Misspelled as interruptable for historical reasons; in the qos.coreweave.cloud namespace.
Gracefully interruptibleqos.coreweave.com/graceful-interruptibleCKS blocks the Node action until the Pod terminates or its terminationGracePeriodSeconds expires. Use for stateful applications that can handle graceful shutdowns.Spelled correctly; in the qos.coreweave.com namespace.
See the sections below for details about each strategy.

​
Non-interruptible

Workloads that should not be interrupted, such as critical training jobs and single-instance stateful apps, should not set either of the two labels in the table above. If neither label is set, the Pod is considered non-interruptible. CKS treats Nodes with non-interruptible Pods as active. This means CKS will not proceed with Node maintenance, reboots, or scale-down while these Pods are running. Non-interruptible Pods are never evicted unless an extreme event occurs, such as complete Node failure or DC power loss. This is the default behavior for Pods.

​
Interruptible

Workloads such as inference Pods or stateless applications that can be restarted without data loss should use the interruptible strategy. CKS terminates these Pods and proceeds with the Node action (such as reboot or scale-down) without waiting for the Pod’s full terminationGracePeriodSeconds. Never apply this label to distributed training jobs, single-instance databases, stateful services, or any workload that cannot tolerate sudden termination. Evicting these workloads may cause data loss or require costly restarts across multiple Nodes. To choose this strategy, apply the label qos.coreweave.cloud/interruptable: "true" to your Pods. This label is in the qos.coreweave.cloud namespace. 
metadata:
  labels:
    qos.coreweave.cloud/interruptable: "true"
Note that for historical reasons, the label is misspelled as interruptable.

​
Gracefully interruptible

Workloads that can exit cleanly before Node maintenance begins should consider using the gracefully interruptible strategy. This is particularly useful for stateful applications that can handle graceful shutdowns, such as replicated databases or stateful services. Unlike interruptible Pods, which are deleted immediately without waiting, gracefully interruptible Pods block the Node action (including force reboots) until the Pod terminates or its terminationGracePeriodSeconds expires. CKS sends a DELETE call to the Pod and waits for the full grace period before proceeding. To choose this, apply the label qos.coreweave.com/graceful-interruptible: "true" to your Pods. This label is in the qos.coreweave.com namespace. 
metadata:
  labels:
    qos.coreweave.com/graceful-interruptible: "true"

​
Key behaviors and limitations

​
NodePool scale-down

Nodes hosting graceful-interruptible Pods are skipped when determining if the Node is a candidate for removal. This means scale-down may stall if all Pods on candidate Nodes carry this label.

​
Tolerations that prevent graceful eviction

Pods that tolerate either node.coreweave.cloud/evict=true:NoExecute or node.coreweave.cloud/reserved:NoExecute will not be processed via the graceful-interruptible logic and may be evicted immediately. See Eviction taints below.

​
Drain time differences

  • Reboots and Maintenance use a default drain timeout of three minutes, and honor terminationGracePeriodSeconds for graceful-interruptible Pods.
  • CKS-initiated scale-down does not wait for DaemonSets unless they carry qos.coreweave.com/graceful-interruptible: "true" and do not tolerate eviction taints.
  • Services that need long termination phases should explicitly set terminationGracePeriodSeconds accordingly.

​
Potential for scale-down stalls

By design, CKS will never remove a Node containing only graceful-interruptible Pods. If every Pod on a candidate Node carries this label, CKS has nowhere to reclaim capacity and will stall waiting for Nodes that can be safely drained. In practice, this can block automated scale-down workflows.

​
Risk of stuck Nodes

If workloads are deployed without accounting for graceful-interruptible semantics, Nodes can remain in a quasi-drained state indefinitely. For example, you may cordon a Node for maintenance, then find it never transitions to “Ready” again because all Pods refuse immediate eviction. Left unchecked, these Nodes consume capacity and can complicate rolling updates. To mitigate these risks:
  1. Plan deployment strategies to ensure some interruptable Pods exist to give CKS safe eviction candidates.
  2. Monitor NodePool capacity and scheduling health. Set up alerts on stalled scale-down events or sustained high utilization to detect when Nodes are being held due to graceful-interruptible Pods.
  3. Establish maintenance procedures that include manual intervention steps (e.g., draining and deleting problematic Nodes) as a fallback when automated processes cannot reclaim resources.
By weighing the benefits of smoother in-place upgrades against these trade-offs, teams can decide how and when to use graceful-interruptible without compromising cluster resilience or cost efficiency.

​
Taints and tolerations

CKS uses taints to guard control-plane Nodes and enforce GPU/CPU scheduling.

​
Eviction taints

  • node.coreweave.cloud/evict=true:NoExecute
  • node.coreweave.cloud/reserved:NoExecute
Pods tolerating these will bypass graceful eviction and may be evicted immediately.

​
User taints

The CPU taint (is_cpu_compute:NoSchedule) is automatically tolerated by Pods without GPU requests.
CPU taint
  - effect: NoSchedule
    key: is_cpu_compute
The GPU taint (is_gpu=true:PreferNoSchedule) prevents CPU-only Pods from scheduling on GPU Nodes unless necessary. A CPU-only Pod can still schedule on a GPU Node if no CPU Nodes are available.
GPU taint
  - effect: PreferNoSchedule
    key: is_gpu
    value: "true"
Customers should exercise caution before attempting to add tolerations to their Pods to ensure workloads always run on healthy Nodes.

​
SUNK-specific scheduling

​
SUNK’s /lock taint

To prevent contention with other Pods that request GPU access while long-running slurmd Pods are active, SUNK adds a new GPU resource to Kubernetes, sunk.coreweave.com/accelerator, in addition to the nvidia.com/gpu resource provided by NVIDIA’s plugin. Because the GPU has two different resource names, Kubernetes tracks the consumption separately, which allows Slurm Pods to request the same underlying GPU as other Kubernetes Pods. However, this requires SUNK to manage GPU contention instead of the Kubernetes scheduler. SUNK manages the contention with a taint called sunk.coreweave.com/lock. SUNK applies this taint to Nodes by making a call to slurm-syncer during the Prolog phase.
SUNK's lock taint
  - effect: NoExecute
    key: sunk.coreweave.com/lock
    value: "true"
Prolog completion is blocked until all Pods that do not tolerate the taint have been evicted.

​
DaemonSets on SUNK Nodes

Kubernetes DaemonSets that run on SUNK Nodes must tolerate the sunk.coreweave.com/lock taint, as well as is_cpu_compute, is_gpu, and node.coreweave.cloud/reserved:
Example toleration
spec:
  tolerations:
  - key: sunk.coreweave.com/lock
    value: "true"
    operator: Equal

  - key: is_cpu_compute
    operator: Exists

  - key: is_gpu
    operator: Exists

​
Scaling down workloads

To scale down Pods in a specific order, use the Cloud Console or CKS API to adjust the cluster specification. For more, see the official Kubernetes guide on Pod deletion cost.
Last modified on April 20, 2026