Skip to main content
When CoreWeave’s Node lifecycle automation detects an issue on a production Node during Day 2+ operations, it remediates the Node by moving it through a lifecycle state transition. That transition might be a reboot inside your cluster or a move to triage for investigation. This page explains how those transitions behave in a CoreWeave Kubernetes Service (CKS) cluster: the difference between pending and immediate transitions, the labels and conditions that mark them, the states you can observe, and how your workloads affect transition timing. For the full lifecycle from initialization through production, see Node lifecycle.

​
Remediation categories

CoreWeave’s health signals fall into two categories:
  • Solved by reboot: The Node reboots inside your cluster and returns to service. Examples include GPU reset requests and kernel deadlocks.
  • Needs triage: The Node leaves your cluster for investigation, repair, preventative maintenance, or vendor return. CKS delivers a replacement Node automatically.
For either category, CoreWeave matches the type of transition, pending or immediate, to the known severity of the issue.

​
Pending state transitions

CoreWeave uses a pending transition when the issue’s severity is uncertain and interrupting a running workload would be worse than waiting. A pending transition works as follows:
  1. CKS sets the node.coreweave.cloud/pending-state label on the Node to the target state, such as production-reboot, and updates the PendingPhaseState Node condition. The condition’s reason names the target state, and its message describes what triggered the transition.
  2. CKS cordons the Node so no new workloads schedule onto it, and records the reason, NLCCPendingExitProduction, in the node.coreweave.cloud/cordonReason annotation. See Node cordoning.
  3. The transition waits until the Node is no longer active. While your workloads are still running, the Node emits CWNCPhasePendingTenantWorkloadRunning events that name the Pods preventing the transition.
  4. When the Node becomes inactive, the state transition proceeds and the action takes place.
Existing workloads keep running during the pending period. This design assumes that a serious fault crashes the workload and takes its Pod down with it. The Pod exits, the Node becomes inactive, and the transition proceeds without interrupting anything that was still healthy.

​
Immediate state transitions

CoreWeave uses an immediate transition when the issue is known to be fatal or to severely degrade performance, such as a confirmed hardware fault or thermal throttling. In these cases, waiting doesn’t protect your workload, because the workload is already failing or running on broken hardware. An immediate transition doesn’t wait for the Node to become idle. The Node leaves the production state and CKS begins evicting Pods right away. Pods labeled qos.coreweave.com/graceful-interruptible block the action until they terminate or their terminationGracePeriodSeconds expires; all other Pods are evicted without a waiting period, even if a pending transition would normally wait for them.

​
Active Nodes and stalled transitions

A pending transition proceeds only when the Node is no longer active. A Node is active when it runs at least one Pod that CKS counts toward the Node’s idle status. In summary:
  • DaemonSet Pods never count as active.
  • Pods with the qos.coreweave.cloud/interruptable label don’t count as active.
  • Pods with the qos.coreweave.com/graceful-interruptible label don’t count as active. Before the transition proceeds, CKS evicts these Pods and waits for their terminationGracePeriodSeconds to pass. See Pod interruption and eviction policies.
  • Control plane Pods don’t count as active.
  • Any other Pod in the Running phase counts as active.
The CWActive Node condition uses a broader rule for Node Pool scale-down. It also counts Pending Pods, and it counts gracefully interruptible Pods until they finish terminating. See Idle Nodes. As a result, a Node can show CWActive = True while a pending transition proceeds.
Crash-looping Pods stall pending transitions.A Pod in CrashLoopBackOff keeps its .status.phase set to Running, even though kubectl reports its status as CrashLoopBackOff. Because the phase stays Running, CKS still counts the Pod as active. If a Deployment keeps restarting a crashed Pod on a Node with a pending transition, the Node stays cordoned indefinitely: the hardware issue prevents the Pod from running, and the restarting Pod prevents the remediation. SUNK avoids this for its slurmd Pods: the SUNK Syncer sets the qos.coreweave.cloud/interruptable label to true whenever no Slurm job is running on the Pod, so an idle slurmd Pod doesn’t count as active. While a Slurm job runs on the Pod, the label is false and the Pod counts as active, so the transition waits for the job.If a Node in your cluster is stuck with a pending state and a crash-looping Pod, delete the Pod or apply an interruptible label so the transition can proceed.

​
Node states in a CKS cluster

Every Node carries a node.coreweave.cloud/state label with its current lifecycle state and, during a pending transition, a node.coreweave.cloud/pending-state label with the target state. The CoreWeave Intelligent CLI shows both in the STATE and PENDING STATE columns of cwic node get. See View Node details. The states you are most likely to observe on Nodes in your cluster are: Nodes also pass through Day 1 states such as Zap and Test before they reach production, but those states occur before a Node joins your cluster.
CoreWeave-managed Node conditions, labels, states, and events can change without notice. Don’t use them for your own automation or change them except through documented procedures. These include setting AdminTemporaryFailure to cordon a Node, requesting a Node reboot, and canceling a pending reboot. You can still add your own custom Node conditions and labels.

​
Example: a pending reboot

The following sequence shows a typical pending reboot, using a GPU reset request as the trigger:
  1. The NVIDIA driver reports that a GPU needs a reset, for example because of a pending memory row remap. CoreWeave monitoring sets the GPUWantsReset condition on the Node.
  2. The Node moves to the production-reboot pending state. The Node emits a CWNCPhaseNccPendingChange event that names the condition that triggered the transition, sets the node.coreweave.cloud/pending-state label, and updates the PendingPhaseState condition.
  3. CKS cordons the Node. Your existing workloads keep running, and the Node emits CWNCPhasePendingTenantWorkloadRunning events while they do.
  4. When the Node becomes inactive, it reboots inside your cluster.
  5. After the reboot, the Node returns to production and CKS uncordons it. An HPC Verification test runs after that, but the Node doesn’t wait for it. The test needs the Node’s GPUs free, so a workload scheduled right away can stop it or prevent it from starting. If the reboot doesn’t complete in time, the Node moves to production-reboot-fail.
To inspect a transition in progress, describe the Node and review its conditions and events:
Replace [NODE-NAME] with the name of your Node. To follow only the Node’s events as a transition progresses, watch them instead of repeatedly describing the Node:

​
Triage transitions

When CoreWeave determines that a Node needs investigation rather than a reboot, the Node moves toward the triage state:
  1. CoreWeave’s Node lifecycle automation sets a condition such as NeedsTriage on the Node. The Node receives a NoSchedule taint and is cordoned, so no new workloads schedule onto it.
  2. For a pending triage transition, existing workloads can continue while the Node waits to become idle. If the Node is in the prefill flow, CKS can start draining it when the prefill timeout expires, even if workloads are still running.
  3. CKS removes the Node from your cluster and returns it to CoreWeave’s fleet for remediation. CoreWeave re-validates the Node before it can return to any production fleet; a Node that needs the full onboarding test suite takes roughly 48 hours to return.
  4. CKS delivers a replacement Node to your Node Pool automatically. Prefill can deliver the replacement before the old Node leaves.
Node Pool prefill provisions the replacement before the old Node drains, so the Node Pool stays at its target capacity throughout. Without prefill, the replacement arrives after the old Node is removed, which can leave a capacity gap of 20 to 40 minutes.

​
Prepare your workloads for transitions

How quickly a transition completes depends on how your workloads signal their tolerance for interruption:
  • Stateless workloads: Apply the qos.coreweave.cloud/interruptable label so CKS excludes the Pods from the Node’s active status. Pending transitions proceed without waiting for these Pods. See Interruptible.
  • Stateful workloads that need cleanup time: Apply the qos.coreweave.com/graceful-interruptible label so Node actions wait for the Pod’s full terminationGracePeriodSeconds, which defaults to 30 seconds; set it longer than your workload’s expected cleanup time. See Gracefully interruptible and Pod termination in the Kubernetes documentation.
  • Critical jobs: Leave both labels off. CKS treats the Node as active and defers pending transitions until the job completes. This also defers remediation, as described in Long-running GPU Pods.
  • Capacity-sensitive Node Pools: Enable Node Pool prefill so replacements arrive before triaged Nodes drain.

​
See also

Last modified on October 8, 2026