Skip to main content
When CoreWeave’s Day 2+ monitoring detects an issue on a production Node, automation remediates it by moving the Node through a lifecycle state transition. That transition might be a reboot inside your cluster or a move to triage for investigation. This page explains how those transitions behave in a CoreWeave Kubernetes Service (CKS) cluster: the difference between pending and immediate transitions, the labels and conditions that mark them, the states you can observe, and how your workloads affect transition timing. For the full lifecycle from initialization through production, see Node lifecycle.

​
Remediation categories

CoreWeave’s health signals fall into two categories:
  • Solved by reboot: The Node reboots inside your cluster, passes a verification test, and returns to service. Examples include GPU reset requests and kernel deadlocks.
  • Needs triage: The Node leaves your cluster for investigation, repair, preventative maintenance, or vendor return. CKS delivers a replacement Node automatically.
For either category, CoreWeave matches the type of transition, pending or immediate, to the known severity of the issue.

​
Pending state transitions

CoreWeave uses a pending transition when the issue’s severity is uncertain and interrupting a running workload would be worse than waiting. A pending transition works as follows:
  1. CKS cordons the Node so no new workloads schedule onto it, and records the reason in the node.coreweave.cloud/cordonReason annotation. See Node cordoning.
  2. CKS sets the node.coreweave.cloud/pending-state label on the Node to the target state, such as production-reboot, and updates the PendingPhaseState Node condition. The condition’s reason names the target state, and its message describes what triggered the transition.
  3. The transition waits until the Node is no longer active. While your workloads are still running, the Node emits CWNCPhasePendingTenantWorkloadRunning events that name the Pods preventing the transition.
  4. When the Node becomes inactive, the state transition proceeds and the action takes place.
Existing workloads keep running during the pending period. This design assumes that a serious fault crashes the workload and takes its Pod down with it. The Pod exits, the Node becomes inactive, and the transition proceeds without interrupting anything that was still healthy.

​
Immediate state transitions

CoreWeave uses an immediate transition when the issue is known to be fatal or to severely degrade performance, such as a confirmed hardware fault or thermal throttling. In these cases, waiting doesn’t protect your workload, because the workload is already failing or running on broken hardware. An immediate transition doesn’t wait for the Node to become idle. The Node leaves the production state and CKS begins evicting Pods right away. Pods labeled qos.coreweave.com/graceful-interruptible block the action until they terminate or their terminationGracePeriodSeconds expires; all other Pods are evicted without a waiting period, even if a pending transition would normally wait for them.

​
Active Nodes and stalled transitions

A pending transition proceeds only when the Node is no longer active. A Node is active when it runs at least one Pod that CKS counts toward the Node’s idle status. In summary:
  • DaemonSet Pods never count as active.
  • Pods with an interruptible label don’t count as active.
  • Control plane Pods don’t count as active.
  • Any other Pod in the Running phase counts as active.
For the complete rules, see Idle Nodes. You can check a Node’s active status through the CWActive condition: CWActive = False means the Node is idle.
Crash-looping Pods stall pending transitions.A Pod in CrashLoopBackOff keeps its .status.phase set to Running, even though kubectl reports its status as CrashLoopBackOff. Because the phase stays Running, CKS still counts the Pod as active. If a Deployment keeps restarting a crashed Pod on a Node with a pending transition, the Node stays cordoned indefinitely: the hardware issue prevents the Pod from running, and the restarting Pod prevents the remediation. SUNK avoids this automatically: it labels its Slurm job Pods with the qos.coreweave.com/graceful-interruptible label, so they don’t count toward the Node’s active status and don’t stall the transition.If a Node in your cluster is stuck with a pending state and a crash-looping Pod, delete the Pod or apply an interruptible label so the transition can proceed.

​
Node states in a CKS cluster

Every Node carries a node.coreweave.cloud/state label with its current lifecycle state and, during a pending transition, a node.coreweave.cloud/pending-state label with the target state. The CoreWeave Intelligent CLI shows both in the STATE and PENDING STATE columns of cwic node get. See View Node details. The states you are most likely to observe on Nodes in your cluster are: Nodes also pass through Day 1 states such as Zap and Test before they reach production, but those states occur before a Node joins your cluster.
Don’t use CoreWeave Node conditions, labels, states, or events for your own automation. They are intended for internal use and can change without notice. You can still add your own custom Node conditions and labels.

​
Example: a pending reboot

The following sequence shows a typical pending reboot, using a GPU reset request as the trigger:
  1. The NVIDIA driver reports that a GPU needs a reset, for example because of a pending memory row remap. CoreWeave monitoring sets the GPUWantsReset condition on the Node.
  2. The Node moves to the production-reboot pending state. The Node emits a CWNCPhaseNccPendingChange event that names the condition that triggered the transition, sets the node.coreweave.cloud/pending-state label, and updates the PendingPhaseState condition.
  3. CKS cordons the Node. Your existing workloads keep running, and the Node emits CWNCPhasePendingTenantWorkloadRunning events while they do.
  4. When the Node becomes inactive, it reboots inside your cluster.
  5. After the reboot, a verification test runs before the Node accepts workloads again. If the test passes, the Node returns to production and is uncordoned. If it fails, the Node moves to triage and CKS delivers a replacement.
To inspect a transition in progress, describe the Node and review its conditions and events:
Replace [NODE-NAME] with the name of your Node. To follow only the Node’s events as a transition progresses, watch them instead of repeatedly describing the Node:

​
Triage transitions

When CoreWeave determines that a Node needs investigation rather than a reboot, the Node moves toward the triage state:
  1. CoreWeave health monitoring sets a condition such as NeedsTriage on the Node. The Node receives a NoSchedule taint and is cordoned, so no new workloads schedule onto it.
  2. For a pending triage transition, existing workloads keep running until the Node is idle. You can move or finish your workloads on your own schedule.
  3. Once the Node is idle, it leaves your cluster and returns to CoreWeave’s fleet for remediation. CoreWeave re-validates the Node before it can return to any production fleet; a Node that needs the full onboarding test suite takes roughly 48 hours to return.
  4. CKS delivers a replacement Node to your Node Pool automatically.
Node Pool prefill provisions the replacement before the old Node drains, so the Node Pool stays at its target capacity throughout. Without prefill, the replacement arrives after the old Node is removed, which can leave a capacity gap of 20 to 40 minutes.

​
Prepare your workloads for transitions

How quickly a transition completes depends on how your workloads signal their tolerance for interruption:
  • Stateless workloads: Apply the qos.coreweave.cloud/interruptable label so CKS excludes the Pods from the Node’s active status. Pending transitions proceed without waiting for these Pods. See Interruptible.
  • Stateful workloads that need cleanup time: Apply the qos.coreweave.com/graceful-interruptible label so Node actions wait for the Pod’s full terminationGracePeriodSeconds, which defaults to 30 seconds; set it longer than your workload’s expected cleanup time. See Gracefully interruptible and Pod termination in the Kubernetes documentation.
  • Critical jobs: Leave both labels off. CKS treats the Node as active and defers pending transitions until the job completes. This also defers remediation, as described in Long-running GPU Pods.
  • Capacity-sensitive Node Pools: Enable Node Pool prefill so replacements arrive before triaged Nodes drain.

​
See also

Last modified on July 22, 2026