> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Node state transitions in CKS

> How CoreWeave automation cordons, reboots, and replaces Nodes inside a CKS cluster

When CoreWeave's [Day 2+ monitoring](/platform/fleet-management/node-lifecycle/day2) detects an issue on a production Node, automation remediates it by moving the Node through a lifecycle state transition. That transition might be a reboot inside your cluster or a move to triage for investigation. This page explains how those transitions behave in a CoreWeave Kubernetes Service (CKS) cluster: the difference between pending and immediate transitions, the labels and conditions that mark them, the states you can observe, and how your workloads affect transition timing.

For the full lifecycle from initialization through production, see [Node lifecycle](/platform/fleet-management/node-lifecycle).

## Remediation categories

CoreWeave's health signals fall into two categories:

* **Solved by reboot**: The Node reboots inside your cluster, passes a verification test, and returns to service. Examples include GPU reset requests and kernel deadlocks.
* **Needs triage**: The Node leaves your cluster for investigation, repair, preventative maintenance, or vendor return. CKS delivers a replacement Node automatically.

For either category, CoreWeave matches the type of transition, pending or immediate, to the known severity of the issue.

## Pending state transitions

CoreWeave uses a pending transition when the issue's severity is uncertain and interrupting a running workload would be worse than waiting. A pending transition works as follows:

1. CKS cordons the Node so no new workloads schedule onto it, and records the reason in the `node.coreweave.cloud/cordonReason` annotation. See [Node cordoning](/products/cks/nodes/cordon).
2. CKS sets the `node.coreweave.cloud/pending-state` label on the Node to the target state, such as `production-reboot`, and updates the `PendingPhaseState` Node condition. The condition's `reason` names the target state, and its `message` describes what triggered the transition.
3. The transition waits until the Node is no longer [active](#active-nodes-and-stalled-transitions). While your workloads are still running, the Node emits `CWNCPhasePendingTenantWorkloadRunning` events that name the Pods preventing the transition.
4. When the Node becomes inactive, the state transition proceeds and the action takes place.

Existing workloads keep running during the pending period. This design assumes that a serious fault crashes the workload and takes its Pod down with it. The Pod exits, the Node becomes inactive, and the transition proceeds without interrupting anything that was still healthy.

## Immediate state transitions

CoreWeave uses an immediate transition when the issue is known to be fatal or to severely degrade performance, such as a confirmed hardware fault or thermal throttling. In these cases, waiting doesn't protect your workload, because the workload is already failing or running on broken hardware.

An immediate transition doesn't wait for the Node to become idle. The Node leaves the `production` state and CKS begins evicting Pods right away. Pods labeled [`qos.coreweave.com/graceful-interruptible`](/products/cks/clusters/scheduling/workload-scheduling#gracefully-interruptible) block the action until they terminate or their `terminationGracePeriodSeconds` expires; all other Pods are evicted without a waiting period, even if a pending transition would normally wait for them.

## Active Nodes and stalled transitions

A pending transition proceeds only when the Node is no longer active. A Node is active when it runs at least one Pod that CKS counts toward the Node's idle status. In summary:

* `DaemonSet` Pods never count as active.
* Pods with an [interruptible label](/products/cks/clusters/scheduling/workload-scheduling#pod-interruption-and-eviction-policies) don't count as active.
* Control plane Pods don't count as active.
* Any other Pod in the `Running` phase counts as active.

For the complete rules, see [Idle Nodes](/products/cks/nodes/nodes-and-node-pools#idle-nodes). You can check a Node's active status through the `CWActive` condition: `CWActive = False` means the Node is idle.

<Warning>
  **Crash-looping Pods stall pending transitions.**

  A Pod in `CrashLoopBackOff` keeps its `.status.phase` set to `Running`, even though `kubectl` reports its status as `CrashLoopBackOff`. Because the phase stays `Running`, CKS still counts the Pod as active. If a Deployment keeps restarting a crashed Pod on a Node with a pending transition, the Node stays cordoned indefinitely: the hardware issue prevents the Pod from running, and the restarting Pod prevents the remediation. SUNK avoids this automatically: it labels its Slurm job Pods with the [`qos.coreweave.com/graceful-interruptible`](/products/cks/clusters/scheduling/workload-scheduling#gracefully-interruptible) label, so they don't count toward the Node's active status and don't stall the transition.

  If a Node in your cluster is stuck with a pending state and a crash-looping Pod, delete the Pod or apply an [interruptible label](/products/cks/clusters/scheduling/workload-scheduling#pod-interruption-and-eviction-policies) so the transition can proceed.
</Warning>

## Node states in a CKS cluster

Every Node carries a `node.coreweave.cloud/state` label with its current lifecycle state and, during a pending transition, a `node.coreweave.cloud/pending-state` label with the target state. The [CoreWeave Intelligent CLI](https://github.com/coreweave/cwic) shows both in the `STATE` and `PENDING STATE` columns of `cwic node get`. See [View Node details](/products/cks/nodes/manage#view-node-details).

The states you are most likely to observe on Nodes in your cluster are:

| State                       | Description                                                                                                                                                                                                                                                                                                   |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `production`                | The Node is in service and accepts workloads. Healthy Nodes show this state with no pending state.                                                                                                                                                                                                            |
| `production-reboot`         | The Node reboots inside your cluster, then runs a verification test before accepting workloads again. If the test fails, the Node moves to `triage`.                                                                                                                                                          |
| `production-collect-reboot` | The same as `production-reboot`, but CoreWeave collects diagnostic data from the Node before rebooting it.                                                                                                                                                                                                    |
| `production-powercycle`     | A variant of the reboot flow that fully power cycles the Node, including its Data Processing Unit (DPU), instead of rebooting it.                                                                                                                                                                             |
| `production-powerreset`     | A variant of the reboot flow that power resets the Node's server while its DPU stays powered on.                                                                                                                                                                                                              |
| `triage`                    | CoreWeave has sidelined the Node for investigation or repair. The Node leaves your cluster once it is idle, and CKS delivers a replacement.                                                                                                                                                                   |
| `production-triage`         | A variant of `triage` where CoreWeave investigates the Node without first returning it to CoreWeave's fleet. Rarely used in CKS: it applies only when unhealthy Node eviction is disabled for the cluster, or in special cases, because it keeps an unusable Node in your Node Pool during the investigation. |

Nodes also pass through [Day 1 states](/platform/fleet-management/node-lifecycle/day1) such as Zap and Test before they reach production, but those states occur before a Node joins your cluster.

<Note>
  Don't use CoreWeave Node conditions, labels, states, or events for your own automation. They are intended for internal use and can change without notice. You can still add your own custom Node conditions and labels.
</Note>

## Example: a pending reboot

The following sequence shows a typical pending reboot, using a GPU reset request as the trigger:

1. The NVIDIA driver reports that a GPU needs a reset, for example because of a pending memory row remap. CoreWeave monitoring sets the `GPUWantsReset` condition on the Node.
2. The Node moves to the `production-reboot` pending state. The Node emits a `CWNCPhaseNccPendingChange` event that names the condition that triggered the transition, sets the `node.coreweave.cloud/pending-state` label, and updates the `PendingPhaseState` condition.
3. CKS cordons the Node. Your existing workloads keep running, and the Node emits `CWNCPhasePendingTenantWorkloadRunning` events while they do.
4. When the Node becomes inactive, it reboots inside your cluster.
5. After the reboot, a verification test runs before the Node accepts workloads again. If the test passes, the Node returns to `production` and is uncordoned. If it fails, the Node moves to `triage` and CKS delivers a replacement.

To inspect a transition in progress, describe the Node and review its conditions and events:

```bash theme={"system"}
kubectl describe node [NODE-NAME]
```

Replace `[NODE-NAME]` with the name of your Node.

To follow only the Node's events as a transition progresses, watch them instead of repeatedly describing the Node:

```bash theme={"system"}
kubectl events --for=node/[NODE-NAME] --watch
```

## Triage transitions

When CoreWeave determines that a Node needs investigation rather than a reboot, the Node moves toward the `triage` state:

1. CoreWeave health monitoring sets a condition such as `NeedsTriage` on the Node. The Node receives a `NoSchedule` taint and is cordoned, so no new workloads schedule onto it.
2. For a pending triage transition, existing workloads keep running until the Node is idle. You can move or finish your workloads on your own schedule.
3. Once the Node is idle, it leaves your cluster and returns to CoreWeave's fleet for remediation. CoreWeave re-validates the Node before it can return to any production fleet; a Node that needs the full onboarding test suite takes roughly 48 hours to return.
4. CKS delivers a replacement Node to your Node Pool automatically.

[Node Pool prefill](/products/cks/nodes/nodes-and-node-pools#node-pool-prefill) provisions the replacement before the old Node drains, so the Node Pool stays at its target capacity throughout. Without prefill, the replacement arrives after the old Node is removed, which can leave a capacity gap of 20 to 40 minutes.

## Prepare your workloads for transitions

How quickly a transition completes depends on how your workloads signal their tolerance for interruption:

* **Stateless workloads**: Apply the `qos.coreweave.cloud/interruptable` label so CKS excludes the Pods from the Node's active status. Pending transitions proceed without waiting for these Pods. See [Interruptible](/products/cks/clusters/scheduling/workload-scheduling#interruptible).
* **Stateful workloads that need cleanup time**: Apply the `qos.coreweave.com/graceful-interruptible` label so Node actions wait for the Pod's full `terminationGracePeriodSeconds`, which defaults to 30 seconds; set it longer than your workload's expected cleanup time. See [Gracefully interruptible](/products/cks/clusters/scheduling/workload-scheduling#gracefully-interruptible) and [Pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) in the Kubernetes documentation.
* **Critical jobs**: Leave both labels off. CKS treats the Node as active and defers pending transitions until the job completes. This also defers remediation, as described in [Long-running GPU Pods](/products/cks/clusters/scheduling/workload-scheduling#long-running-gpu-pods).
* **Capacity-sensitive Node Pools**: Enable [Node Pool prefill](/products/cks/nodes/nodes-and-node-pools#node-pool-prefill) so replacements arrive before triaged Nodes drain.

## See also

* [Node lifecycle](/platform/fleet-management/node-lifecycle): The full lifecycle from Day 0 through Day 2+.
* [Node cordoning](/products/cks/nodes/cordon): Common cordon reasons and how to respond.
* [Workload scheduling on CKS](/products/cks/clusters/scheduling/workload-scheduling): Eviction strategies, labels, and taints.
* [Reboot Nodes](/products/cks/nodes/reboot): Request safe or force reboots manually.
* [Node Pool reference](/products/cks/reference/node-pool): Node conditions and events set by CKS.