> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # Reboot Nodes > Request safe and immediate reboots of CKS Nodes with the CoreWeave Intelligent CLI, kubectl, or the Kubernetes API This guide explains how to manually reboot CKS Nodes, including how to choose between safe and immediate reboots and how to cancel a reboot that hasn't started yet. You can manually reboot Nodes in two ways: * **Standard Node reboot**: A typical reboot to apply a system update or other change. * **Reconfigure reboot**: A reboot to apply an updated [Node Pool configuration](/products/cks/nodes/manage#manage-node-pool-configuration) to existing Nodes. This guide covers standard Node reboots. To reboot Nodes to apply Node Pool configuration updates, see [Apply Node Pool Configuration Updates](/products/cks/nodes/apply-updates). Nodes are managed as part of CoreWeave's [Node lifecycle](/platform/fleet-management/node-lifecycle). Node lifecycle automation handles *required* reboots. For most faults it waits for your workloads to finish, but critical faults reboot the Node immediately. To learn how CoreWeave-initiated reboots behave, see [Node state transitions in CKS](/platform/fleet-management/node-lifecycle/state-transitions). ## Reboot options You can request a reboot in the following ways: | Method | Use it when | | - | - | | [CoreWeave Intelligent CLI](#request-a-node-reboot) | You want the shortest command and built-in safety checks. This is the recommended method. | | [`kubectl`](#request-a-node-reboot) | You manage the cluster with `kubectl` and don't want to install the CoreWeave Intelligent CLI. | | [Kubernetes API](#request-a-node-reboot) | You don't have `kubectl` available, or you're automating reboots from a system that already uses the Kubernetes API. | ### Safe versus immediate reboots Before you request a reboot, choose the right reboot option. You can reboot a Node whether it's active or idle. * **Safe reboot**: A safe reboot marks the Node to reboot *after* it becomes idle and all active workloads stop. CKS requests a cordon to prevent new workload scheduling while existing workloads finish. * **Immediate reboot**: An immediate reboot reboots the Node without waiting for active workloads to stop. It still waits for [gracefully interruptible](/products/cks/clusters/scheduling/workload-scheduling#gracefully-interruptible) Pods to terminate or reach their `terminationGracePeriodSeconds`. A Node is active when it runs at least one Pod that counts toward the Node's idle status. `DaemonSet` Pods, [interruptible Pods](/products/cks/clusters/scheduling/workload-scheduling#pod-interruption-and-eviction-policies), and CoreWeave system Pods don't count as active, so you don't need to remove them before a reboot. The `CWActive` condition is a broader indicator: it also counts `Pending` Pods and gracefully interruptible Pods until they finish terminating, so it can read `True` while a safe reboot proceeds. For the rule a safe reboot waits on, see [Active Nodes and stalled transitions](/platform/fleet-management/node-lifecycle/state-transitions#active-nodes-and-stalled-transitions). To see which conditions CKS ignores when determining whether a Node is idle, see [Idle Nodes](/products/cks/nodes/nodes-and-node-pools#idle-nodes). After the Node reboots and rejoins the cluster, it returns to the `production` state and accepts customer workloads. CoreWeave triggers an [HPC Verification](/platform/fleet-management/hpc-verification) test after the reboot, but the Node doesn't wait for the result. The test needs the Node's GPUs free, so a workload scheduled right away either stops a running test or prevents it from starting. ## Prerequisites Before you begin a Node reboot, ensure you have the following: * An active CoreWeave account. * An [API access token](/security/authn-authz/managed-auth/api-access). * [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed locally, if you use the `kubectl` method. Depending on the method you use to reboot Nodes, confirm the following: Install the latest version of the [CoreWeave Intelligent CLI](https://github.com/coreweave/cwic). You need permission to get Nodes and to patch the `status` subresource of Nodes. To cancel a pending reboot, you also need permission to patch Nodes. Verify your permissions: ```bash theme={"system"} kubectl auth can-i get nodes kubectl auth can-i patch nodes --subresource=status kubectl auth can-i patch nodes ``` Each command must return `yes`. You only need the `patch nodes` permission to cancel a pending reboot. ## Check the Node before rebooting Only reboot a Node that is in the `production` lifecycle state. Don't manually request a reboot while a Node is transitioning between lifecycle states, because the reboot request can conflict with the transition that CoreWeave already has in progress. The examples in this guide use a `NODE` environment variable. Replace `[NODE-NAME]` with the name of the Node you want to reboot: ```bash theme={"system"} export NODE="[NODE-NAME]" ``` Run the following: ```bash theme={"system"} kubectl get node "${NODE}" \ -o jsonpath='{.status.conditions[?(@.type=="PhaseState")].reason}{"\n"}' ``` Continue only if the command returns `production`: ```text title="Example output" theme={"system"} production ``` Run the following: ```bash theme={"system"} kubectl get node "${NODE}" \ -o jsonpath='{.status.conditions[?(@.type=="CWActive")].status}{"\n"}' ``` `CWActive` is CKS's activity indicator for Node Pool scale-down. It can count `Pending` Pods and gracefully interruptible Pods that are still terminating. Before you request an immediate reboot, check the Node's Pods. An immediate reboot can interrupt running workloads. For the workload types a safe reboot waits on, see [Active Nodes and stalled transitions](/platform/fleet-management/node-lifecycle/state-transitions#active-nodes-and-stalled-transitions). If you have the CoreWeave Intelligent CLI, you can also check a Node's active status with the `cwic node get` command. Read the `ACTIVE` column, which reflects `CWActive`. Don't use the `READY` column, which reflects the Kubernetes `Ready` condition and is `true` on idle Nodes as well. Inspect the Node's taints: ```bash theme={"system"} kubectl get node "${NODE}" -o jsonpath='{.spec.taints}{"\n"}' ``` If you find a customer-applied `NoExecute` taint, change its effect to `NoSchedule` before you reboot. Update the Node or Node Pool configuration that manages it. Leave CoreWeave-managed taints unchanged. Customer-applied `NoExecute` taints can block post-reboot health verification, which leaves the Node unverified after it returns to `production`. See [Customer taints and HPC Verification](/platform/fleet-management/hpc-verification#taints). ## Request a Node reboot Choose one of the following methods to request a reboot. To use the CoreWeave Intelligent CLI to reboot a Node, run the following command, choosing a flag from the following list and replacing `[NODE-NAMES]` with a space-separated list of Nodes you want to reboot: ```bash theme={"system"} cwic node reboot [--safe|--force|--unset] [NODE-NAMES] ``` The `--safe` flag queues a safe reboot, and the `--unset` flag cancels a pending safe reboot. The `--force` flag, or its alias `--yes`, requests an immediate reboot without prompting. Without a flag, `cwic node reboot` requests an immediate reboot. If any of the Nodes are active, it prompts you with three options: ```text title="Example output" theme={"system"} The following 2 nodes are currently Active (running non-preemptible workloads): - node-1 - node-2 Options: [Y]es - Proceed with immediate reboot (may disrupt workloads) [S]afe - Switch to safe reboot (waits for workloads to terminate) [N]o - Cancel reboot Choose an option [Y/s/n]: ``` Enter `S` to switch to a safe reboot, which waits for the active workloads to stop. Pressing Enter accepts the default, `Y`, and reboots the Nodes immediately. The CoreWeave Intelligent CLI refuses to reboot a Node that isn't in the `production` state, so you don't need to check the lifecycle state yourself. For example, to request an immediate reboot for the Nodes `node-1` and `node-2` without a confirmation prompt, run: ```bash theme={"system"} cwic node reboot --force node-1 node-2 ``` Adding a reason for the reboot is optional, but recommended. To set a message for the reboot, use the `--message` flag. For example, run: ```bash theme={"system"} cwic node reboot --message "Reason for this reboot" node-1 node-2 ``` Run `cwic node reboot --help` for more information. You can request a reboot with `kubectl`. A reboot request is a patch to the Node's `status` subresource that sets a lifecycle condition. Before you patch a Node, [check the Node](#check-the-node-before-rebooting). Always use a strategic merge patch. A JSON merge patch can replace the complete `status.conditions` array and remove conditions managed by Kubernetes or CoreWeave. ### Request a safe reboot A safe reboot sets the `PendingPhaseState` condition and waits for active workloads to stop before rebooting the Node: ```bash theme={"system"} kubectl patch node "${NODE}" \ --type=strategic \ --subresource=status \ -p '{ "status": { "conditions": [ { "type": "PendingPhaseState", "status": "True", "reason": "production-powerreset", "message": "Customer-requested safe reboot" } ] } }' ``` The reboot remains pending until CKS determines that the Node can be rebooted. ### Request an immediate reboot An immediate reboot sets the `PhaseState` condition and starts without waiting for active workloads to stop: ```bash theme={"system"} kubectl patch node "${NODE}" \ --type=strategic \ --subresource=status \ -p '{ "status": { "conditions": [ { "type": "PhaseState", "status": "True", "reason": "production-powerreset", "message": "Customer-requested immediate reboot" } ] } }' ``` An immediate reboot interrupts any workloads running on the Node. Only gracefully interruptible Pods delay it, until they terminate or their grace period expires. If you don't have `kubectl` available, call the Node `status` endpoint directly. A reboot request is a patch to the Node's `status` subresource that sets a lifecycle condition. Before you patch a Node, [check the Node](#check-the-node-before-rebooting). Always use a strategic merge patch. A JSON merge patch can replace the complete `status.conditions` array and remove conditions managed by Kubernetes or CoreWeave. Replace the following placeholders: * `[APISERVER-URL]`: The Kubernetes API server URL from your kubeconfig. * `[API-ACCESS-TOKEN]`: An [API access token](/security/authn-authz/managed-auth/api-access) authorized to patch `nodes/status`. * `[NODE-NAME]`: The name of the Node you want to reboot. ```bash theme={"system"} export APISERVER="[APISERVER-URL]" export API_ACCESS_TOKEN="[API-ACCESS-TOKEN]" export NODE="[NODE-NAME]" ``` ### Request a safe reboot ```bash theme={"system"} curl --fail-with-body --request PATCH \ "${APISERVER}/api/v1/nodes/${NODE}/status" \ -H "Authorization: Bearer ${API_ACCESS_TOKEN}" \ -H "Content-Type: application/strategic-merge-patch+json" \ --data '{ "status": { "conditions": [ { "type": "PendingPhaseState", "status": "True", "reason": "production-powerreset", "message": "Customer-requested safe reboot" } ] } }' ``` ### Request an immediate reboot ```bash theme={"system"} curl --fail-with-body --request PATCH \ "${APISERVER}/api/v1/nodes/${NODE}/status" \ -H "Authorization: Bearer ${API_ACCESS_TOKEN}" \ -H "Content-Type: application/strategic-merge-patch+json" \ --data '{ "status": { "conditions": [ { "type": "PhaseState", "status": "True", "reason": "production-powerreset", "message": "Customer-requested immediate reboot" } ] } }' ``` ### Reboot request reference Refer to the following table when rebooting with `kubectl` or the Kubernetes API: | Condition type | Status | Reason | Behavior | | - | - | - | - | | `PendingPhaseState` | `True` | `production-powerreset` | Waits for active workloads to stop, then reboots the Node. | | `PhaseState` | `True` | `production-powerreset` | Reboots the Node immediately. | **Note**: The condition types reflect the current CKS implementation. CoreWeave lifecycle conditions are not a typed, versioned API and may change. We recommend using the CoreWeave Intelligent CLI for automation. The condition `type`, `status`, and `reason` fields form the reboot request contract and must use the values shown in the preceding table. The `message` field is descriptive. Use it to record the source or purpose of the request. ## Monitor the reboot To watch the Node's lifecycle and readiness conditions, run: ```bash theme={"system"} watch "kubectl get node \"${NODE}\" -o custom-columns='NAME:.metadata.name,STATE:.status.conditions[?(@.type==\"PhaseState\")].reason,PENDING:.status.conditions[?(@.type==\"PendingPhaseState\")].reason,READY:.status.conditions[?(@.type==\"Ready\")].status'" ``` During the reboot, the Node becomes unavailable and its readiness state changes. The reboot is complete when: * The `State` reason returns to `production`. * The Kubernetes `Ready` condition is `True`. * Your workloads can be scheduled or resumed as expected. The Node returns to `production` as soon as the reboot completes. The post-reboot [HPC Verification](/platform/fleet-management/hpc-verification) test runs after that, so let it finish before scheduling workloads onto the Node. The test needs the Node's GPUs free: a workload that lands first prevents it from starting, and one that lands mid-test stops it. ## Cancel a pending reboot You can cancel a safe reboot while it's still pending. A pending reboot has no timeout: it waits indefinitely until no workloads block the transition, so a pending request can remain in place for a long time. Cancel the request if you no longer want the Node to reboot. You can't cancel a reboot after the Node starts rebooting, and you can't cancel an immediate reboot, because an immediate reboot doesn't wait before it starts. If a Node has already begun rebooting, wait for it to return to the `production` state. To confirm that the reboot is still pending, run: ```bash theme={"system"} kubectl get node "${NODE}" \ -o jsonpath='{.status.conditions[?(@.type=="PendingPhaseState")].reason}{"\n"}' ``` If the command returns `production-powerreset`, the reboot is still pending and you can cancel it. To cancel a pending safe reboot, run the following command, replacing `[NODE-NAMES]` with a space-separated list of Nodes: ```bash theme={"system"} cwic node reboot --unset [NODE-NAMES] ``` The `--unset` flag clears both the pending reboot condition and the pending state label. Run the following commands to cancel a pending reboot with `kubectl`. You must complete both steps. If you clear the condition without clearing the label, the Node can retain a stale pending state. ```bash theme={"system"} kubectl patch node "${NODE}" \ --type=strategic \ --subresource=status \ -p '{ "status": { "conditions": [ { "type": "PendingPhaseState", "status": "False", "reason": "none", "message": "Customer canceled the pending reboot" } ] } }' ``` ```bash theme={"system"} kubectl label node "${NODE}" \ node.coreweave.cloud/pending-state=none --overwrite ``` ```bash theme={"system"} kubectl get node "${NODE}" \ -o custom-columns='NAME:.metadata.name,PENDING:.status.conditions[?(@.type=="PendingPhaseState")].reason,PENDING_LABEL:.metadata.labels.node\.coreweave\.cloud/pending-state' ``` Both `PENDING` and `PENDING_LABEL` must return `none`. ## Troubleshooting ### The API returns 403 Forbidden Your Kubernetes identity doesn't have permission to patch the Node status. If you're canceling a pending reboot, you need permission to patch both the Node and its `status` subresource. To verify your permissions, run: ```bash theme={"system"} kubectl auth can-i patch nodes --subresource=status kubectl auth can-i patch nodes ``` Ask your cluster administrator to grant the minimum required permission. ### The Node doesn't reboot Check that: * The Node was in the `production` state when you submitted the request. * The patch was applied to the `status` subresource. * The condition type and reason use the values in [Reboot request reference](#reboot-request-reference). * A safe reboot isn't waiting for active workloads to stop. To inspect the relevant conditions, run: ```bash theme={"system"} kubectl get node "${NODE}" \ -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` ### A canceled reboot still occurs Check that both the condition and the label were cleared: ```bash theme={"system"} kubectl get node "${NODE}" \ -o custom-columns='PENDING:.status.conditions[?(@.type=="PendingPhaseState")].reason,PENDING_LABEL:.metadata.labels.node\.coreweave\.cloud/pending-state' ``` If either value isn't `none`, repeat the cancellation steps. A reboot that has already started can't be canceled. [Contact CoreWeave Support](/support) if the Node reboots after both values return `none`. ### The Node stays in `production-powerreset` Check for customer-applied `NoExecute` taints on the Node: ```bash theme={"system"} kubectl get node "${NODE}" -o jsonpath='{.spec.taints}{"\n"}' ``` HPC Verification Pods don't tolerate `NoExecute` taints. These taints can prevent verification Pods from scheduling or evict them after they start, which leaves the Node unverified rather than holding it in a reboot state. CoreWeave might delete and replace a Node that remains in `production-powerreset` or `production-reboot` too long. Change taints to `NoSchedule` in the Node or Node Pool configuration that manages them. Verification Pods tolerate `NoSchedule`, so checks can run while Pods without matching tolerations stay off the Node. Leave CoreWeave-managed taints unchanged. To request verification on an idle Node that has returned to `production`, follow [Verify a Node on demand](/platform/fleet-management/hpc-verification#verify-a-node-on-demand). Allow verification to finish before scheduling GPU workloads. If verification still can't run, contact CoreWeave Support. ### The Node doesn't return to Ready To collect the Node conditions and recent events, run: ```bash theme={"system"} kubectl describe node "${NODE}" ``` [Contact CoreWeave Support](/support) if the `PhaseState` doesn't return to `production` or the Node remains unavailable after the expected reboot interval. Include the following information: * The cluster name. * The Node name. * The time when you submitted the reboot request. * Whether you requested a safe or immediate reboot. * The output of `kubectl describe node`.