Reboot Nodes - CoreWeave Docs

This guide explains how to manually reboot CKS Nodes, including how to choose between safe and immediate reboots and how to cancel a reboot that hasn’t started yet. You can manually reboot Nodes in two ways:

Standard Node reboot: A typical reboot to apply a system update or other change.
Reconfigure reboot: A reboot to apply an updated Node Pool configuration to existing Nodes.

This guide covers standard Node reboots. To reboot Nodes to apply Node Pool configuration updates, see Apply Node Pool Configuration Updates. Nodes are managed as part of CoreWeave’s Node lifecycle. The Node Life Cycle controller handles required reboots to ensure that customer workloads aren’t interrupted. To learn how CoreWeave-initiated reboots behave, see Node state transitions in CKS.

Reboot options

You can request a reboot in the following ways:

Method	Use it when
CoreWeave Intelligent CLI	You want the shortest command and built-in safety checks. This is the recommended method.
`kubectl`	You manage the cluster with `kubectl` and don’t want to install the CoreWeave Intelligent CLI.
Kubernetes API	You don’t have `kubectl` available, or you’re automating reboots from a system that already uses the Kubernetes API.

Safe versus immediate reboots

Before you request a reboot, choose the right reboot option. You can reboot a Node whether it’s active or idle.

Safe reboot: A safe reboot marks the Node to reboot after it becomes idle and all active workloads stop. New Pods might still be scheduled while the reboot waits.
Immediate reboot: An immediate reboot reboots the Node without waiting for active workloads to stop.

A Node is active when the CWActive condition is set to True, which means it runs at least one Pod that counts toward the Node’s idle status. DaemonSet Pods, interruptible Pods, and CoreWeave system Pods don’t count as active, so you don’t need to remove them before a reboot. To see which conditions CKS ignores when determining whether a Node is idle, see Idle Nodes. After the Node reboots, it remains in a reboot state while a short self-test runs. After the test passes, the Node returns to the production state and accepts customer workloads.

Prerequisites

Before you begin a Node reboot, ensure you have the following:

An active CoreWeave account.
An API Access Token.
kubectl installed locally, if you use the kubectl method.

Depending on the method you use to reboot Nodes, confirm the following:

CoreWeave Intelligent CLI
kubectl and Kubernetes API

Install the latest version of the CoreWeave Intelligent CLI.

You need permission to get Nodes and to patch the status subresource of Nodes. To cancel a pending reboot, you also need permission to patch Nodes. Verify your permissions:

kubectl auth can-i get nodes
kubectl auth can-i patch nodes --subresource=status
kubectl auth can-i patch nodes

Each command must return yes. You only need the patch nodes permission to cancel a pending reboot.

Check the Node before rebooting

Only reboot a Node that is in the production lifecycle state. Don’t manually request a reboot while a Node is transitioning between lifecycle states, because the reboot request can conflict with the transition that CoreWeave already has in progress. The examples in this guide use a NODE environment variable. Replace [NODE-NAME] with the name of the Node you want to reboot:

export NODE="[NODE-NAME]"

Check the Node lifecycle state

Run the following:

kubectl get node "${NODE}" \
  -o jsonpath='{.status.conditions[?(@.type=="PhaseState")].reason}{"\n"}'

Continue only if the command returns production:

Example output

production

Check for active workloads

Run the following:

kubectl get node "${NODE}" \
  -o jsonpath='{.status.conditions[?(@.type=="CWActive")].status}{"\n"}'

A value of True means the Node has active workloads, so an immediate reboot interrupts them. DaemonSet Pods, interruptible Pods, and CoreWeave system Pods aren’t counted as active, so you don’t need to remove them.If you have the CoreWeave Intelligent CLI, you can also check a Node’s active status with the cwic node get command. Read the ACTIVE column, which reflects CWActive. Don’t use the READY column, which reflects the Kubernetes Ready condition and is true on idle Nodes as well.

Request a Node reboot

Choose one of the following methods to request a reboot.

CoreWeave Intelligent CLI
kubectl
Kubernetes API

To use the CoreWeave Intelligent CLI to reboot a Node, run the following command, choosing a flag from the following list and replacing [NODE-NAMES] with a space-separated list of Nodes you want to reboot:

cwic node reboot [--safe|--force|--unset] [NODE-NAMES]

The --safe flag queues a safe reboot, and the --unset flag cancels a pending safe reboot. The --force flag, or its alias --yes, requests an immediate reboot without prompting.Without a flag, cwic node reboot requests an immediate reboot. If any of the Nodes are active, it prompts you with three options:

Example output

The following 2 nodes are currently Active (running non-preemptible workloads):
  - node-1
  - node-2

Options:
  [Y]es - Proceed with immediate reboot (may disrupt workloads)
  [S]afe - Switch to safe reboot (waits for workloads to terminate)
  [N]o - Cancel reboot

Choose an option [Y/s/n]:

Enter S to switch to a safe reboot, which waits for the active workloads to stop. Pressing Enter accepts the default, Y, and reboots the Nodes immediately.The CoreWeave Intelligent CLI refuses to reboot a Node that isn’t in the production state, so you don’t need to check the lifecycle state yourself.For example, to request an immediate reboot for the Nodes node-1 and node-2 without a confirmation prompt, run:

cwic node reboot --force node-1 node-2

Adding a reason for the reboot is optional, but recommended. To set a message for the reboot, use the --message flag. For example, run:

cwic node reboot --message "Reason for this reboot" node-1 node-2

Run cwic node reboot --help for more information.

You can request a reboot with kubectl. A reboot request is a patch to the Node’s status subresource that sets a lifecycle condition. Before you patch a Node, check the Node.

Always use a strategic merge patch. A JSON merge patch can replace the complete status.conditions array and remove conditions managed by Kubernetes or CoreWeave.

Request a safe reboot

A safe reboot sets the PendingPhaseState condition and waits for active workloads to stop before rebooting the Node:

kubectl patch node "${NODE}" \
  --type=strategic \
  --subresource=status \
  -p '{
    "status": {
      "conditions": [
        {
          "type": "PendingPhaseState",
          "status": "True",
          "reason": "production-powerreset",
          "message": "Customer-requested safe reboot"
        }
      ]
    }
  }'

The reboot remains pending until CKS determines that the Node can be rebooted.

Request an immediate reboot

An immediate reboot sets the PhaseState condition and starts without waiting for active workloads to stop:

kubectl patch node "${NODE}" \
  --type=strategic \
  --subresource=status \
  -p '{
    "status": {
      "conditions": [
        {
          "type": "PhaseState",
          "status": "True",
          "reason": "production-powerreset",
          "message": "Customer-requested immediate reboot"
        }
      ]
    }
  }'

An immediate reboot interrupts any workloads running on the Node.

If you don’t have kubectl available, call the Node status endpoint directly. A reboot request is a patch to the Node’s status subresource that sets a lifecycle condition. Before you patch a Node, check the Node.

Always use a strategic merge patch. A JSON merge patch can replace the complete status.conditions array and remove conditions managed by Kubernetes or CoreWeave.

Replace the following placeholders:

[APISERVER-URL]: The Kubernetes API server URL from your kubeconfig.
[API-ACCESS-TOKEN]: An API Access Token authorized to patch nodes/status.
[NODE-NAME]: The name of the Node you want to reboot.

export APISERVER="[APISERVER-URL]"
export API_ACCESS_TOKEN="[API-ACCESS-TOKEN]"
export NODE="[NODE-NAME]"

Request a safe reboot

curl --fail-with-body --request PATCH \
  "${APISERVER}/api/v1/nodes/${NODE}/status" \
  -H "Authorization: Bearer ${API_ACCESS_TOKEN}" \
  -H "Content-Type: application/strategic-merge-patch+json" \
  --data '{
    "status": {
      "conditions": [
        {
          "type": "PendingPhaseState",
          "status": "True",
          "reason": "production-powerreset",
          "message": "Customer-requested safe reboot"
        }
      ]
    }
  }'

Request an immediate reboot

curl --fail-with-body --request PATCH \
  "${APISERVER}/api/v1/nodes/${NODE}/status" \
  -H "Authorization: Bearer ${API_ACCESS_TOKEN}" \
  -H "Content-Type: application/strategic-merge-patch+json" \
  --data '{
    "status": {
      "conditions": [
        {
          "type": "PhaseState",
          "status": "True",
          "reason": "production-powerreset",
          "message": "Customer-requested immediate reboot"
        }
      ]
    }
  }'

Reboot request reference

Refer to the following table when rebooting with kubectl or the Kubernetes API:

Condition type	Status	Reason	Behavior
`PendingPhaseState`	`True`	`production-powerreset`	Waits for active workloads to stop, then reboots the Node.
`PhaseState`	`True`	`production-powerreset`	Reboots the Node immediately.

Note: The condition types reflect the current CKS implementation. CoreWeave lifecycle conditions are not a typed, versioned API and may change. We recommend using the CoreWeave Intelligent CLI for automation.

The condition type, status, and reason fields form the reboot request contract and must use the values shown in the preceding table. The message field is descriptive. Use it to record the source or purpose of the request.

Monitor the reboot

To watch the Node’s lifecycle and readiness conditions, run:

watch "kubectl get node \"${NODE}\" -o custom-columns='NAME:.metadata.name,STATE:.status.conditions[?(@.type==\"PhaseState\")].reason,PENDING:.status.conditions[?(@.type==\"PendingPhaseState\")].reason,READY:.status.conditions[?(@.type==\"Ready\")].status'"

During the reboot, the Node becomes unavailable and its readiness state changes. The reboot is complete when:

The State reason returns to production.
The Kubernetes Ready condition is True.
Your workloads can be scheduled or resumed as expected.

CKS doesn’t return the Node to customer workloads until its post-reboot checks complete.

Cancel a pending reboot

You can cancel a safe reboot while it’s still pending. A pending reboot has no timeout: it waits indefinitely until no workloads block the transition, so a pending request can remain in place for a long time. Cancel the request if you no longer want the Node to reboot. You can’t cancel a reboot after the Node starts rebooting, and you can’t cancel an immediate reboot, because an immediate reboot doesn’t wait before it starts. If a Node has already begun rebooting, wait for it to return to the production state. To confirm that the reboot is still pending, run:

kubectl get node "${NODE}" \
  -o jsonpath='{.status.conditions[?(@.type=="PendingPhaseState")].reason}{"\n"}'

If the command returns production-powerreset, the reboot is still pending and you can cancel it.

CoreWeave Intelligent CLI
kubectl

To cancel a pending safe reboot, run the following command, replacing [NODE-NAMES] with a space-separated list of Nodes:

cwic node reboot --unset [NODE-NAMES]

The --unset flag clears both the pending reboot condition and the pending state label.

Run the following commands to cancel a pending reboot with kubectl. You must complete both steps. If you clear the condition without clearing the label, the Node can retain a stale pending state.

Clear the pending reboot condition

kubectl patch node "${NODE}" \
  --type=strategic \
  --subresource=status \
  -p '{
    "status": {
      "conditions": [
        {
          "type": "PendingPhaseState",
          "status": "False",
          "reason": "none",
          "message": "Customer canceled the pending reboot"
        }
      ]
    }
  }'

Clear the pending state label

kubectl label node "${NODE}" \
  node.coreweave.cloud/pending-state=none --overwrite

Verify that the cancellation applied

kubectl get node "${NODE}" \
  -o custom-columns='NAME:.metadata.name,PENDING:.status.conditions[?(@.type=="PendingPhaseState")].reason,PENDING_LABEL:.metadata.labels.node\.coreweave\.cloud/pending-state'

Both PENDING and PENDING_LABEL must return none.

Troubleshooting

The API returns 403 Forbidden

Your Kubernetes identity doesn’t have permission to patch the Node status. If you’re canceling a pending reboot, you need permission to patch both the Node and its status subresource. To verify your permissions, run:

kubectl auth can-i patch nodes --subresource=status
kubectl auth can-i patch nodes

Ask your cluster administrator to grant the minimum required permission.

The Node doesn’t reboot

Check that:

The Node was in the production state when you submitted the request.
The patch was applied to the status subresource.
The condition type and reason use the values in Reboot request reference.
A safe reboot isn’t waiting for active workloads to stop.

To inspect the relevant conditions, run:

kubectl get node "${NODE}" \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

A canceled reboot still occurs

Check that both the condition and the label were cleared:

kubectl get node "${NODE}" \
  -o custom-columns='PENDING:.status.conditions[?(@.type=="PendingPhaseState")].reason,PENDING_LABEL:.metadata.labels.node\.coreweave\.cloud/pending-state'

If either value isn’t none, repeat the cancellation steps. A reboot that has already started can’t be canceled. Contact CoreWeave Support if the Node reboots after both values return none.

The Node stays in `production-powerreset`

If you add NoExecute taints, post-reboot HPC verification might not schedule. The Node can remain in production-powerreset, and CoreWeave might replace it. Use cwic node verify to trigger HPC verifications.

The Node doesn’t return to Ready

To collect the Node conditions and recent events, run:

kubectl describe node "${NODE}"

Contact CoreWeave Support if the PhaseState doesn’t return to production or the Node remains unavailable after the expected reboot interval. Include the following information:

The cluster name.
The Node name.
The time when you submitted the reboot request.
Whether you requested a safe or immediate reboot.
The output of kubectl describe node.

​Reboot options

​Safe versus immediate reboots

​Prerequisites

​Check the Node before rebooting

​Request a Node reboot

​Request a safe reboot

​Request an immediate reboot

​Request a safe reboot

​Request an immediate reboot

​Reboot request reference

​Monitor the reboot

​Cancel a pending reboot

​Troubleshooting

​The API returns 403 Forbidden

​The Node doesn’t reboot

​A canceled reboot still occurs

​The Node stays in production-powerreset

​The Node doesn’t return to Ready

Reboot options

Safe versus immediate reboots

Prerequisites

Check the Node before rebooting

Request a Node reboot

Request a safe reboot

Request an immediate reboot

Request a safe reboot

Request an immediate reboot

Reboot request reference

Monitor the reboot

Cancel a pending reboot

Troubleshooting

The API returns 403 Forbidden

The Node doesn’t reboot

A canceled reboot still occurs

The Node stays in `production-powerreset`

The Node doesn’t return to Ready