What is Node cordoning?
In Kubernetes, a cordoned Node prevents new workloads from scheduling onto it. Existing workloads on the Node continue to run until you explicitly drain them or they complete. Cordoning typically indicates maintenance, troubleshooting, or reliability actions. Both you and CoreWeave can cordon Nodes. You can cordon a Node to isolate it while you troubleshoot or perform your own maintenance. CoreWeave cordons Nodes proactively to ensure reliability, maintain cluster health, and manage maintenance activities with minimal impact on workloads.Cordon a Node yourself
Choose the cordon method that matches how long the isolation needs to last:- Use
cwic node cordonwhen you want the cordon integrated with CoreWeave tooling and recorded in the cordon reason annotation. This is the preferred method. - Use
kubectl cordonfor quick, short-lived cordons. - Use a custom taint when isolation must survive Node reboots. A taint applied directly to a Node doesn’t carry over to a replacement Node.
Cordon with the CoreWeave Intelligent CLI
The preferred way to cordon a Node is the CoreWeave Intelligent CLInode cordon command. To install and authenticate the CLI, see Manage Node Pools.
[NODE-NAME] with the name of your Node. You can pass multiple Node names separated by spaces. To record why you cordoned the Node, add a reason and a message:
[REASON] with a brief reason and [MESSAGE] with a longer description. Both are recorded on the Node’s AdminTemporaryFailure condition. If you can’t use the CLI, you can set the condition with kubectl patch instead.
cwic node cordon sets the AdminTemporaryFailure Node condition, which CoreWeave automation recognizes as an intentional manual cordon and leaves in place. The Node’s node.coreweave.cloud/cordonReason annotation reports AdminTemporaryFailure, so anyone who inspects the Node can see that it was cordoned manually.
To clear your manual cordon, run:
The
AdminTemporaryFailure condition clears when the Node reboots. The Node can remain cordoned for other reasons. If your scheduling restriction must survive reboots, use a custom taint.Cordon with kubectl
You can also use a standard Kubernetes cordon:
kubectl cordon sets only the Node’s unschedulable flag, with no CoreWeave-recognized reason behind it. CoreWeave automation manages this flag during Node lifecycle operations, so it can clear your cordon. For example, automation uncordons a Node when its CoreWeave-recognized cordon conditions clear, and a plain cordon doesn’t survive a Node reboot because the Node uncordons itself when it boots. On clusters with strict cordon reconciliation enabled, automation also reverts manual cordons that aren’t backed by a recognized condition. When the cordon matters for more than a brief period, prefer cwic node cordon or a custom taint.
To remove the cordon, run:
Isolate a Node long-term with a custom taint
For long-term isolation, apply a custom taint with theNoSchedule effect instead of a cordon:
[TAINT-KEY] with a key you own, such as example.com/maintenance-hold. Keep your custom key outside the *.coreweave.cloud and *.coreweave.com domains to avoid conflicting with CoreWeave-managed taints.
CoreWeave automation preserves a custom taint that you apply directly to the Node if its key and effect aren’t managed through the Node Pool. Use a dedicated key that isn’t also configured in the Node Pool’s spec.nodeTaints. The taint persists across reboots while the same Kubernetes Node object exists. If CoreWeave replaces the Node or deletes and recreates its Node object during remediation, the taint doesn’t carry over. Apply the taint to the replacement Node if it also needs isolation.
To remove the taint from the current Node, run:
NoSchedule taint prevents new Pods from scheduling onto the Node without evicting the Pods already running there. Pods that tolerate the taint can still schedule onto the Node. A taint by itself doesn’t cause SchedulingDisabled to appear in kubectl get nodes output the way a cordon does. Check a Node’s taints with kubectl describe node.
Reasons CoreWeave cordons Nodes
The following sections describe the reasons CoreWeave might cordon Nodes.Maintenance and updates
CoreWeave cordons Nodes that require hardware maintenance, kernel updates, or hardware replacements to avoid scheduling new workloads. This practice allows maintenance activities without disrupting running workloads.Health monitoring
CoreWeave monitors Node health and performance. If a Node exhibits issues that could affect workload reliability, CoreWeave may cordon it to prevent new workloads from being scheduled. These issues can include:- Hardware issues: GPU hardware errors trigger automatic Node cordoning and rebooting, or an immediate move to triage for faults such as a GPU falling off the PCI bus.
- Kernel deadlocks: Severe kernel issues may also cause Nodes to reboot and remain cordoned until resolved.
- InfiniBand or Ethernet link flaps: CoreWeave cordons Nodes with unstable network connections (intermittent connectivity) so workloads aren’t placed on unreliable Nodes. This detection looks for instability, so a link that fails cleanly and stays down doesn’t cordon the Node. If a Node keeps running but delivers less network bandwidth than you expect, contact support with the Node name.
- Connectivity failures: If Kubernetes detects critical network issues, CoreWeave cordons the affected Nodes until stability is restored.
Time synchronization
CKS Nodes synchronize time to CoreWeave’s internal time infrastructure. CoreWeave monitors time on all Nodes, and a Node whose clock drifts beyond acceptable bounds may be cordoned for triage. Running any competing time daemon (includingchrony, ntpd, systemd-timesyncd) directly on a Node or in containers may cause clock drift and trigger this behaviour. Overriding the time servers on CKS Nodes is not supported. Containers with time daemons must be configured to disable the daemon and/or prevent the daemon from altering the kernel clock.
Temporary issues
Temporary issues detected through Kubernetes health checks can lead to short-term Node cordoning. These usually resolve automatically, and Nodes become schedulable again without intervention.Scheduled maintenance
CoreWeave intentionally cordons Nodes during scheduled maintenance.Lifecycle management
CoreWeave manages Node lifecycle events, such as provisioning, decommissioning, reboots, and upgrades. Temporary cordoning may occur when Nodes transition between different operational states to maintain cluster stability and reliability during lifecycle transitions. To understand how these transitions behave inside your cluster, see Node state transitions in CKS.Check why a Node is cordoned
When CoreWeave cordons a Node, it records the reason in thenode.coreweave.cloud/cordonReason annotation. Check this annotation to understand whether the cordon is an expected maintenance action or a possible sign of poor Node health.
Read the cordon reason for a Node:
[NODE-NAME] with the name of your Node.
For the Node’s full status, including whether it is unschedulable and its current conditions, use describe:
NLCCPendingExitProduction, the matching Node condition message names the state the Node is moving to and what initiated the transition. A username identifies a CoreWeave engineer who started it by hand, while a message such as Reboot from Slurm identifies automation:
Example condition
Common cordon and drain reasons
The following table lists the cordon reasons you are most likely to see on a CKS Node, what each one means, and whether it represents an expected maintenance action or a possible health issue.Health and lifecycle cordon reasons
CoreWeave’s Node lifecycle automation sets the following cordon reasons from Node conditions. Each reason names the fault, and the last column describes what CoreWeave’s automation does next: keep the Node cordoned until the condition clears, reboot it, or move it totriage. A reboot waits for the Node to become idle unless the row says it’s immediate. For what each lifecycle state means, see Node states in a CKS cluster.
On a SUNK cluster, these reasons also appear as Slurm drain reasons in the form k8s: cordon: [CORDON-REASON]. See Kubernetes failure cases.
Set AdminTemporaryFailure on a Node
Always patch Node conditions with--type=strategic. A merge patch replaces the whole status.conditions array, which deletes every other condition on the Node, including Ready. CoreWeave automation then misreads the Node’s state, and your Pods can be terminated.
Use the following command to set AdminTemporaryFailure on a Node:
Set AdminTemporaryFailure on a Node
What a cordon affects
A cordon by itself prevents new workloads from scheduling onto a Node. It does not evict the workloads already running there. For most maintenance reasons, CoreWeave waits for the Node to become idle before continuing, which lets your workloads finish on their own schedule. A cordoned Node still counts toward its Node Pool’sCURRENT, so a Node Pool can sit at target while its schedulable capacity is reduced. If Pods stay Pending while the Node Pool looks healthy, see At target but Pods won’t schedule.
Interpret current Node conditions
Standard Kubernetes Node conditions are visible to you and safe to act on. View them in theConditions section of kubectl describe node, or list them directly:
CoreWeave also sets its own conditions on Nodes to drive its Node lifecycle automation. Don’t build automation on those conditions, and change them only where this guide says you can. See Important considerations.
What to do when a Node is cordoned
If you notice that one of your Nodes is cordoned, the following actions can help you assess the situation and decide whether to intervene.Check Node status
Confirm the Node state withkubectl describe node [NODE-NAME] to verify the reason for cordoning. If workloads remain running and stable, you don’t need to take immediate action.
Monitor workloads
Ensure your workloads have redundancy or are configured for high availability to tolerate temporary Node unavailability. If workloads become impacted, evaluate whether manual intervention is necessary, such as deleting affected Pods to trigger rescheduling.Persistent issues
If a Node remains cordoned for an extended period or your workload is severely impacted, open a support ticket to inform CoreWeave about the issue. Clearly document your observations, such as the Node status and workload behavior, in your support request.Important considerations
Keep the following considerations in mind when working with cordoned Nodes:- Don’t build automation on Node conditions defined by CoreWeave: The conditions CoreWeave sets drive its own Node lifecycle automation and can change without notice, so don’t rely on them for your own automation or management. Change those conditions only through documented procedures. For a manual cordon, set
AdminTemporaryFailurewith one of the methods in Cordon a Node yourself. For a reboot, follow Reboot Nodes. You can also add your own custom Node conditions. - Cordoning does not always indicate a serious or permanent Node issue: Nodes often recover automatically after temporary issues. If the issue is severe or persistent, CoreWeave proactively moves affected Nodes out of production into triage.