What is Node cordoning?
In Kubernetes, a cordoned Node prevents new workloads from scheduling onto it. Existing workloads on the Node continue to run until you explicitly drain them or they complete. Cordoning typically indicates maintenance, troubleshooting, or reliability actions. CoreWeave cordons Nodes proactively to ensure reliability, maintain cluster health, and manage maintenance activities with minimal impact on workloads.Reasons for Node cordoning
Nodes in CKS may be cordoned for several reasons:Maintenance and updates
CoreWeave cordons Nodes that require hardware maintenance, kernel updates, or hardware replacements to avoid scheduling new workloads. This practice allows maintenance activities without disrupting running workloads.Health monitoring
CoreWeave monitors Node health and performance. If a Node exhibits issues that could affect workload reliability, CoreWeave may cordon it to prevent new workloads from being scheduled. These issues can include:- Hardware issues: GPU hardware errors trigger automatic Node cordoning and rebooting.
- Kernel deadlocks: Severe kernel issues may also cause Nodes to reboot and remain cordoned until resolved.
- InfiniBand or Ethernet link flaps: CoreWeave cordons Nodes with unstable network connections (intermittent connectivity) so workloads aren’t placed on unreliable Nodes. This detection looks for instability, so a link that fails cleanly and stays down doesn’t cordon the Node. If a Node keeps running but delivers less network bandwidth than you expect, contact support with the Node name.
- Connectivity failures: If Kubernetes detects critical network issues, CoreWeave cordons the affected Nodes until stability is restored.
Time synchronization
CKS Nodes synchronize time to CoreWeave’s internal time infrastructure. CoreWeave monitors time on all Nodes, and a Node whose clock drifts beyond acceptable bounds may be cordoned for triage. Running any competing time daemon (includingchrony, ntpd, systemd-timesyncd) directly on a Node or in containers may cause clock drift and trigger this behaviour. Overriding the time servers on CKS Nodes is not supported. Containers with time daemons must be configured to disable the daemon and/or prevent the daemon from altering the kernel clock.
Temporary issues
Temporary issues detected through Kubernetes health checks can lead to short-term Node cordoning. These usually resolve automatically, and Nodes become schedulable again without intervention.Scheduled maintenance
CoreWeave intentionally cordons Nodes during scheduled maintenance.Lifecycle management
CoreWeave manages Node lifecycle events, such as provisioning, decommissioning, reboots, and upgrades. Temporary cordoning may occur when Nodes transition between different operational states to maintain cluster stability and reliability during lifecycle transitions. To understand how these transitions behave inside your cluster, see Node state transitions in CKS.Check why a Node is cordoned
When CoreWeave cordons a Node, it records the reason in thenode.coreweave.cloud/cordonReason annotation. Check this annotation to understand whether the cordon is an expected maintenance action or a possible sign of poor Node health.
Read the cordon reason for a Node:
[NODE-NAME] with the name of your Node.
For the Node’s full status, including whether it is unschedulable and its current conditions, use describe:
NLCCPendingExitProduction, the matching Node condition message names the state the Node is moving to and what initiated the transition. A username identifies a CoreWeave engineer who started it by hand, while a message such as Reboot from Slurm identifies automation:
Example condition
Common cordon and drain reasons
The following table lists the cordon reasons you are most likely to see on a CKS Node, what each one means, and whether it represents an expected maintenance action or a possible health issue.Health and lifecycle cordon reasons
CoreWeave’s node controller sets the following cordon reasons from Node conditions. Each reason names the fault, and the last column describes what CoreWeave’s automation does next: keep the Node cordoned until the condition clears, reboot it, or move it totriage. A reboot waits for the Node to become idle unless the row says it’s immediate. For what each lifecycle state means, see Node states in a CKS cluster.
On a SUNK cluster, these reasons also appear as Slurm drain reasons in the form k8s: cordon: [CORDON-REASON]. See Kubernetes failure cases.
Set AdminTemporaryFailure on a Node
Always patch Node conditions with--type=strategic. A merge patch replaces the whole status.conditions array, which deletes every other condition on the Node, including Ready. CoreWeave automation then misreads the Node’s state, and your Pods can be terminated.
Use the following command to set AdminTemporaryFailure on a Node:
Set AdminTemporaryFailure on a Node
What a cordon affects
A cordon by itself prevents new workloads from scheduling onto a Node. It does not evict the workloads already running there. For most maintenance reasons, CoreWeave waits for the Node to become idle before continuing, which lets your workloads finish on their own schedule. A cordoned Node still counts toward its Node Pool’sCURRENT, so a Node Pool can sit at target while its schedulable capacity is reduced. If Pods stay Pending while the Node Pool looks healthy, see At target but Pods won’t schedule.
Interpret current Node conditions
Standard Kubernetes Node conditions are visible to you and safe to act on. View them in theConditions section of kubectl describe node, or list them directly:
CoreWeave also sets its own conditions on Nodes to drive its health automation. Don’t build automation on those conditions, and change them only where this guide says you can. See Important considerations.
What to do when a Node is cordoned
If you notice that one of your Nodes is cordoned, the following actions can help you assess the situation and decide whether to intervene.Check Node status
Confirm the Node state withkubectl describe node [NODE-NAME] to verify the reason for cordoning. If workloads remain running and stable, you don’t need to take immediate action.
Monitor workloads
Ensure your workloads have redundancy or are configured for high availability to tolerate temporary Node unavailability. If workloads become impacted, evaluate whether manual intervention is necessary, such as deleting affected Pods to trigger rescheduling.Persistent issues
If a Node remains cordoned for an extended period or your workload is severely impacted, open a support ticket to inform CoreWeave about the issue. Clearly document your observations, such as the Node status and workload behavior, in your support request.Important considerations
Keep the following considerations in mind when working with cordoned Nodes:- Don’t build automation on Node conditions defined by CoreWeave: The conditions CoreWeave sets drive its own health automation and can change without notice, so don’t rely on them for your own automation or management. Don’t change the conditions CoreWeave’s automation manages either.
AdminTemporaryFailureis the exception: you can set it yourself to hold a cordon in place, as described in Common cordon and drain reasons. You can also add your own custom Node conditions. - Cordoning does not always indicate a serious or permanent Node issue: Nodes often recover automatically after temporary issues. If the issue is severe or persistent, CoreWeave proactively moves affected Nodes out of production into triage.