Why Kubernetes static CPU allocation is incompatible with the SUNK Pod Scheduler and how to prevent node drains.
SUNK StandardSUNK Self-ServiceThis page is for cluster operators and workload owners who run SUNK on CoreWeave CKS. It explains how to identify Pods that trigger static CPU allocation, why the issue causes SUNK nodes to drain, and how to configure Pod resources to prevent it. Follow this guidance to avoid silent Slurm job failures and unexpected node drains caused by CPU pinning.For a related allocation issue where oversized per-task CPU or memory requests leave a GPU unusable, see Avoid stranding GPUs with CPU and memory requests.Kubernetes can pin specific CPU cores to a Pod so that no other process on the Node shares those cores. This feature is called static CPU allocation, and the CPU Manager static policy in the kubelet controls it. CoreWeave CKS Nodes enable this policy by default.Static CPU allocation is useful for latency-sensitive workloads that benefit from dedicated cores. However, it’s incompatible with the SUNK Pod Scheduler because Slurm can’t account for CPU cores that the kubelet has pinned to other Pods.
Do not schedule Pods that have Guaranteed QoS and a whole-number CPU request on Kubernetes Nodes used for SUNK. These Pods trigger static CPU allocation, which causes resource contention in Slurm. As a result, Slurm nodes drain with misleading error messages such as batch job complete failure.
Slurm is configured with a fixed CPU count for each node. It doesn’t account for CPUs that the kubelet has pinned to Guaranteed QoS Pods. When static CPU allocation removes cores from the shared pool, the CPUs available to Slurm shrink to the configured total minus whatever the kubelet has pinned. Slurm continues to schedule jobs against the original count, which leads to resource contention.The following sequence shows how the issue occurs:
Kubernetes schedules a Pod with Guaranteed QoS and integer CPU requests onto a SUNK node.
The CPU Manager in the kubelet pins specific CPU cores to the Pod using cpuset cgroups.
Slurm schedules a job to the node with pinned resources.
Static CPU allocation drains a node with one of the following reasons. These messages can be misleading because none of them point to the underlying CPU pinning by the kubelet.
Reason
Cause
Safe to undrain?
batch job complete failure
A job fails from resource contention on the pinned cores.
Only after you fix the Pod. Otherwise, the node drains again the next time a contended job lands on it.
Low socket*core*thread count, Low CPUs
slurmd detects fewer CPUs than configured, such as 127 instead of 128, because the kubelet removed the pinned CPUs from the shared pool.
A Pod triggers static CPU allocation only when it has Guaranteed QoS and a whole-number CPU request, so breaking either condition prevents the drain. Choose one of the following fixes for every Pod that runs on SUNK nodes:
Switch the Pod to Burstable QoS by setting its CPU request lower than its limit, or by omitting the CPU limit.
Keep Guaranteed QoS, but request a fractional CPU value such as 3500m.
Either change excludes the Pod from the kubelet’s static CPU manager, so it no longer pins cores. For the rules the kubelet applies, see CPU management policies in the Kubernetes documentation. The following sections describe each fix and how to verify the configuration.
You can also omit the CPU limits field altogether to produce Burstable QoS. With Burstable QoS in place, the kubelet no longer pins CPU cores, and Slurm can account for the full CPU capacity of the node.
If the workload needs Guaranteed QoS, for example so that the kubelet evicts it last under memory pressure, keep requests equal to limits but use a fractional CPU value:
The Pod keeps its Guaranteed QoS class. Because its CPU request isn’t a whole number, the kubelet runs it from the shared CPU pool instead of pinning cores to it.
To check whether a Pod has Guaranteed or Burstable QoS, query its status. Replace [POD-NAME] and [NAMESPACE] with your Pod’s name and namespace:
kubectl get pod [POD-NAME] -n [NAMESPACE] -o jsonpath='{.status.qosClass}'
A result of Burstable means the Pod can’t trigger static CPU allocation. A result of Guaranteed is a problem only if the Pod also requests a whole number of CPUs, so check its CPU request as well.To check all Pods on a specific Node, replace [NODE-NAME] with the Node’s name:
kubectl get pods -A --field-selector spec.nodeName=[NODE-NAME] -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.qosClass}{"\n"}{end}'
For any Pod with the Guaranteed QoS class, check whether its CPU request is a whole number. If it is, update its resource specification as described in Prevent node drains from static CPU allocation.
If this issue has already drained nodes, address the underlying CPU pinning before the nodes can return to service. Update the Pod resource specification so that CPU requests don’t equal CPU limits, or so that the CPU request is a fractional value. The Pod reschedules without pinned cores. Then restart slurmd on each affected node so that it re-reads the CPU count, and undrain the node. For the slurmd log lines to expect before and after the fix, see CPU topology mismatch from a statically pinned Pod. For instructions on undraining nodes, see Drain and undrain Slurm nodes.To identify nodes drained by this issue, look for the drain reasons listed in Common drain reasons:
