This page is for cluster operators and workload owners who run SUNK on CoreWeave CKS. It explains how to identify Pods that trigger static CPU allocation, why the issue causes SUNK nodes to drain, and how to configure Pod resources to prevent it. Follow this guidance to avoid silent Slurm job failures and unexpected node drains caused by CPU pinning.
For a related allocation issue where oversized per-task CPU or memory requests leave a GPU unusable, see Avoid stranding GPUs with CPU and memory requests.
Kubernetes can pin specific CPU cores to a Pod so that no other process on the Node shares those cores. This feature is called static CPU allocation, and the CPU Manager static policy in the kubelet controls it. CoreWeave CKS Nodes enable this policy by default.
Static CPU allocation is useful for latency-sensitive workloads that benefit from dedicated cores. However, it’s incompatible with the SUNK Pod Scheduler because Slurm can’t account for CPU cores that the kubelet has pinned to other Pods.
Do not schedule Pods that have Guaranteed QoS and a whole-number CPU request on Kubernetes Nodes used for SUNK. These Pods trigger static CPU allocation, which causes resource contention in Slurm. As a result, Slurm nodes drain with misleading error messages such as batch job complete failure.
Static CPU allocation
A Pod triggers static CPU allocation when both of these conditions are true:
- The Pod has Guaranteed QoS class, meaning every container, including init containers, sets CPU and memory requests equal to its limits.
- The CPU request is a whole integer (for example,
cpu: 4), not a fractional value (for example, cpu: 3.5 or cpu: 750m).
For example, this resource specification triggers static CPU allocation because requests and limits are equal and the CPU value is an integer:
CPU allocation with SUNK
Slurm is configured with a fixed CPU count for each node. It doesn’t account for CPUs that the kubelet has pinned to Guaranteed QoS Pods. When static CPU allocation removes cores from the shared pool, the CPUs available to Slurm shrink to the configured total minus whatever the kubelet has pinned. Slurm continues to schedule jobs against the original count, which leads to resource contention.
The following sequence shows how the issue occurs:
- Kubernetes schedules a Pod with Guaranteed QoS and integer CPU requests onto a SUNK node.
- The CPU Manager in the
kubelet pins specific CPU cores to the Pod using cpuset cgroups.
- Slurm schedules a job to the node with pinned resources.
- The job encounters resource contention and fails.
- Slurm drains the node because of the job failure.
Common drain reasons
When static CPU allocation causes a job to fail from resource contention, Slurm drains the node with the reason batch job complete failure. This message can be misleading because it doesn’t point to the underlying CPU pinning by the kubelet.
Prevent node drains from static CPU allocation
A Pod triggers static CPU allocation only when it has Guaranteed QoS and a whole-number CPU request, so breaking either condition prevents the drain. Choose one of the following fixes for every Pod that runs on SUNK nodes:
- Switch the Pod to Burstable QoS by setting its CPU request lower than its limit, or by omitting the CPU limit.
- Keep Guaranteed QoS, but request a fractional CPU value such as
3500m.
Either change excludes the Pod from the kubelet’s static CPU manager, so it no longer pins cores. For the rules the kubelet applies, see CPU management policies in the Kubernetes documentation. The following sections describe each fix and how to verify the configuration.
Switch Pods to Burstable QoS
Set CPU requests lower than CPU limits to change the Pod’s QoS class from Guaranteed to Burstable:
You can also omit the CPU limits field altogether to produce Burstable QoS. With Burstable QoS in place, the kubelet no longer pins CPU cores, and Slurm can account for the full CPU capacity of the node.
Keep Guaranteed QoS with a fractional CPU request
If the workload needs Guaranteed QoS, for example so that the kubelet evicts it last under memory pressure, keep requests equal to limits but use a fractional CPU value:
The Pod keeps its Guaranteed QoS class. Because its CPU request isn’t a whole number, the kubelet runs it from the shared CPU pool instead of pinning cores to it.
Verify a Pod’s QoS class
To check whether a Pod has Guaranteed or Burstable QoS, query its status. Replace [POD-NAME] and [NAMESPACE] with your Pod’s name and namespace:
A result of Burstable means the Pod can’t trigger static CPU allocation. A result of Guaranteed is a problem only if the Pod also requests a whole number of CPUs, so check its CPU request as well.
To check all Pods on a specific Node, replace [NODE-NAME] with the Node’s name:
For any Pod with the Guaranteed QoS class, check whether its CPU request is a whole number. If it is, update its resource specification as described in Prevent node drains from static CPU allocation.
Recover drained nodes
If this issue has already drained nodes, address the underlying CPU pinning before the nodes can return to service. Update the Pod resource specification so that CPU requests don’t equal CPU limits, or so that the CPU request is a fractional value. The Pod reschedules without pinned cores. Then restart slurmd on each affected node so that it re-reads the CPU count, and undrain the node. For the slurmd log lines to expect before and after the fix, see CPU topology mismatch from a statically pinned Pod. For instructions on undraining nodes, see Drain and undrain Slurm nodes.
To identify nodes drained by this issue, look for the drain reason listed in Common drain reasons:
