Skip to main content
A Pod stays Pending when the Kubernetes scheduler cannot place it on any Node. The Pod’s FailedScheduling event names the exact constraint the scheduler could not satisfy, so read that first, then apply the matching fix. All commands in this article are read-only.

​
Find the cause

1

Read the scheduling events

Replace [POD-NAME] and [NAMESPACE] with your Pod and namespace, then run:
Read the FailedScheduling warning in the Events section at the bottom.
2

Match the message to a fix

​
Free up GPU, CPU, or memory capacity

The scheduler reserves capacity based on each Pod’s requests, whether or not the Pod uses it. A Node can report near-zero utilization while all of its GPUs, CPU, or memory are already allocated to other Pods.
1

Check the Pod's requests

To view the Pod’s resource requests, run:
2

Check whether any Node has room

List each Node’s allocatable GPUs:
On a large cluster, use the Cluster Resource Overview dashboard instead. Its GPU Info and Node Info panels show allocated GPUs and each Node’s requests against capacity without querying every Node through the Kubernetes API server.
3

Lower the request or add capacity

If the request exceeds what a single Node provides, lower the request. If every Node of the right type is full, scale up the Node Pool or add one with the correct instance type. See Node Pool autoscaling.If a scaled Node Pool is not delivering Nodes, the problem is capacity or quota, not scheduling. See Node Pool status.
For GPU-specific checks, see Why are my Pods not scheduling on GPU Nodes?. For how the autoscaler responds to pending Pods, see What is the autoscaler behavior when Pods are pending?.

​
Check why the Node is tainted

1

Inspect the taints on the target Node

Replace [NODE-NAME] with the Node named in the FailedScheduling message, then run:
2

Identify the taint's source

In CKS, Pods do not need extra tolerations to schedule onto production Nodes. A blocking taint in the FailedScheduling message usually means one of the following:
  • CoreWeave moved the Node out of production, for example for triage or maintenance. The Node is intentionally unschedulable. Do not add a toleration. CKS replaces the Node. See Node cordoning.
  • Your own tooling added a custom taint. Remove the taint or adjust which Node Pools your automation taints.
You may also see is_gpu=true:PreferNoSchedule in the taint output. It is a soft preference, never blocks scheduling, and is not the cause of a Pending Pod.If a CoreWeave-managed taint appears to block a Pod that should schedule, contact support instead of adding a toleration.
For the full taint reference, see Workload scheduling on CKS.

​
Fix the Node selector or affinity

Your Pod requires a label that no current Node carries. A common cause is a selector on the deprecated node.coreweave.cloud/class label. Do not target it for scheduling. Use node.kubernetes.io/instance-type instead.
1

Show the labels the Pod requires

To list the labels the Pod’s selector requires, run:
2

Confirm which Nodes carry that label

Replace [LABEL-KEY] and [LABEL-VALUE] with the label from the Pod’s nodeSelector, then run:
3

Correct the selector

Change the selector to use a label that exists on your Nodes. For rack, superpod, or fabric targeting, use the labels documented at InfiniBand and RoCE labels.
See also How do I request specific GPU types using Node selectors?.

​
Leave headroom for Node services

CoreWeave runs a small set of managed per-Node services (networking, storage drivers, monitoring, and health agents) that reserve a few cores and some memory on every Node. A Pod that requests nearly the entire Node cannot fit, even on a Node with no other customer workloads, and preemption does not help.

​
Admit the Kueue Workload

Job queueing and orchestration systems manage Pods through their own higher-level resources. Kueue, for example, represents each job as a Workload resource and controls execution by toggling the job’s suspended state or managing its scheduling gates. Because Kueue intercepts and queues the job before it can run, the reason your workloads are delayed, or why underlying Pods remain Pending, is usually recorded in the status and events of that higher-level resource, not on the Pod itself. This section covers Kueue. If a different controller manages your job, inspect its governing resource the same way. A Workload that Kueue has not admitted never reaches the scheduler, so its Pods show no FailedScheduling events.
1

Check the admission state

Replace [WORKLOAD-NAME] with the Workload name from the first command’s output, then run:
A message such as couldn't assign flavors to pod set means no Nodes match the requested ResourceFlavor or topology at the requested scale.
2

Restore quota or matching capacity

Confirm your ClusterQueue has quota, confirm Nodes exist that match the ResourceFlavor, and reduce the requested topology size if a full rack is not available.
3

Re-trigger the TAS assignment

This step applies only if you use Topology-Aware Scheduling (TAS). TAS assigns a Workload to specific Nodes at admission time and does not re-evaluate if one of those Nodes is later cordoned. Suspend and unsuspend the JobSet (set spec.suspend to true, then back to false) so Kueue recomputes the assignment onto healthy Nodes. If this recurs, ask support about enabling waitForPodsReady in the Kueue controller configuration.
See Kueue and How do I use Kueue for job queuing?.

​
Remove the Node name pin

A Pod that sets .spec.nodeName bypasses the scheduler entirely: no preemption, no Kueue or TAS integration, and no eviction of HPC verification Pods to make room.
1

Check whether the Pod is pinned

To check whether the Pod sets .spec.nodeName, run:
2

Replace the pin with a selector

If the command returns a Node name, replace .spec.nodeName with a nodeSelector or nodeAffinity that targets the same Node. See the Node type selection labels.

​
When Grafana shows free GPUs

Trust the scheduler’s FailedScheduling message over the dashboard, and cross-check any dashboard against kubectl get nodes. Free GPUs in Grafana do not guarantee the scheduler can use them, for any of the following reasons:
  • The GPUs are allocated to Pods that are idle. Utilization panels show them as free, but the scheduler counts them as reserved.
  • The Node hosting them carries a taint the Pod does not tolerate.
  • The free GPUs are spread across Nodes, but the Pod needs them on one Node or in a contiguous topology.
  • Another requested resource (CPU, memory, an extended resource) is exhausted.
  • The dashboard panel is stale because metrics ingestion is degraded.

​
When to open a support ticket

If a scaled Node Pool is not delivering Nodes and Node Pool status shows a capacity or delivery problem, or if Pods stay Pending with Insufficient cpu on idle Nodes and nothing changed on your side, open a ticket. Include the Pod name and namespace, the output of kubectl describe pod, and the Node Pool name. Workload Scheduling
Last modified on July 28, 2026