> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why is my Pod stuck in pending on CKS?

A Pod stays `Pending` when the Kubernetes scheduler cannot place it on any Node. The Pod's `FailedScheduling` event names the exact constraint the scheduler could not satisfy, so read that first, then apply the matching fix. All commands in this article are read-only.

## Find the cause

<Steps>
  <Step title="Read the scheduling events">
    Replace `[POD-NAME]` and `[NAMESPACE]` with your Pod and namespace, then run:

    ```bash theme={"system"}
    kubectl describe pod [POD-NAME] -n [NAMESPACE]
    ```

    Read the `FailedScheduling` warning in the `Events` section at the bottom.
  </Step>

  <Step title="Match the message to a fix">
    | You see                                                                            | Fix                                                                         |
    | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
    | `Insufficient nvidia.com/gpu`, `Insufficient cpu`, or `Insufficient memory`        | [Free up GPU, CPU, or memory capacity](#free-up-gpu-cpu-or-memory-capacity) |
    | `node(s) had untolerated taint`                                                    | [Check why the Node is tainted](#check-why-the-node-is-tainted)             |
    | `didn't match Pod's node affinity/selector`                                        | [Fix the Node selector or affinity](#fix-the-node-selector-or-affinity)     |
    | `No preemption victims found` or `Preemption is not helpful` on idle-looking Nodes | [Leave headroom for Node services](#leave-headroom-for-node-services)       |
    | No events at all, and you use Kueue                                                | [Admit the Kueue Workload](#admit-the-kueue-workload)                       |
    | `node(s) were unschedulable` on a Pod pinned by Kueue TAS                          | [Re-trigger the TAS assignment](#admit-the-kueue-workload)                  |
    | No events at all, and the Pod sets `.spec.nodeName`                                | [Remove the Node name pin](#remove-the-node-name-pin)                       |
    | Grafana shows free GPUs, but the Pod stays `Pending`                               | [When Grafana shows free GPUs](#when-grafana-shows-free-gpus)               |
  </Step>
</Steps>

## Free up GPU, CPU, or memory capacity

The scheduler reserves capacity based on each Pod's `requests`, whether or not the Pod uses it. A Node can report near-zero utilization while all of its GPUs, CPU, or memory are already allocated to other Pods.

<Steps>
  <Step title="Check the Pod's requests">
    To view the Pod's resource requests, run:

    ```bash theme={"system"}
    kubectl get pod [POD-NAME] -n [NAMESPACE] \
      -o jsonpath='{.spec.containers[*].resources.requests}'
    ```
  </Step>

  <Step title="Check whether any Node has room">
    List each Node's allocatable GPUs:

    ```bash theme={"system"}
    kubectl get nodes -o custom-columns=\
    NAME:.metadata.name,\
    GPU:.status.allocatable.nvidia\\.com/gpu
    ```

    <Info>
      On a large cluster, use the [Cluster Resource Overview dashboard](/observability/managed-grafana/cks/cluster-resource) instead. Its GPU Info and Node Info panels show allocated GPUs and each Node's requests against capacity without querying every Node through the Kubernetes API server.
    </Info>
  </Step>

  <Step title="Lower the request or add capacity">
    If the request exceeds what a single Node provides, lower the request. If every Node of the right type is full, scale up the Node Pool or add one with the correct instance type. See [Node Pool autoscaling](/products/cks/nodes/autoscaling).

    If a scaled Node Pool is not delivering Nodes, the problem is capacity or quota, not scheduling. See [Node Pool status](/products/cks/nodes/nodepool-status).
  </Step>
</Steps>

For GPU-specific checks, see [Why are my Pods not scheduling on GPU Nodes?](/support/cks/articles/why-are-my-pods-not-scheduling-on-gpu-nodes). For how the autoscaler responds to pending Pods, see [What is the autoscaler behavior when Pods are pending?](/support/cks/articles/what-is-the-autoscaler-behavior-when-pods-are-pending).

## Check why the Node is tainted

<Steps>
  <Step title="Inspect the taints on the target Node">
    Replace `[NODE-NAME]` with the Node named in the `FailedScheduling` message, then run:

    ```bash theme={"system"}
    kubectl get node [NODE-NAME] -o jsonpath='{.spec.taints}'
    ```
  </Step>

  <Step title="Identify the taint's source">
    In CKS, Pods do not need extra tolerations to schedule onto production Nodes. A blocking taint in the `FailedScheduling` message usually means one of the following:

    * CoreWeave moved the Node out of production, for example for triage or maintenance. The Node is intentionally unschedulable. Do not add a toleration. CKS replaces the Node. See [Node cordoning](/products/cks/nodes/cordon).
    * Your own tooling added a custom taint. Remove the taint or adjust which Node Pools your automation taints.

    You may also see `is_gpu=true:PreferNoSchedule` in the taint output. It is a soft preference, never blocks scheduling, and is not the cause of a `Pending` Pod.

    If a CoreWeave-managed taint appears to block a Pod that should schedule, [contact support](/support/contact) instead of adding a toleration.
  </Step>
</Steps>

For the full taint reference, see [Workload scheduling on CKS](/products/cks/clusters/scheduling/workload-scheduling#taints-and-tolerations).

## Fix the Node selector or affinity

Your Pod requires a label that no current Node carries. A common cause is a selector on the deprecated `node.coreweave.cloud/class` label. Do not target it for scheduling. Use `node.kubernetes.io/instance-type` instead.

<Steps>
  <Step title="Show the labels the Pod requires">
    To list the labels the Pod's selector requires, run:

    ```bash theme={"system"}
    kubectl get pod [POD-NAME] -n [NAMESPACE] -o jsonpath='{.spec.nodeSelector}'
    ```
  </Step>

  <Step title="Confirm which Nodes carry that label">
    Replace `[LABEL-KEY]` and `[LABEL-VALUE]` with the label from the Pod's `nodeSelector`, then run:

    ```bash theme={"system"}
    kubectl get nodes -l [LABEL-KEY]=[LABEL-VALUE]
    ```
  </Step>

  <Step title="Correct the selector">
    Change the selector to use a label that exists on your Nodes. For rack, superpod, or fabric targeting, use the labels documented at [InfiniBand and RoCE labels](/products/networking/hpc-interconnect/infiniband-roce-labels).
  </Step>
</Steps>

See also [How do I request specific GPU types using Node selectors?](/support/cks/articles/how-do-i-request-specific-gpu-types-using-node-selectors).

## Leave headroom for Node services

CoreWeave runs a small set of managed per-Node services (networking, storage drivers, monitoring, and health agents) that reserve a few cores and some memory on every Node. A Pod that requests nearly the entire Node cannot fit, even on a Node with no other customer workloads, and preemption does not help.

* Leave headroom in the Pod's CPU and memory requests rather than requesting the full Node. See [How do CPU and memory requests work with GPU Pods?](/support/cks/articles/how-do-cpu-and-memory-requests-work-with-gpu-pods).

* If the shortfall appeared suddenly with no change on your side, [contact support](/support/contact) with the Pod name, its requests, and the `FailedScheduling` events.

## Admit the Kueue Workload

Job queueing and orchestration systems manage Pods through their own higher-level resources. Kueue, for example, represents each job as a Workload resource and controls execution by toggling the job's suspended state or managing its scheduling gates. Because Kueue intercepts and queues the job before it can run, the reason your workloads are delayed, or why underlying Pods remain `Pending`, is usually recorded in the status and events of that higher-level resource, not on the Pod itself. This section covers Kueue. If a different controller manages your job, inspect its governing resource the same way.

A Workload that Kueue has not admitted never reaches the scheduler, so its Pods show no `FailedScheduling` events.

<Steps>
  <Step title="Check the admission state">
    Replace `[WORKLOAD-NAME]` with the Workload name from the first command's output, then run:

    ```bash theme={"system"}
    kubectl get workloads.kueue.x-k8s.io -n [NAMESPACE]
    kubectl describe workloads.kueue.x-k8s.io [WORKLOAD-NAME] -n [NAMESPACE]
    ```

    A message such as `couldn't assign flavors to pod set` means no Nodes match the requested `ResourceFlavor` or topology at the requested scale.
  </Step>

  <Step title="Restore quota or matching capacity">
    Confirm your `ClusterQueue` has quota, confirm Nodes exist that match the `ResourceFlavor`, and reduce the requested topology size if a full rack is not available.
  </Step>

  <Step title="Re-trigger the TAS assignment">
    This step applies only if you use Topology-Aware Scheduling (TAS). TAS assigns a Workload to specific Nodes at admission time and does not re-evaluate if one of those Nodes is later cordoned. Suspend and unsuspend the `JobSet` (set `spec.suspend` to `true`, then back to `false`) so Kueue recomputes the assignment onto healthy Nodes. If this recurs, ask support about enabling `waitForPodsReady` in the Kueue controller configuration.
  </Step>
</Steps>

See [Kueue](/products/cks/clusters/coreweave-charts/kueue) and [How do I use Kueue for job queuing?](/support/cks/articles/how-do-i-use-kueue-for-job-queuing).

## Remove the Node name pin

A Pod that sets `.spec.nodeName` bypasses the scheduler entirely: no preemption, no Kueue or TAS integration, and no eviction of [HPC verification](/platform/fleet-management/hpc-verification) Pods to make room.

<Steps>
  <Step title="Check whether the Pod is pinned">
    To check whether the Pod sets `.spec.nodeName`, run:

    ```bash theme={"system"}
    kubectl get pod [POD-NAME] -n [NAMESPACE] -o jsonpath='{.spec.nodeName}'
    ```
  </Step>

  <Step title="Replace the pin with a selector">
    If the command returns a Node name, replace `.spec.nodeName` with a `nodeSelector` or `nodeAffinity` that targets the same Node. See the [Node type selection labels](/products/cks/clusters/scheduling/workload-scheduling#node-type-selection-labels).
  </Step>
</Steps>

## When Grafana shows free GPUs

Trust the scheduler's `FailedScheduling` message over the dashboard, and cross-check any dashboard against `kubectl get nodes`. Free GPUs in Grafana do not guarantee the scheduler can use them, for any of the following reasons:

* The GPUs are allocated to Pods that are idle. Utilization panels show them as free, but the scheduler counts them as reserved.
* The Node hosting them carries a taint the Pod does not tolerate.
* The free GPUs are spread across Nodes, but the Pod needs them on one Node or in a contiguous topology.
* Another requested resource (CPU, memory, an extended resource) is exhausted.
* The dashboard panel is stale because metrics ingestion is degraded.

## When to open a support ticket

If a scaled Node Pool is not delivering Nodes and Node Pool status shows a capacity or delivery problem, or if Pods stay `Pending` with `Insufficient cpu` on idle Nodes and nothing changed on your side, open a ticket. Include the Pod name and namespace, the output of `kubectl describe pod`, and the Node Pool name.

## Related pages

* [Node Pool status](/products/cks/nodes/nodepool-status): read `targetNodes`, capacity, and quota states.
* [Node cordoning](/products/cks/nodes/cordon): when Nodes stop accepting Pods.
* [Workload scheduling on CKS](/products/cks/clusters/scheduling/workload-scheduling): taints, labels, and eviction policies.

<Badge stroke shape="pill" color="blue" size="md">[Workload Scheduling](/support/cks/tags/workload-scheduling)</Badge>