Pending when they request a resource the available GPU Nodes do not provide, when no Node Pool of the requested instance type has capacity, or when the Pod’s nodeSelector or tolerations do not match any Node. Run kubectl describe pod [POD-NAME] and read the events at the bottom. The scheduler logs the exact mismatch (for example, 0/3 nodes are available: 3 Insufficient nvidia.com/gpu).
Common causes and fixes:
- No Node has enough GPUs: lower the
nvidia.com/gpurequest, or add or scale a Node Pool of the right instance type. See What is the autoscaler behavior when Pods are pending?. nodeSelectordoesn’t match any Node: correct the selector to match the labels actually present on your Nodes. See How do I request specific GPU types using node selectors?.- Tolerations don’t match Node taints: confirm your Pod tolerates the correct user taints for the hardware you target.
- Resource quota exceeded: check your namespace quota. See How do I check my resource quota?.
Workload Scheduling Server Errors