How do I troubleshoot suspended Kueue workloads on CKS?

Kueue keeps a Job or JobSet suspended (spec.suspend: true) until it admits the matching Workload. The usual causes are a kueue.x-k8s.io/queue-name label that doesn’t name a LocalQueue, a ClusterQueue without enough quota for the request, or a topology-aware scheduling (TAS) fit failure. Describe the Workload, read its admission events and status conditions, and match the message to a fix in the table below.

kubectl get workloads.kueue.x-k8s.io -n [NAMESPACE]
kubectl describe workloads.kueue.x-k8s.io [WORKLOAD-NAME] -n [NAMESPACE]

Use the fully qualified resource name. The k9s workloads view is a synthetic summary of Pods, Deployments, and similar resources, not the Kueue resource. If the Workload is admitted but its Pods stay Pending, see Why is my Pod stuck in pending on CKS? instead.

Match the message to a fix

Message or symptom	Cause	Fix
The Workload reports that the `LocalQueue` doesn’t exist.	The `kueue.x-k8s.io/queue-name` label doesn’t name a `LocalQueue` in the Job’s namespace, for example because it names a `ClusterQueue` instead.	Set the label to a `LocalQueue` in the Job’s namespace (`kubectl get localqueue -n [NAMESPACE]`) and resubmit.
An admission event names a resource and flavor without enough quota.	The request exceeds the available quota for that resource and flavor. Borrowing or preemption might be unavailable, insufficient, or still pending.	Run `kubectl describe clusterqueue [CLUSTER-QUEUE-NAME]`, then wait for resources, reduce the request, or increase the quota. If eligible workloads can release enough resources, configure a `preemption` policy and workload priorities, such as a `WorkloadPriorityClass`.
`couldn't assign flavors to pod set worker: topology "default" allows to fit only 23 out of 25 slice(s)`	TAS can’t fit every slice. Possible causes include fragmented capacity, cordoned or tainted Nodes, and domains occupied by admitted workloads.	Check for cordoned Nodes in the target domains (see Why is my Node cordoned?). Reduce `kueue.x-k8s.io/podset-slice-size`, for example from `8` to `4`, if your application can tolerate less topology locality. If the topology constraint for the entire Pod set is too strict, change `podset-required-topology` to `podset-preferred-topology`; this does not relax a separate `podset-slice-required-topology` constraint.
The webhook rejects the Job with an error about `kueue.x-k8s.io/podset-slice-size`.	`podset-slice-size` and `podset-slice-required-topology` must be set together, the size must be at least `1` and at most the Pod set count (for a Job, `parallelism`, capped by `completions` when specified; for a JobSet replicated Job, that number times `replicas`), and a Pod template carries at most one of `podset-required-topology`, `podset-preferred-topology`, and `podset-unconstrained-topology`, in addition to the paired slice annotations.	Fix the annotations on the Pod template. In a JobSet, that is `spec.replicatedJobs[].template.spec.template.metadata.annotations`, not the Job’s `metadata`. A replicated Job with `replicas: 1` and `parallelism: 1` has a Pod set count of at most `1`, so it rejects any slice size above `1`.
The fit failure reports many Nodes excluded by `affinity`.	TAS excludes Nodes that do not satisfy required node affinity, tolerations, or resource requests when it counts domain capacity.	Target the Node Pool with a `nodeSelector`. Check that the `nodeSelector` and any required node affinity allow the intended Nodes.
Fit failures coincide with HPC Verification.	Older or customized Kueue integrations can count verification Pods against topology capacity. The current HPC Verification integration excludes these Pods from TAS capacity accounting.	If this symptom persists, contact CoreWeave Support with your cluster ID, installed Kueue version, and relevant Workload events so Support can check the integration.

Admitted, but a Pod is stuck Pending on a cordoned Node

Understand automatic Node recovery

TAS records a topology assignment at admission and adds Node selectors when it ungates Pods. The selector includes kubernetes.io/hostname only when that label is in the assignment’s topology levels. Automatic Node recovery depends on your installed Kueue version and feature gates. In Kueue 0.19, a NotReady Node is not unconditionally replaced after 30 seconds with the default feature gates. Recovery also considers whether the Workload’s Pods are still running on the Node. An untolerated NoSchedule or NoExecute taint can trigger recovery, but a cordon adds a NoSchedule taint without evicting running Pods. With the default gates, Pods that keep running can delay replacement. See Kueue’s Node failure handling. When kubernetes.io/hostname is the lowest topology level, Kueue records a single failed Node in the Workload’s status.unhealthyNodes and attempts replacement. With the default TASFailedNodeReplacementFailFast gate enabled, it evicts and requeues the Workload if replacement fails.

Force fresh admission for a stuck JobSet

To force fresh admission for a stuck JobSet, deactivate and reactivate its Kueue Workload. This interrupts the workload, including any running Pods. Toggling only the JobSet’s spec.suspend can resume the existing assignment.

Set the Workload’s spec.active to false.
Wait for eviction to finish and the Workload’s status.admission to clear.
Set the Workload’s spec.active to true to allow fresh admission. Kueue reevaluates quota, topology fit, and any admission checks.

Configure recovery for Pods that do not become ready

To recover automatically when Pods do not become ready within a configured timeout, configure waitForPodsReady through the chart’s kueue.managerConfig.controllerManagerConfigYaml value. This value replaces the whole configuration. Set recoveryTimeout for readiness failures after startup and requeuingStrategy.backoffLimitCount to limit retries; reaching the retry limit deactivates the Workload. See Set up all-or-nothing scheduling with ready Pods. Topology.spec.levels is immutable. Before you change topology levels on a production queue, contact CoreWeave Support. For chart installation and topology examples, see Kueue. To monitor admission over time, use the Kueue Scheduling Dashboard. Workload Scheduling

​Match the message to a fix

​Admitted, but a Pod is stuck Pending on a cordoned Node

​Understand automatic Node recovery

​Force fresh admission for a stuck JobSet

​Configure recovery for Pods that do not become ready

Match the message to a fix

Admitted, but a Pod is stuck Pending on a cordoned Node

Understand automatic Node recovery

Force fresh admission for a stuck JobSet

Configure recovery for Pods that do not become ready