> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # How do I troubleshoot suspended Kueue workloads on CKS? Kueue keeps a Job or JobSet suspended (`spec.suspend: true`) until it admits the matching Workload. The usual causes are a `kueue.x-k8s.io/queue-name` label that doesn't name a `LocalQueue`, a `ClusterQueue` without enough quota for the request, or a topology-aware scheduling (TAS) fit failure. Describe the Workload, read its admission events and status conditions, and match the message to a fix in the table below. ```bash theme={"system"} kubectl get workloads.kueue.x-k8s.io -n [NAMESPACE] kubectl describe workloads.kueue.x-k8s.io [WORKLOAD-NAME] -n [NAMESPACE] ``` Use the fully qualified resource name. The k9s **workloads** view is a synthetic summary of Pods, Deployments, and similar resources, not the Kueue resource. If the Workload is admitted but its Pods stay `Pending`, see [Why is my Pod stuck in pending on CKS?](/support/cks/articles/why-is-my-pod-stuck-in-pending) instead. ## Match the message to a fix | Message or symptom | Cause | Fix | | - | - | - | | The Workload reports that the `LocalQueue` doesn't exist. | The `kueue.x-k8s.io/queue-name` label doesn't name a `LocalQueue` in the Job's namespace, for example because it names a `ClusterQueue` instead. | Set the label to a `LocalQueue` in the Job's namespace (`kubectl get localqueue -n [NAMESPACE]`) and resubmit. | | An admission event names a resource and flavor without enough quota. | The request exceeds the available quota for that resource and flavor. Borrowing or preemption might be unavailable, insufficient, or still pending. | Run `kubectl describe clusterqueue [CLUSTER-QUEUE-NAME]`, then wait for resources, reduce the request, or increase the quota. If eligible workloads can release enough resources, configure a `preemption` policy and workload priorities, such as a `WorkloadPriorityClass`. | | `couldn't assign flavors to pod set worker: topology "default" allows to fit only 23 out of 25 slice(s)` | TAS can't fit every slice. Possible causes include fragmented capacity, cordoned or tainted Nodes, and domains occupied by admitted workloads. | Check for cordoned Nodes in the target domains (see [Why is my Node cordoned?](/products/cks/nodes/cordon)). Reduce `kueue.x-k8s.io/podset-slice-size`, for example from `8` to `4`, if your application can tolerate less topology locality. If the topology constraint for the entire Pod set is too strict, change `podset-required-topology` to `podset-preferred-topology`; this does not relax a separate `podset-slice-required-topology` constraint. | | The webhook rejects the Job with an error about `kueue.x-k8s.io/podset-slice-size`. | `podset-slice-size` and `podset-slice-required-topology` must be set together, the size must be at least `1` and at most the Pod set count (for a Job, `parallelism`, capped by `completions` when specified; for a JobSet replicated Job, that number times `replicas`), and a Pod template carries at most one of `podset-required-topology`, `podset-preferred-topology`, and `podset-unconstrained-topology`, in addition to the paired slice annotations. | Fix the annotations on the Pod template. In a JobSet, that is `spec.replicatedJobs[].template.spec.template.metadata.annotations`, not the Job's `metadata`. A replicated Job with `replicas: 1` and `parallelism: 1` has a Pod set count of at most `1`, so it rejects any slice size above `1`. | | The fit failure reports many Nodes excluded by `affinity`. | TAS excludes Nodes that do not satisfy required node affinity, tolerations, or resource requests when it counts domain capacity. | Target the Node Pool with a `nodeSelector`. Check that the `nodeSelector` and any required node affinity allow the intended Nodes. | | Fit failures coincide with [HPC Verification](/platform/fleet-management/hpc-verification). | Older or customized Kueue integrations can count verification Pods against topology capacity. The current HPC Verification integration excludes these Pods from TAS capacity accounting. | If this symptom persists, contact [CoreWeave Support](/support/contact) with your cluster ID, installed Kueue version, and relevant Workload events so Support can check the integration. | ## Admitted, but a Pod is stuck Pending on a cordoned Node ### Understand automatic Node recovery TAS records a topology assignment at admission and adds Node selectors when it ungates Pods. The selector includes `kubernetes.io/hostname` only when that label is in the assignment's topology levels. Automatic Node recovery depends on your installed Kueue version and feature gates. In Kueue 0.19, a `NotReady` Node is not unconditionally replaced after 30 seconds with the default feature gates. Recovery also considers whether the Workload's Pods are still running on the Node. An untolerated `NoSchedule` or `NoExecute` taint can trigger recovery, but a cordon adds a `NoSchedule` taint without evicting running Pods. With the default gates, Pods that keep running can delay replacement. See [Kueue's Node failure handling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/#hot-swap-support). When `kubernetes.io/hostname` is the lowest topology level, Kueue records a single failed Node in the Workload's `status.unhealthyNodes` and attempts replacement. With the default `TASFailedNodeReplacementFailFast` gate enabled, it evicts and requeues the Workload if replacement fails. ### Force fresh admission for a stuck JobSet To force fresh admission for a stuck JobSet, deactivate and reactivate its Kueue Workload. This interrupts the workload, including any running Pods. Toggling only the JobSet's `spec.suspend` can resume the existing assignment. 1. Set the Workload's `spec.active` to `false`. 2. Wait for eviction to finish and the Workload's `status.admission` to clear. 3. Set the Workload's `spec.active` to `true` to allow fresh admission. Kueue reevaluates quota, topology fit, and any admission checks. ### Configure recovery for Pods that do not become ready To recover automatically when Pods do not become ready within a configured timeout, configure `waitForPodsReady` through the chart's `kueue.managerConfig.controllerManagerConfigYaml` value. This value replaces the whole configuration. Set `recoveryTimeout` for readiness failures after startup and `requeuingStrategy.backoffLimitCount` to limit retries; reaching the retry limit deactivates the Workload. See [Set up all-or-nothing scheduling with ready Pods](https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/). `Topology.spec.levels` is immutable. Before you change topology levels on a production queue, contact [CoreWeave Support](/support/contact). For chart installation and topology examples, see [Kueue](/products/cks/clusters/coreweave-charts/kueue). To monitor admission over time, use the [Kueue Scheduling Dashboard](/observability/managed-grafana/kubernetes/kueue-metrics). [Workload Scheduling](/support/cks/tags/workload-scheduling)