> ## Documentation Index > Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt > Use this file to discover all available pages before exploring further. # Why is my Slurm job stuck in PENDING? A Slurm job stays in `PENDING` (`PD`) when the scheduler has accepted the job but can't start it yet. Run `squeue -u $USER` and read the `REASON` column to learn whether to wait, change your submission, or escalate. Common cases: `Resources` and `Priority` mean the queue is working and you should wait, `AssocGrp*` and `QOS*` mean an accounting limit blocks the job, and configuration reasons such as `BadConstraints` or `PartitionConfig` mean you must change the request. A pending job isn't an error on its own. For why a job waits and what the `REASON` column means, see [Monitor Slurm job states](/products/sunk/manage_sunk/slurm-job-states#determine-the-reason-a-job-is-pending). This article maps each `REASON` value to a concrete action. ## What you see `squeue` shows your job in state `PD` with a value in the `REASON` (or `NODELIST(REASON)`) column. Read it with `squeue -u $USER`: ```bash theme={"system"} # Read the state and reason for your jobs. Read-only. squeue -u $USER ``` ```text title="Example output" theme={"system"} JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 123456 defq train alice PD 0:00 4 (Resources) 123457 defq train alice PD 0:00 4 (AssocGrpGRES) ``` To see the full reason and the job's exact request, use `scontrol`: ```bash theme={"system"} # Show the full job record, including TRES request, partition, and reason. Read-only. scontrol show job [JOB-ID] ``` For background on the job lifecycle, see [Monitor Slurm job states](/products/sunk/manage_sunk/slurm-job-states). ## Decision tree Find your `REASON` value in the sections that follow. Reasons fall into five groups: resource, quota and account, configuration, dependency and held, and the "Requested node configuration is not available" message that has several distinct causes. * If the reason is `Resources`, `Priority`, `BeginTime`, `Reservation`, or `ReqNodeNotAvail`, see [Resource and timing reasons](#resource-and-timing-reasons). * If the reason starts with `Assoc`, `QOS`, or is `InvalidAccount`, see [Quota and account reasons](#quota-and-account-reasons). * If the reason is `BadConstraints`, `PartitionConfig`, `PartitionTimeLimit`, or `PartitionNodeLimit`, see [Configuration reasons](#configuration-reasons). * If the reason is `Dependency`, `JobHeldUser`, or `JobHeldAdmin`, see [Dependency and held reasons](#dependency-and-held-reasons). * If nodes look idle but the job still waits, or you see `Requested node configuration is not available`, see [Idle nodes but still pending](#idle-nodes-but-still-pending). ## Resource and timing reasons These reasons mean the scheduler accepted your request and the job waits its turn. Usually, the correct action is to wait. | Reason | What it means | What to do | | ----------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Resources` | The cluster doesn't have enough free resources to run the job. | Wait. If it persists, confirm idle capacity exists (`sinfo`) and check for [idle nodes but still pending](#idle-nodes-but-still-pending). | | `Priority` | Higher-priority jobs are ahead of yours in the queue. The job runs once they complete or free resources. | Wait. To understand ordering, run `sprio -j [JOB-ID]` and `scontrol show job [JOB-ID]`. | | `BeginTime` | The job has a start time in the future, set with `--begin`. | Wait until the start time, or resubmit without `--begin`. | | `Reservation` | The job targets a Slurm reservation that hasn't started, or it's excluded from a reservation that holds nodes. | Check the reservation window with `scontrol show reservation`. Submit with the correct `--reservation` name, or wait for the reservation to start. | | `ReqNodeNotAvail` | A node the job requires (by name, feature, or reservation) is unavailable, for example, drained or down. | Run `sinfo -R` to list unavailable nodes and their reasons. If you pinned specific nodes with `--nodelist`, remove the constraint or wait for those nodes to return. See [Drain and undrain Slurm nodes](/products/sunk/manage_sunk/drain-and-undrain-nodes). | To confirm the cluster has idle capacity: ```bash theme={"system"} # Summarize node states by partition. Read-only. sinfo ``` ```bash theme={"system"} # List unavailable nodes and the reason each is unavailable. Read-only. sinfo -R ``` ## Quota and account reasons These reasons mean a Slurm accounting limit on your association (the combination of user, account, partition, and QoS) blocks the job. The job waits until usage drops below the limit, or until an administrator raises the limit. If you're at a hard cap and nothing else is running, waiting alone doesn't help. | Reason | What it means | What to do | | ------------------------ | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AssocGrpGRES` | Your association has reached its group limit on a generic resource (GRES), most often GPUs. | Wait for your other jobs to release GPUs, or request fewer GPUs. If the limit is too low for your work, ask an administrator to adjust it. | | `AssocGrpCpuLimit` | Your association has reached its group CPU limit. | Wait for running jobs to finish, or request fewer CPUs. | | `AssocGrpNodeLimit` | Your association has reached its group node-count limit. | Reduce `--nodes`, or wait for your other allocations to release nodes. | | `AssocMaxJobsLimit` | Your association is at its maximum number of running jobs. | Wait for a running job to finish. | | `QOSMaxJobsPerUserLimit` | The job's QoS limits how many jobs one user can run at once, and you're at the limit. | Wait, or submit under a different QoS if one is available to you. | | `QOSMaxGRESPerUser` | The job's QoS limits GPUs per user, and you're at the limit. | Wait, or request fewer GPUs. | | `InvalidAccount` | The account on the job doesn't exist or your user isn't a member of it. | Confirm your accounts with `sacctmgr show assoc user=$USER`. Submit with a valid `--account`. If your user is missing from accounting, see [nsscache](/products/sunk/manage_sunk/manage_cluster_access/nsscache). | To inspect your own limits and current usage: ```bash theme={"system"} # Show your accounting associations and their limits. Read-only. sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES,MaxJobs ``` A cluster administrator sets group GRES and CPU limits. If a limit is consistently too low for your workload, raise it with your administrator rather than retrying. ## Configuration reasons These reasons mean the job's request conflicts with the partition or constraint configuration. Waiting doesn't help. You must change the submission or the partition. | Reason | What it means | What to do | | -------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `BadConstraints` | No node can satisfy the `--constraint` expression, for example, a feature that no node has or a contradictory combination. | Run `sinfo -o "%P %f"` to list features by partition. Fix or remove the `--constraint` expression. | | `PartitionConfig` | The request exceeds a static partition limit (for example, node count, CPU count, or memory per node). | Compare your request to the partition limits in `scontrol show partition [NAME]`. Reduce the request or choose a partition that can satisfy it. | | `PartitionTimeLimit` | The job's `--time` exceeds the partition maximum. | Lower `--time`, or submit to a partition with a higher time limit. | | `PartitionNodeLimit` | The job's `--nodes` exceeds the partition node limit. | Lower `--nodes`, or choose a different partition. | To compare your request against partition limits: ```bash theme={"system"} # Show a partition's limits (MaxNodes, MaxTime, MaxMemPerNode, and so on). Read-only. scontrol show partition [PARTITION-NAME] ``` ## Dependency and held reasons These reasons mean the job is intentionally not eligible to run yet. | Reason | What it means | What to do | | -------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Dependency` | You submitted the job with `--dependency`, and the job it waits on hasn't reached the required state. | Confirm the dependency job's state. The job runs once the dependency is satisfied. | | `DependencyNeverSatisfied` | The dependency can never be met, for example, the prerequisite job failed when the dependency required success. | Cancel and resubmit. By default the job stays pending indefinitely, unless your cluster sets `DependencyParameters=kill_invalid_depend`, which cancels it for you. | | `JobHeldUser` | You placed the job on hold with `scontrol hold`. | Release it with `scontrol release [JOB-ID]` when you're ready. | | `JobHeldAdmin` | An administrator placed the job on hold. | Contact the administrator who set the hold before releasing. Don't release an administrator hold without confirming why it was set. | ## Idle nodes but still pending If `sinfo` shows idle nodes but your job still waits with `Resources`, `Priority`, or `ReqNodeNotAvail`, the idle nodes usually can't satisfy this specific request. Work through these causes in order. 1. **Partition mismatch.** The idle nodes are in a different partition than the one your job targets. Confirm with `sinfo` which partition holds idle nodes, then submit to that partition with `--partition`. 2. **Feature or constraint mismatch.** The idle nodes lack a feature your `--constraint` requires. List features per partition with `sinfo -o "%P %t %f"` and reconcile against your constraint. 3. **Topology constraints.** With [topology-aware scheduling](/products/sunk/optimize_workloads/topology-scheduling), a job that requests `--exclusive=topo` or a `--segment` waits until a whole block (or segment) is free. Idle nodes spread across several blocks don't satisfy a single-block request. After drains, a block may temporarily hold fewer than its full node count, so a block-sized request waits. 4. **Gang or all-or-nothing requirement.** A multi-node job needs all requested nodes at once. A few idle nodes scattered across the cluster can't start an N-node job until N matching nodes are free together. 5. **Reserved or blocked nodes.** Nodes in `RESERVED` or `BLOCKED` state appear in `sinfo` but aren't schedulable for your job. See [Monitor Slurm node states](/products/sunk/manage_sunk/slurm-node-states). 6. **Backfill can't plan around an unlimited time limit.** A running job with no `TimeLimit` has no predictable end time, so backfill never evaluates lower-priority jobs behind it. Your job may show `Reason=Priority` with `StartTime=Unknown` and `Scheduler=Main` even when nodes are idle. Set an accurate `--time` on every job, or see [Job priority, QoS, and preemption](/products/sunk/run_workloads/job-priority-qos-preemption#backfill-starvation-from-a-job-with-no-time-limit). ### "Requested node configuration is not available" This message (it can appear as a job reason or as an `sbatch` or `srun` rejection) means no node matches the exact combination of resources, features, and constraints the job requested. The following are common causes: * **A partition and feature combination that no node satisfies.** For example, a `--constraint` feature that exists, paired with a partition whose nodes lack it. List features with `sinfo -o "%P %f"` and align the request. * **A `--constraint` expression with a typo or an impossible combination** (for example, two mutually exclusive features joined with `&`). * **`--segment` plus `--exclude` math.** When you request a segment size but exclude enough nodes that no remaining block can supply a full segment, the request becomes unsatisfiable. Reduce the segment size, reduce exclusions, or wait for more nodes to free up. See [topology-aware scheduling](/products/sunk/optimize_workloads/topology-scheduling). * **A request that exceeds per-node hardware**, for example, more GPUs or memory per node than any node in the partition provides. Compare the request against `sinfo -o "%P %c %m %G"`. ```bash theme={"system"} # List partition, CPUs, memory, and GRES per node config. Read-only. sinfo -o "%P %c %m %G" ``` ```bash theme={"system"} # List partition and available features per node. Read-only. sinfo -o "%P %f" ``` ## What this is not Keep these distinctions in mind: * A `PENDING` job isn't a failed job. Jobs that fail show `F`, `NF`, or another terminal state, not `PD`. * `Priority` and `Resources` aren't errors. They mean the queue is working normally. Don't resubmit repeatedly. Resubmitting adds load to `slurmctld` without helping. * A quota reason (`AssocGrp*`, `QOS*`) isn't a hardware shortage. Free nodes can exist while your association is capped. ## When to file a support ticket Open a support ticket in any of these situations: * A job pends on `Resources` for an extended period while `sinfo` shows ample idle nodes in the correct partition with matching features, and none of the [idle nodes but still pending](#idle-nodes-but-still-pending) causes apply. * `ReqNodeNotAvail` persists because nodes stay drained or down and don't recover. See [Drain and undrain Slurm nodes](/products/sunk/manage_sunk/drain-and-undrain-nodes) first. * You believe an accounting limit is set incorrectly and your administrator can't adjust it. Include the output of `scontrol show job [JOB-ID]`, `sinfo`, and `sinfo -R` in the ticket. Confirm whether the issue is yours to resolve or CoreWeave's. ## Related pages * [Monitor Slurm job states](/products/sunk/manage_sunk/slurm-job-states): job state codes and the `REASON` column. * [Drain and undrain Slurm nodes](/products/sunk/manage_sunk/drain-and-undrain-nodes): why nodes that would run your job are unavailable. * [Topology and block scheduling in Slurm](/products/sunk/optimize_workloads/topology-scheduling): block, segment, and `--exclusive=topo` behavior. * [SUNK troubleshooting](/support/sunk): all SUNK triage flows. [Workload Scheduling](/support/sunk/tags/workload-scheduling) ## Related topics - [Job priority, QoS, and preemption](/products/sunk/run_workloads/job-priority-qos-preemption.md)