Why is my Slurm job stuck in PENDING?

A Slurm job stays in PENDING (PD) when the scheduler has accepted the job but can’t start it yet. Run squeue -u $USER and read the REASON column to learn whether to wait, change your submission, or escalate. Common cases: Resources and Priority mean the queue is working and you should wait, AssocGrp* and QOS* mean an accounting limit blocks the job, and configuration reasons such as BadConstraints or PartitionConfig mean you must change the request. A pending job isn’t an error on its own. For why a job waits and what the REASON column means, see Monitor Slurm job states. This article maps each REASON value to a concrete action.

What you see

squeue shows your job in state PD with a value in the REASON (or NODELIST(REASON)) column. Read it with squeue -u $USER:

# Read the state and reason for your jobs. Read-only.
squeue -u $USER

Example output

   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  123456   defq      train  alice   PD       0:00      4 (Resources)
  123457   defq      train  alice   PD       0:00      4 (AssocGrpGRES)

To see the full reason and the job’s exact request, use scontrol:

# Show the full job record, including TRES request, partition, and reason. Read-only.
scontrol show job [JOB-ID]

For background on the job lifecycle, see Monitor Slurm job states.

Decision tree

Find your REASON value in the sections that follow. Reasons fall into five groups: resource, quota and account, configuration, dependency and held, and the “Requested node configuration is not available” message that has several distinct causes.

If the reason is Resources, Priority, BeginTime, Reservation, or ReqNodeNotAvail, see Resource and timing reasons.
If the reason starts with Assoc, QOS, or is InvalidAccount, see Quota and account reasons.
If the reason is BadConstraints, PartitionConfig, PartitionTimeLimit, or PartitionNodeLimit, see Configuration reasons.
If the reason is Dependency, JobHeldUser, or JobHeldAdmin, see Dependency and held reasons.
If nodes look idle but the job still waits, or you see Requested node configuration is not available, see Idle nodes but still pending.

Resource and timing reasons

These reasons mean the scheduler accepted your request and the job waits its turn. Usually, the correct action is to wait.

Reason	What it means	What to do
`Resources`	The cluster doesn’t have enough free resources to run the job.	Wait. If it persists, confirm idle capacity exists (`sinfo`) and check for idle nodes but still pending.
`Priority`	Higher-priority jobs are ahead of yours in the queue. The job runs once they complete or free resources.	Wait. To understand ordering, run `sprio -j [JOB-ID]` and `scontrol show job [JOB-ID]`.
`BeginTime`	The job has a start time in the future, set with `--begin`.	Wait until the start time, or resubmit without `--begin`.
`Reservation`	The job targets a Slurm reservation that hasn’t started, or it’s excluded from a reservation that holds nodes.	Check the reservation window with `scontrol show reservation`. Submit with the correct `--reservation` name, or wait for the reservation to start.
`ReqNodeNotAvail`	A node the job requires (by name, feature, or reservation) is unavailable, for example, drained or down.	Run `sinfo -R` to list unavailable nodes and their reasons. If you pinned specific nodes with `--nodelist`, remove the constraint or wait for those nodes to return. See Drain and undrain Slurm nodes.

To confirm the cluster has idle capacity:

# Summarize node states by partition. Read-only.
sinfo

# List unavailable nodes and the reason each is unavailable. Read-only.
sinfo -R

Quota and account reasons

These reasons mean a Slurm accounting limit on your association (the combination of user, account, partition, and QoS) blocks the job. The job waits until usage drops below the limit, or until an administrator raises the limit. If you’re at a hard cap and nothing else is running, waiting alone doesn’t help.

Reason	What it means	What to do
`AssocGrpGRES`	Your association has reached its group limit on a generic resource (GRES), most often GPUs.	Wait for your other jobs to release GPUs, or request fewer GPUs. If the limit is too low for your work, ask an administrator to adjust it.
`AssocGrpCpuLimit`	Your association has reached its group CPU limit.	Wait for running jobs to finish, or request fewer CPUs.
`AssocGrpNodeLimit`	Your association has reached its group node-count limit.	Reduce `--nodes`, or wait for your other allocations to release nodes.
`AssocMaxJobsLimit`	Your association is at its maximum number of running jobs.	Wait for a running job to finish.
`QOSMaxJobsPerUserLimit`	The job’s QoS limits how many jobs one user can run at once, and you’re at the limit.	Wait, or submit under a different QoS if one is available to you.
`QOSMaxGRESPerUser`	The job’s QoS limits GPUs per user, and you’re at the limit.	Wait, or request fewer GPUs.
`InvalidAccount`	The account on the job doesn’t exist or your user isn’t a member of it.	Confirm your accounts with `sacctmgr show assoc user=$USER`. Submit with a valid `--account`. If your user is missing from accounting, see nsscache.

To inspect your own limits and current usage:

# Show your accounting associations and their limits. Read-only.
sacctmgr show assoc user=$USER format=Account,Partition,QOS,GrpTRES,MaxJobs

A cluster administrator sets group GRES and CPU limits. If a limit is consistently too low for your workload, raise it with your administrator rather than retrying.

Configuration reasons

These reasons mean the job’s request conflicts with the partition or constraint configuration. Waiting doesn’t help. You must change the submission or the partition.

Reason	What it means	What to do
`BadConstraints`	No node can satisfy the `--constraint` expression, for example, a feature that no node has or a contradictory combination.	Run `sinfo -o "%P %f"` to list features by partition. Fix or remove the `--constraint` expression.
`PartitionConfig`	The request exceeds a static partition limit (for example, node count, CPU count, or memory per node).	Compare your request to the partition limits in `scontrol show partition [NAME]`. Reduce the request or choose a partition that can satisfy it.
`PartitionTimeLimit`	The job’s `--time` exceeds the partition maximum.	Lower `--time`, or submit to a partition with a higher time limit.
`PartitionNodeLimit`	The job’s `--nodes` exceeds the partition node limit.	Lower `--nodes`, or choose a different partition.

To compare your request against partition limits:

# Show a partition's limits (MaxNodes, MaxTime, MaxMemPerNode, and so on). Read-only.
scontrol show partition [PARTITION-NAME]

Dependency and held reasons

These reasons mean the job is intentionally not eligible to run yet.

Reason	What it means	What to do
`Dependency`	You submitted the job with `--dependency`, and the job it waits on hasn’t reached the required state.	Confirm the dependency job’s state. The job runs once the dependency is satisfied.
`DependencyNeverSatisfied`	The dependency can never be met, for example, the prerequisite job failed when the dependency required success.	Cancel and resubmit. By default the job stays pending indefinitely, unless your cluster sets `DependencyParameters=kill_invalid_depend`, which cancels it for you.
`JobHeldUser`	You placed the job on hold with `scontrol hold`.	Release it with `scontrol release [JOB-ID]` when you’re ready.
`JobHeldAdmin`	An administrator placed the job on hold.	Contact the administrator who set the hold before releasing. Don’t release an administrator hold without confirming why it was set.

Idle nodes but still pending

If sinfo shows idle nodes but your job still waits with Resources, Priority, or ReqNodeNotAvail, the idle nodes usually can’t satisfy this specific request. Work through these causes in order.

Partition mismatch. The idle nodes are in a different partition than the one your job targets. Confirm with sinfo which partition holds idle nodes, then submit to that partition with --partition.
Feature or constraint mismatch. The idle nodes lack a feature your --constraint requires. List features per partition with sinfo -o "%P %t %f" and reconcile against your constraint.
Topology constraints. With topology-aware scheduling, a job that requests --exclusive=topo or a --segment waits until a whole block (or segment) is free. Idle nodes spread across several blocks don’t satisfy a single-block request. After drains, a block may temporarily hold fewer than its full node count, so a block-sized request waits.
Gang or all-or-nothing requirement. A multi-node job needs all requested nodes at once. A few idle nodes scattered across the cluster can’t start an N-node job until N matching nodes are free together.
Reserved or blocked nodes. Nodes in RESERVED or BLOCKED state appear in sinfo but aren’t schedulable for your job. See Monitor Slurm node states.
Backfill can’t plan around an unlimited time limit. A running job with no TimeLimit has no predictable end time, so backfill never evaluates lower-priority jobs behind it. Your job may show Reason=Priority with StartTime=Unknown and Scheduler=Main even when nodes are idle. Set an accurate --time on every job, or see Job priority, QoS, and preemption.

”Requested node configuration is not available”

This message (it can appear as a job reason or as an sbatch or srun rejection) means no node matches the exact combination of resources, features, and constraints the job requested. The following are common causes:

A partition and feature combination that no node satisfies. For example, a --constraint feature that exists, paired with a partition whose nodes lack it. List features with sinfo -o "%P %f" and align the request.
A --constraint expression with a typo or an impossible combination (for example, two mutually exclusive features joined with &).
--segment plus --exclude math. When you request a segment size but exclude enough nodes that no remaining block can supply a full segment, the request becomes unsatisfiable. Reduce the segment size, reduce exclusions, or wait for more nodes to free up. See topology-aware scheduling.
A request that exceeds per-node hardware, for example, more GPUs or memory per node than any node in the partition provides. Compare the request against sinfo -o "%P %c %m %G".

# List partition, CPUs, memory, and GRES per node config. Read-only.
sinfo -o "%P %c %m %G"

# List partition and available features per node. Read-only.
sinfo -o "%P %f"

What this is not

Keep these distinctions in mind:

A PENDING job isn’t a failed job. Jobs that fail show F, NF, or another terminal state, not PD.
Priority and Resources aren’t errors. They mean the queue is working normally. Don’t resubmit repeatedly. Resubmitting adds load to slurmctld without helping.
A quota reason (AssocGrp*, QOS*) isn’t a hardware shortage. Free nodes can exist while your association is capped.

When to file a support ticket

Open a support ticket in any of these situations:

A job pends on Resources for an extended period while sinfo shows ample idle nodes in the correct partition with matching features, and none of the idle nodes but still pending causes apply.
ReqNodeNotAvail persists because nodes stay drained or down and don’t recover. See Drain and undrain Slurm nodes first.
You believe an accounting limit is set incorrectly and your administrator can’t adjust it.

Include the output of scontrol show job [JOB-ID], sinfo, and sinfo -R in the ticket. Confirm whether the issue is yours to resolve or CoreWeave’s.

Monitor Slurm job states: job state codes and the REASON column.
Drain and undrain Slurm nodes: why nodes that would run your job are unavailable.
Topology and block scheduling in Slurm: block, segment, and --exclusive=topo behavior.
SUNK troubleshooting: all SUNK triage flows.

Workload Scheduling

​What you see

​Decision tree

​Resource and timing reasons

​Quota and account reasons

​Configuration reasons

​Dependency and held reasons

​Idle nodes but still pending

​”Requested node configuration is not available”

​What this is not

​When to file a support ticket

​Related pages