Skip to main content
On CoreWeave SUNK, Slurm priority and preemption are upstream Slurm features. SUNK doesn’t change how they work. What surprises users is what SUNK leaves alone: the operator defaults set no PriorityWeight* values and no PreemptType, so a new cluster has flat priority and no preemption until an administrator configures them. If your job sits PENDING, start with Why is my Slurm job stuck in PENDING?, which maps every REASON code to a next step. Return here when you need to understand why ordering is flat, how preemption is scoped, or why backfill never evaluates your job.

​
Symptoms

A job sits PENDING with Reason=Priority or Reason=Resources while other jobs run, or a running job gets preempted and seems to vacate slowly. This page explains the scheduling model behind those outcomes.

​
How Slurm priority works on SUNK

Slurm uses a multifactor priority formula to order PENDING jobs:
The factors include job age (how long the job has waited), fairshare (your account’s historical resource use), QoS, partition tier, and job size. Each factor has a configurable weight (PriorityWeightAge, PriorityWeightFairshare, PriorityWeightQOS, PriorityWeightPartition, PriorityWeightJobSize). Every PriorityWeight* factor defaults to 0 in Slurm, and the SUNK operator defaults don’t set any of them. The multifactor plugin is still running, because PriorityType defaults to priority/multifactor. Every job just scores the same, so Slurm falls through to its remaining tie-breaks: jobs that can preempt, jobs with an advanced reservation, partition PriorityTier, job priority, submit time, then job ID. In practice that means submission order. Set only PriorityWeightQOS to a non-zero value and QoS becomes the only factor that changes the order. QoS priority and preemption are separate concerns. PriorityWeightQOS affects which PENDING jobs get resources first. It doesn’t control which running jobs get preempted. The Preempt field on each QoS controls preemption, as described in Preemption on SUNK. For accounting setup that feeds fairshare and priority, see How do I set up Slurm accounting for priority scheduling?.

​
Inspect priorities

Use these read-only commands from a login pod to see why a job has the priority it does.
In the squeue output, the %Q (priority) and %q (QoS) columns tell you how a job ranks and which QoS it uses.

​
Preemption on SUNK

Preemption lets a higher-priority job reclaim resources from a lower-priority job. Changing PreemptType, PreemptMode, or QoS Preempt fields requires cluster-admin access. PreemptType selects the plugin that decides what can preempt what. Slurm leaves it unset by default and the SUNK operator defaults don’t set it either, so preemption is off until an administrator picks a plugin:
  • preempt/qos: QoS definitions control which QoS can preempt which others. A QoS can preempt another only if the other QoS appears in its Preempt field.
  • preempt/partition_prio: partition PriorityTier controls preemption. Jobs in a higher PriorityTier partition may preempt jobs in a lower one. This is independent of PriorityWeightPartition, so preemption can work even when scheduling priority weights are zero.
PreemptMode selects what happens to the preempted job:
  • OFF: no preemption. This is the default, and it’s compatible only with preemption disabled cluster-wide.
  • CANCEL: Slurm cancels the preempted job.
  • REQUEUE: Slurm returns the preempted job to the queue to run again later. The job must be submitted with --requeue or the cluster must set JobRequeue=1. Design the job to be safe to re-run from a checkpoint.
  • SUSPEND: Slurm suspends the preempted job in place.
Slurm honors a per-partition PreemptMode only under PreemptType=preempt/partition_prio. Under PreemptType=preempt/qos it reads PreemptMode from the QoS instead, so PreemptMode=OFF on a partition does nothing to stop QoS-driven preemption there. See PreemptMode in the slurm.conf reference. That mismatch is the usual reason a partition you believe is preemption-free still loses jobs. Check the plugin first, then the partition and QoS settings:
For background on GANG preemption and other modes, see the Slurm preemption documentation. SUNK doesn’t add product-specific behavior to those modes.

​
Why a preempted job vacates slowly

Two settings govern the gap between “Slurm selected this job for preemption” and “its nodes are free.” Both are cluster-admin settings. GraceTime is the preemption grace window. Slurm moves the job’s end time to now plus GraceTime and sends SIGCONT and SIGTERM right away as notice. Partitions and QoS levels each carry their own value, both default to 0, and each has a condition: a partition’s GraceTime applies only under PreemptType=preempt/partition_prio, and a QoS GraceTime applies only to PreemptMode=CANCEL and PreemptMode=REQUEUE. KillWait governs the kill that follows. At the job’s new end time Slurm sends SIGCONT and SIGTERM, waits KillWait seconds, then sends SIGKILL. On SUNK, KillWait defaults to 30 seconds. A large KillWait makes working preemption look broken: the preempting job starts, and the preempted job sits in COMPLETING until the window closes.
To vacate quickly under preemption, trap SIGTERM in your job and checkpoint and exit promptly. For the full signal sequence and a worked handler, see Handle Slurm signals.

​
Request a higher QoS

To request a higher-priority QoS at submission, if your account is authorized for it:
If your job’s priority is too low to ever schedule and you can’t change weights, contact your cluster administrator. Raising priority weights, adjusting QoS priority values, or moving a job to an unpreemptible partition are administrator actions. For the upstream reference, see the Slurm priority documentation.

​
Backfill starvation from a job with no time limit

The backfill scheduler fits lower-priority jobs into gaps before higher-priority jobs are ready. To do that, it predicts when running jobs finish, based on each job’s TimeLimit. A job with an unlimited time limit has no predictable end time, so backfill can’t plan around it. Lower-priority jobs behind it are never backfilled. The signature of this problem is a job that stays PENDING even though nodes are idle:
StartTime=Unknown with Scheduler=Main means only the main scheduler considers the job, and the main scheduler enforces strict priority order. The fix is to set an accurate TimeLimit on every job so backfill can plan:
This matters most for many same-priority single-node jobs, which rely on backfill to pack onto idle nodes.

​
Common scenarios

The following scenarios show how priority and preemption interact in practice. Scavenger workloads. A scavenger job uses a low-priority QoS that higher-priority QoS levels can preempt. It runs on otherwise idle resources and yields when higher-priority work arrives. This requires the higher-priority QoS levels to list the scavenger QoS in their Preempt field. Development versus production tiers. Administrators commonly define a tier of QoS levels (for example, a production QoS that can’t be preempted and a lower QoS for development jobs). The exact layout is cluster-specific. Use sacctmgr show qos to see what your cluster defines. Over-weighting QoS causes starvation. When PriorityWeightQOS is the only non-zero weight and is set high, small differences in QoS priority values become large effective gaps. Lower-priority jobs can then sit PENDING with Reason=Resources even when nodes are idle. The fix is to balance the weights: reduce PriorityWeightQOS, adjust the relative spacing of QoS priority values, or enable other factors such as age and fairshare.

​
When to file a support ticket

Open a ticket when scheduling behavior still doesn’t match your configuration after you have confirmed the priority weights, QoS definitions, PreemptType, PreemptMode, GraceTime, and KillWait. Include the output of scontrol show job [JOB-ID], sprio -l, sacctmgr show qos, and scontrol show config | grep -E 'Priority|Preempt|KillWait', along with the partition’s GraceTime.
Last modified on September 16, 2026