srun --exclusive gives your job the whole node, but when the same srun command also starts the job step and you set --ntasks, Slurm limits the step to the CPUs those tasks request. With --ntasks=1, the step has one task, and Slurm allocates one CPU per task by default, so the step runs on a single core and every other core on the node sits idle. CPU-heavy work in the step, such as compiling code or reading and writing data, slows down to what one core can do.
To give the step every CPU on the node, remove --ntasks=1 or add --cpus-per-task.
What you see
A job started with a command like the following runs slowly, and tools such astop show one core busy while the rest are idle:
scontrol show job shows that the job holds every CPU on the node (AllocTRES=cpu=128 on an H100 node) and requested one CPU per task (CPUs/Task=1, ReqTRES=cpu=1). These fields describe the job, not the step, and they don’t change after you remove --ntasks=1. To see what the step can use, check its CPU affinity inside the job. It lists only the two hardware threads of a single core instead of the whole node:
Example output
How to fix it
Choose one of the following options:-
Remove
--ntasks=1. This is the simplest fix. Without it, the step can use every CPU in the node allocation. -
Set
--cpus-per-taskto the node’s CPU count. Use this option if your workflow requires--ntasks=1. Replace[CPU-COUNT]with the value ofCPUTotfromscontrol show node [NODE], for example128on an H100 node:
grep Cpus_allowed_list /proc/self/status again inside the job. The list should cover every CPU on the node, for example 0-127.
Don’t use nproc to check the result. If OMP_NUM_THREADS is set in the job environment, nproc reports that value instead of the CPUs the step can run on. For more information, see Verify the binding worked.
For the full behavior of these options, see --exclusive and --cpus-per-task in the Slurm srun documentation.
Workload Scheduling