Skip to main content
This part of the SUNK training tutorial walks you through preparing a basic Slurm batch job script, submitting it to the cluster, monitoring its progress, and inspecting its output. By the end, you’ll be familiar with the core Slurm commands (sbatch, squeue, scancel, sacct, and sinfo) that you’ll use throughout the rest of the tutorial series to run real training workloads on SUNK.

​
Prepare the job script

The most common approach to running jobs on Slurm is to submit job scripts to Slurm. Scripts typically consist of three sections:
  1. Directives: Slurm directives begin with #SBATCH. These directives specify which resources and settings Slurm uses to run the script.
  2. Environment variables: Environment variables are the variables your application requires to run. In this section, load modules and set up environment variables such as $PATH and $LD_LIBRARY_PATH.
  3. Code: The actual code that runs your application.
The following example shows a basic Slurm job script, which sets Slurm directives using sbatch, then calls a Python program: 
#!/bin/bash

# SBATCH directives
#SBATCH --job-name=my_job # The name of your job
#SBATCH --output=my_job.out # The output file location of your job
#SBATCH --error=my_job.err # The error file location of your job
#SBATCH --time=01:00:00 # The maximum time your job can run. In this case, 1 hour
#SBATCH --nodes=1 # The number of nodes to use
#SBATCH --ntasks-per-node=1 # The number of tasks to run per node
#SBATCH --gres=gpu:1 # The number of GPUs to use

# Export environment variables
export VARIABLE1="variable_1"

echo "Starting job"
date
hostname

echo "Running calculation..."

# In this section, your application is called.
# For example, to run a Python script called `myprog.py`:
python3 /home/username/myprog.py

echo "Calculation complete"

echo "Ending job"
date

​
Sbatch directives

Use #SBATCH directives to configure the job submission. Each directive specifies a different aspect of the job, such as its name, output files, time limits, and resource requirements. The following table summarizes some common directives:
DirectiveDescription
#SBATCH --job-name=Sets the name of the job. Useful for locating it in status reports later
#SBATCH --output=Specifies the output file for standard output. You can use a full path, or one relative to the submission directory
#SBATCH --error=Specifies the output file for standard error
#SBATCH --time=Sets the time limit for the job. In the preceding example, 1 hour. If the job is still running after the specified duration, the job is stopped
#SBATCH --nodes=How many nodes the job should request. In the preceding example, 1 node
#SBATCH --ntasks-per-node=How many tasks per node this job should request. In the preceding example, 1 task per node
#SBATCH --gres=gpu:How many GPUs this task should request. In the preceding example, 1 GPU

​
Submit the job

With the job script prepared, you’re ready to hand it off to the Slurm scheduler so it can be queued and run on the cluster. Submit your job to the Slurm queue by running the script using sbatch. This command submits the my_job_script.sh script to the Slurm scheduler, and Slurm allocates resources to run your job based on the sbatch directives in the script. If resources matching these directives are available, the job may start immediately. Otherwise, it waits in a queue until other jobs finish.
Example command
sbatch my_job_script.sh
After job submission, Slurm returns a job ID. In this example, the job ID is 371. Keep this job ID. You’ll use it to check status, cancel the job, or look up results later.
Example output
Submitted batch job 371

​
Check the job status

Once a job is submitted, you’ll want to know whether it’s still queued, running, or already finished. Slurm exposes that information through the squeue command. To check the status of your job, use the squeue command. The squeue command shows the status of all jobs in the cluster. To see a specific job, target the job ID from the submission command. In this example, where the job ID is 371, locate the job by running squeue 371. You can also target jobs by the person who submitted them, identified by username, by running squeue -u myname. Running squeue without any arguments shows all jobs in the queue, including their status, time running, and the nodes they are running on. 
squeue
You should see output similar to the following: 
JOBID PARTITION     NAME     USER   ST      TIME   NODES NODELIST(REASON)
371        h100     my_job   user1  R       2:01      1   h100-208-101
372        h100     calc     user2  R       1:38      1   h100-208-153
363        h100     calc2    user3  R       1:25      1   h100-208-153
Job stateExplanation
PDJob is pending. The Nodelist(reason) column details why.
RJob is running. The Nodelist(reason) column shows which nodes the job uses.
CG/CDJob has completed or is completing.
FThe job has failed.
TOBecause of a time out, Slurm terminated the job. The time limit is usually specified in the job script or partition configuration.
For more detailed metrics on your job, view your job in the Slurm / Job Metrics Grafana Dashboard.

​
Cancel the job

If you submit a job in error or no longer need its results, you can stop it from running (or remove it from the queue) so it doesn’t consume cluster resources. To cancel your job, use the scancel command. 
scancel 371
If your job is pending, Slurm removes it from the queue. If your job is running, Slurm stops it.

​
View job output

After your job finishes, you’ll typically want to inspect what it produced and check for any errors. Slurm collects standard output and error output in the files specified by the sbatch directives in the job script. After your job completes, you can view the output in these files.
Use a text editor to view these files, since their contents may be lengthy.

​
View cluster information

Before submitting more jobs, it’s often useful to understand what nodes and partitions are available, and which are already busy. To view more details about your cluster, use the sinfo command.
Example command
sinfo
This example output displays information on a small cluster that consists of 3 nodes. The output includes the partition name, availability, time limit, number of nodes, state, and node list.
Example output
PARTITION      AVAIL  TIMELIMIT  NODES  STATE NODELIST
h100              up   infinite      1    mix h100-208-101
h100              up   infinite      1   idle h100-208-[101,103,125]
h100              up   infinite      1  alloc h100-208-153

​
Node states

StatusDescription
idleNodes are idle
allocNodes are fully allocated to jobs
mixNodes are running jobs, but still have some available space

​
View completed job history

Because squeue only shows recent jobs, you’ll need a different command to look up history for jobs that have already completed or failed. The squeue command only shows job status while jobs are in a pending or running state, and for a few hours after completion. To see the status of a completed or failed job, use the sacct command instead.
Example command
sacct
In this example output, the selected job completed successfully. The output shows the states of this simple job in three parts:
Example output
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
371              my_job       h100       root          1  COMPLETED      0:0
371.batch         batch                  root          1  COMPLETED      0:0
371.0        all_reduc+                  root          1  COMPLETED      0:0
  1. The line with a JobID of 371 and a JobName of my_job displays the state of the entire job.
  2. The line prefaced with a JobID of 371.batch and a JobName of batch displays the state of the submitted script.
  3. The final line, with a JobID of 371.0 and a JobName of all_reduc+, displays the status of one of the tests CoreWeave runs after each job to confirm the cluster performs correctly. The test typically takes a few seconds to run.
Busy clusters may contain thousands of jobs. It can be helpful to restrict the output to a specific time range or user, for example:To see jobs only from a specific user:
Example command
sacct --user=user1
To see jobs from a specific time range:
Example command
sacct --starttime=2025-03-24T06:10 --endtime=2025-03-24T07:20
The --starttime option displays only jobs that have begun after the given start time. The --endtime option displays only jobs that have ended before the given end time. Specify all times in ISO format.
Last modified on May 27, 2026