Documentation Index
Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
Use this file to discover all available pages before exploring further.
scrontab is a utility designed for scheduling recurring Slurm batch jobs in a High-Performance Computing (HPC) Slurm cluster. It uses similar syntax to cron, but has additional functionality specific to Slurm, making it a suitable option for scheduling complex and resource-intensive jobs.
scrontab uses the sbatch mechanism to submit Slurm jobs at specified intervals, where they are then managed and executed by the Slurm workload manager. The Slurm workload manager dynamically allocates resources, queues jobs, and distributes jobs across the cluster.
Preliminary steps
- Enable
scrontabin the cluster by addingScronParameters=enableto theslurmConfig.extraConfigsection. See the Slurm parameter reference for a list of available parameters. - Enable the ability to run additional syscalls required by
scrontabby adding the “SYS_ADMIN” security context capability. See the Slurm parameter reference. - Connect to the login node. For detailed instructions, see Connect to the Slurm Login Node.
- Ensure that scripts intended to be used with
scrontabare executable by runningchmod +x <script>. - Ensure that scripts intended to be used with
scrontabare in a shared location.
Edit the scrontab file
To edit the scrontab file, execute the following command:
scrontab file does not already exist, scrontab -e will provide a default, unconfigured example for you to edit.
scrontab behavior and considerations
scrontab behaves differently than cron and #SBATCH directives in several ways. To ensure a smooth scheduling experience, remember the following concepts:
- Options set using
#SCRONwill be reset after eachscrontabentry. - The entered times specify when a job becomes eligible to run. The actual execution time is dependent on Slurm’s scheduler and resource availability.
#SCRONdirectives configure the Slurm job when it is submitted. Thus, Slurm is likely to ignore#SBATCHdirectives within the script itself.- When
scrontabjobs are canceled, their future executions are also canceled. When an entirescrontabfile is deleted usingscrontab -r, running jobs which were defined in that file will complete but will not be rescheduled afterwards. scrontabjobs retain theirJobIDacross executions. If a job is still running when its next scheduled time arrives, new instances will not be created.- Jobs can be rescheduled based on the times specified in the
scrontabentries usingscontrol requeue <JobID>. - The time specified in
scrontabis relative to the Slurm controller’s timezone, not your local time.
Apply #SCRON directives
To apply #SCRON options to an entry, add a line beginning with #SCRON before the given entry. Options set with #SCRON will apply only to the single following crontab entry.
Example scrontab directive
#SBATCH directives are also available to use as #SCRON directives. For a complete list of options, see SchedMD’s sbatch documentation.
These are some example options which are frequently useful:
#SCRON --chdir=<dir>sets the directory the job will be executed in.#SCRON --time=<time>sets the time limit for the job.#SCRON --output=<file>sets the file location for capturingstdoutandstderroutputs. The default location for these is/root/slurm-<JobID>.out.
Scheduling
The entered time specifies when a job becomes eligible to run. The actual execution time depends on Slurm’s scheduler and resource availability.
cron. Times are defined in one line, using five fields to define the minute, hour, day of month, month, and day of week respectively.
The format is as follows:
Example scrontab directive
| Field | Values |
|---|---|
<minute> | (0-59) |
<hour> | (0-23) |
<day of month> | (1-31) |
<month> | 1-12 or Jan-Dec |
<day of week> | (0-7) where 0 and 7 are Sunday |
| Special Character | Name | Behavior |
|---|---|---|
* | Wildcard | Matches all values for that field |
, | Comma | Specifies a list of values |
- | Hyphen | Specifies a range of values |
/ | Slash | Specifies step values |
scrontab also includes aliases for common settings. For example, the @hourly alias will schedule a job to become eligible at the first minute of each hour. For more examples, see SchedMD’s scrontab documentation.
Rescheduling
To skip the next run of acron job and reschedule it for the next available time, use scontrol requeue, as shown below:
Examples
This section includes examples ofscrontab scheduling options and demonstrates how they fit with #SCRON directives.
Recording timestamped job statistics
The following code block records data for selected jobs every other hour, and saves the log files to a specified location:Example scrontab directive
#SCRON directives
This command begins with a #SCRON directive:
Example #SCRON directive
--job-name specifies the name for the slurm job. In this example, we name the job "log_jobs". This name will appear in Slurm’s account systems, including sacct and squeue, making it easier to track and identify.
Scheduling
Next, the command specifies the scheduling behavior of this job:Example scheduling syntax
0 in the <minute> field specifies that the job should run exactly on the hour.
A value of */2 in the <hour> field specifies that this will execute every other hour. * indicates “every hour”, but adding /2 indicates “every 2nd occurrence”. Thus, every other hour.
The * in the <day of month>, <month>, and <day of week> fields specify that this job will run every day of every month.
Slurm command
After the scheduling specifications, we have the actual command to be executed as a Slurm job. In this example, we usesacct. The sacct command is a Slurm CLI utility that provides detailed information about past jobs, including resource usage, status, and more.
Example Slurm command with options
--jobs=[JOB-ID-LIST] option for sacct specifies the job to retrieve information about. Replace [JOB-ID-LIST] with the job IDs you would like to query. This can be done by listing specific job IDs or by providing a range of job IDs.
The --format= option specifies what fields of information you want sacct to provide for each job. The names of these fields correspond with column names.
In this example, we use --format= to direct sacct to return the following information:
| Field | Content |
|---|---|
JobID | The unique identifier for the job. |
AllocCPUS | The number of CPUs allocated to the specified job. |
ConsumedEnergy | The amount of energy consumed by the job, in Joules. To see this metric, you must have energy accounting enabled in your Slurm configuration. |
NTasks | The number of tasks, or processes, within the job. |
Save the logs to a specified location
To save the requested logs to a specified location, use> to redirect the output of sacct to a specified directory and use the date command to create a logfile with the current date and time.
Example logfile
| Option | Format |
|---|---|
%Y | Displays the full year. |
%m | Displays the month, (01-12) |
%d | Day of the month, (01-31) |
%H | Hour, (00-23) |
%M | Minute, (00-59) |
\ | An escape character, used before the %, to prevent scrontab from reading % as a newline character. |
.log | The file extension. |
/logs/jobs/20250626-1600.log
Using time shortcuts and additional sbatch options
@teatime is an alias provided by scrontab. It specifies that the job will become eligible at 16:00 (4:00 PM) each day.