This is an old revision of the document!

Scheduling a Job using the SLURM job scheduler

Intro to SLURM slides

The Computer Science Department uses a “job scheduler” called SLURM. The purpose of a job scheduler is to allocate computational resources (servers) to users who submit “jobs” to a queue. The job scheduler looks at the requirements stated in the job's script and allocates to the job a server (or servers) which matches the requirements specified in the job script. For example, if the job script specifies that a job needs 192GB of memory, the job scheduler will find a server with at least that much memory free.

Information Gathering

SLURM provides a set of tools to use for interacting with the scheduling system. To view information about compute nodes in the SLURM system, we can use the command sinfo.

[pgh5a@portal04 ~]$ sinfo
main*        up   infinite      3  drain falcon[3-5]
main*        up   infinite     27   idle cortado[01-10],falcon[1-2,6-10],lynx[08-12],slurm[1-5]
gpu          up   infinite      4    mix ai[02-03,05],lynx07
gpu          up   infinite      1  alloc ai04
gpu          up   infinite     12   idle ai[01,06],lynx[01-06],ristretto[01-04]

With sinfo we can see a listing of the job queues or “partitions” and a list of nodes associated with these partitions. A partition is a grouping of nodes, for example our main partition is a group of all general purpose nodes, and the gpu partition is a group of nodes that each contain GPUs. Sometimes hosts can be listed in two or more partitions.

To view jobs running on the queue, we can use the command squeue. Say we have submitted one job to the main partition, running squeue will look like this:

pgh5a@portal01 ~ $ squeue
            467039      main    my_job    pgh5a  R      0:06      1 artemis1

and now that a node has been allocated, that node artemis1 will show as alloc in sinfo

pgh5a@portal01 ~ $ sinfo
main*            up   infinite     37   idle hermes[1-4],artemis[2-7],slurm[1-5],nibbler[1-4],trillian[1-3],granger[1-6],granger[7-8],ai0[1-6]
main*            up   infinite      1  alloc artemis1
qdata            up   infinite      8   idle qdata[1-8]
qdata-preempt    up   infinite      8   idle qdata[1-8]
falcon           up   infinite     10   idle falcon[1-10]
intel            up   infinite     24   idle artemis7,slurm[1-5],granger[1-6],granger[7-8],nibbler[1-4],ai0[1-6]
amd              up   infinite     13   idle hermes[1-4],artemis[1-6],trillian[1-3]


To use SLURM resources, you must submit your jobs (program/script/etc.) to the SLURM controller. The controller will then send your job to compute nodes for execution, after which time your results will be returned.

Users can submit SLURM jobs from From a shell, you can submit jobs using the commands srun or sbatch. Let's look at a very simple example script and sbatch command.

Here is our script, all it does is print the hostname of the server running the script. We must add SBATCH options to our script to handle various SLURM options.

#SBATCH --job-name="Slurm Simple Test Job" #Name of the job which appears in squeue
#SBATCH --mail-type=ALL
#SBATCH --error="my_job.err"                    # Where to write std err
#SBATCH --output="my_job.output"                # Where to write stdout
#SBATCH --nodelist=slurm1

Let's put this in a directory called slurm-test in our home directory. We run the script with sbatch and the results will be put in the file we specified with --output. If no output file is specified, output will be saved to a file with the same name as the SLURM jobid.

pgh5a@portal01 ~ $ cd slurm-test/ 
pgh5a@portal01 ~/slurm-test $ chmod +x 
pgh5a@portal01 ~/slurm-test $ sbatch 
Submitted batch job 466977
pgh5a@portal01 ~/slurm-test $ ls
my_job.err  my_job.output
pgh5a@portal01 ~/slurm-test $ cat my_job.output 

Here is a similar example using srun running on multiple nodes:

pgh5a@portal01 ~ $ srun -w slurm[1-5] -N5 hostname

Terminating Jobs

Please be aware of jobs you start and make sure that they finish executing. If your job does not exit gracefully, it will continue running on the server, taking up resources and preventing others from running their jobs.

To cancel a running job, use the scancel [jobid] command

abc1de@portal01 ~ $ squeue
            467039      main    sleep    abc1de  R       0:06      1 artemis1           <--  Running job
abc1de@portal01 ~ $ scancel 467039


Slurm refers to job queues as partitions. These queues can have unique constraints such as compute nodes, max runtime, resource limits, etc. There is a main queue, which will make use of all non-GPU compute nodes.

Partition is indicated by -p partname or --partition partname.

To specify a partition with sbatch file:

#SBATCH --partition=gpu

Or from the command line with srun

-p gpu

An example running the command hostname on the main partition, this will run on any node in the partition:

srun -p main hostname


Slurm handles GPUs and other non-CPU computing resources using what are called GRES Resources (Generic Resource). To use the GPU(s) on a system using Slurm, either using sbatch or srun, you must request the GPUs using the --gres:x option. You must specify the gres flag followed by : and the quantity of resources

Say we want to use 4 GPUs on a system, we would use the following sbatch option:

#SBATCH --gres=gpu:4

Or from the command line


Direct login to servers (without a job script)

You can use srun to login directly to a server controlled by the SLURM job scheduler. This can be useful for debugging purposes as well as running your applications without using a job script. Creating a login session also reserves the node for your exclusive use.

To spawn a shell we must pass the --pty option to srun so output is directed to a pseudo-terminal:

pgh5a@portal ~$ srun -w slurm1 --pty bash -i -l -
pgh5a@slurm1 ~$ hostname
pgh5a@slurm1 ~$

The -i argument tells bash to run as interactive. The -l arg instructs bash that this is a login shell, this, along with the final - are important to reset environment variables that otherwise might cause issues using Environment Modules

If a node is in a partition other than the default “main” partition (for example, the “gpu” partition), then you must specify the partition in your command, for example:

pgh5a@portal ~$ srun -w lynx05 -p gpu --pty bash -i -l -

Reservations for specific resources or nodes can be made by submitting a request to For more information about using reservations, see the main article on SLURM Reservations

Due to the way sbatch spawns a bash session (non-login session), some init files are not loaded from /etc/profile.d. This prevents the initialization of the Environment Modules system and will prevent you from loading software modules.

To fix this, simply include the following line in your sbatch scripts:

source /etc/profile.d/
  • compute_slurm.1598406173.txt.gz
  • Last modified: 2020/08/26 01:42
  • by pgh5a