skip to primary navigationskip to content
 

Submitting jobs

Sample submission scripts

In normal use of SLURM, one creates a batch job which is a shell script containing the set of commands to run, plus the resource requirements for the job which are coded as specially formatted shell comments at the top of the script. The batch job script is then submitted to SLURM with the sbatch command. A job script can be resubmitted with different parameters (e.g. different sets of data or variables).

Please copy and edit the sample submission scripts that can be found under

/usr/local/Cluster-Docs/SLURM

New user accounts also have symbolic links to template files in their home directories. Lines beginning #SBATCH are directives to the batch system. The rest of each directive specifies arguments to the sbatch command. SLURM stops reading directives at the first executable (i.e. non-blank, and doesn't begin with #) line.

The main directives to modify are as in the following:

#! Which project should be charged:
#SBATCH -A MYPROJECT-CPU
#! Which partition/cluster am I using?
#SBATCH -p skylake #! How many nodes should be allocated? If not specified SLURM
#! assumes 1 node. #SBATCH --nodes=2 #! How many tasks will there be in total? By default SLURM
#! will assume 1 task per node and 1 CPU per task. #SBATCH --ntasks=64 #! How much memory in MB is required _per node_? Not setting this #! as here will lead to a default amount per task. #! Setting a larger amount per task increases the number of CPUs. ##SBATCH --mem= #! How much wallclock time will be required? #SBATCH --time=02:00:00

in particular, the name of the project is required for the job to be scheduled (use the command mybalance to check what this is for you in case of doubt). Charging is reported in units of compute hours (what these represent depends on the cluster).

See the following sections for more details on the setting of directives for each of the three CSD3 clusters.

Peta4-Skylake

Peta4-Skylake assigns usage in units of CPU core hours. By convention projects containing CPU core hours have names ending in -CPU.

Jobs require the partitions skylake or skylake-himem, i.e.

#SBATCH -p skylake

or

#SBATCH -p skylake-himem

and will be allocated the number of CPUs required for the number of tasks requested and a corresponding amount of memory.

By default, the skylake partition provides 1 CPU and 5990MB of RAM per task, and the skylake-himem partition provides 1 CPU and 12040MB per task.

Requesting more CPUs per task, or more memory per task, may both increase the number of CPUs allocated (and hence the charge). It is more cost efficient to submit jobs requiring more than 6020MB per task to the skylake-himem partition since more memory per CPU is natively available there.

NB Hyperthreading is disabled on the Skylake nodes so there is no distinction between CPUs and cores.

Peta4-KNL

Peta4-KNL assigns usage in units of KNL node hours. By convention projects containing KNL node hours have names ending in -KNL.

Jobs require the partition knl, i.e.

#SBATCH -p knl

and will be allocated entire KNL nodes. Each KNL node has 64 physical cores but presents 256 cpus via hyperthreading, has 96GB DDR RAM plus 16GB MCDRAM high bandwidth memory and has been configured in quadrant/cache mode by default (in cache mode, the MCDRAM works invisibly as cache).

It is possible to vary the MCDRAM mode required at job submission time - please use either   --constraint or the equivalent -C sbatch option to select the mode. We recommend using either

#SBATCH --constraint=cache

or

#SBATCH --constraint=flat

Flat mode makes the MCDRAM visible as a second 16GB NUMA zone. Please note that hybrid MCDRAM mode, or any NUMA mode other than quad(rant), are not recommended.


Wilkes2-GPU

Wilkes2-GPU assigns usage in units of GPU hours. By convention projects containing GPU hours have names ending in -GPU.

Jobs require the partition pascal, i.e.

#SBATCH -p pascal

and may request any number of GPUs per node from the range 1 to 4, which is done via the directive

#SBATCH --gres=gpu:N

where 1 <= N <= 4.

Each GPU node contains 4 Nvidia Pascal P100 GPUs, with 96GB RAM and a single 12-core Broadwell processor.

Any jobs requesting more than one node must request 4 GPUs per node. Jobs less than one node in size will be prevented from requesting more than 3 CPUs per GPU. The enforcement is performed by a job submission filter which will produce an explanatory message if it rejects a job outright.

 

Submitting the job to the queuing system

The command sbatch is used to submit jobs, e.g.

sbatch submission_script

The command will return a unique job identifier, which is used to query and control the job and to identify output. See the man page (man sbatch) for more options.

The following more complex example submits a job array with index values between 1 and 7 with a step size of 2 (i.e. 1, 3, 5 and 7) to the project STARS-SL2-CPU:

sbatch --array=1-7:2 -A STARS-SL2-CPU submission_script