Slurm examples

SLURM sequential job

  • seq.sh is a SLURM sequential job:
$ cat seq.sh
#!/bin/bash
#SBATCH --job-name=seq
#SBATCH --output=%x.o%j
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --partition=cpu_short           # (see available partitions)

# To clean and load modules defined at the compile and link phases
module purge
module load ...

# echo of commands
set -x

# To compute in the submission directory
cd ${SLURM_SUBMIT_DIR}

# execution
./a.out
  • To submit seq.sh with the sbatch command:
# Soumission du script en batch
$ sbatch seq.sh

SLURM OpenMP parallel job

  • openmp.sh is a SLURM OpenMP job with 20 OpenMP threads
$ cat openmp.sh
#!/bin/bash
#SBATCH --job-name=openmp
#SBATCH --output=%x.o%j
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=20
#SBATCH --mem=80G
#SBATCH --partition=cpu_short       # (see available partitions)

# To clean and to load the same modules at the compilation phases
module purge
module load ...

# echo of commands
set -x

# To compute in the submission directory
cd ${SLURM_SUBMIT_DIR}

# number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} 

# Binding OpenMP threads on core
export OMP_PLACES=cores

# execution with 'OMP_NUM_THREADS' OpenMP threads
./a.out
  • To submit openmp.sh with the sbatch command:
$ sbatch openmp.sh
  • Remarks:
    • To adjust the memory per node, add --mem

SLURM MPI parallel job

  • mpi.sh is a SLURM MPI job with 80 MPI processes
$ cat mpi.sh
#!/bin/bash
#SBATCH --job-name=mpi
#SBATCH --output=%x.o%j
#SBATCH --time=01:00:00
#SBATCH --ntasks=80
##SBATCH --partition=cpu_short      # (see available partitions)

# To clean and to load the same modules at the compilation phases
module purge
module load ...

# echo of commands
set -x

# To compute in the submission directory
cd ${SLURM_SUBMIT_DIR}

# execution with 'ntasks' MPI processes
srun ./a.out
  • To submit mpi.sh with the sbatch command:
$ sbatch mpi.sh
  • Remarks:
    • Parallel compute nodes have 40 cores. Try to use a multiple number of MPI processes of 40 to use all the cores of nodes.
    • If you don't use all cores of nodes, other jobs can share the same nodes and performances can decrease.

SLURM hybrid MPI/OpenMP parallel job

mpi_openmp.sh is a SLURM MPI job allocating 40 cores : * 2 MPI processes (--ntasks=2), * each MPI process will spawn 20 OpenMP threads (--cpus-per-task=20)

$ cat mpi_openmp.sh
#!/bin/bash
#SBATCH --job-name=mpi_openmp
#SBATCH --output=%x.o%j
#SBATCH --time=01:00:00
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=20
##SBATCH --partition=cpu_short      # (see available partitions)

# To clean and to load the same modules at the compilation phases
module purge
module load ...

# echo of commands
set -x

# To compute in the submission directory
cd ${SLURM_SUBMIT_DIR}

# number of OpenMP threads
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} 

# Binding OpenMP Threads of each MPI process on cores
export OMP_PLACES=cores

# execution 
# with 'ntasks' MPI processes
# with 'cpus-per-task' OpenMP threads per MPI process
srun ./a.out
  • To submit mpi_openmp.sh with the sbatch command:
$ sbatch mpi_openmp.sh
  • Remarks:
    • Parallel compute nodes have 40 cores. Try to use a multiple number of [MPI processes * OpenMP threads] of 40 to use all the cores of nodes.
    • If you don't use all cores of nodes, other jobs can share the same nodes and performances can decrease.