.. _scheduler-mpi:

**************************
Running MPI or OpenMP jobs
**************************

What is MPI?
============

MPI stands for Message Passing Interface. It is a standardised way for parallel processes to exchange messages while
running on systems with distributed memory.

MPI is commonly used when code needs to coordinate work across more than one CPU core. Those cores may be on the same
physical computer, or they may be on multiple physically separate computers, such as different compute nodes in an HPC
cluster.

When you run a job across multiple compute nodes in an HPC cluster, each node is a separate physical computer and
each node typically works on a portion of the overall computing problem. The challenge is to synchronise the actions of
the parallel processes, exchange data between them, and coordinate the whole parallel job.

The Message Passing Interface defines a standard suite of functions for these tasks.

https://en.wikipedia.org/wiki/Message_Passing_Interface


Running an MPI Job
==================

For non-MPI jobs, you may have used Slurm options such as:

.. code-block:: bash

    #SBATCH --nodes=<number>,
    #SBATCH --ntasks-per-node=<number>,
    #SBATCH --ntasks=<number>, or
    #SBATCH --cpus-per-task=<number>

in submit files to request resources.

For MPI jobs it is important to understand how these options affect the number of processes Slurm starts and where those
processes are placed. There can be several ways to request a similar amount of hardware, but they may look different to
Slurm and to MPI.

For example, the following request in a submit file:

.. code-block:: bash

    #SBATCH --nodes=3
    #SBATCH --ntasks=3
    #SBATCH --cpus-per-task=3

is equivalent in terms of resource allocation to:

.. code-block:: bash

    #SBATCH --ntasks=9
    #SBATCH --ntasks-per-node=3

but is seen differently by Slurm and MPI. In the first case, 3 processes are launched, each with 3 CPU cores available.
In the second case, 9 processes are launched, with 3 processes placed on each node.

Examples
--------

Consider the following examples where multiple cores are required. Add the relevant ``#SBATCH`` lines near the top of
your submit file, before the commands that run your program. The right request depends on whether your application uses
MPI distributed processes or OpenMP-style shared-memory threading.

MPI (or distributed)
^^^^^^^^^^^^^^^^^^^^

- You use MPI and do not care where the processes are placed:

  .. code-block:: bash

      #SBATCH --ntasks=16

- You want to launch 16 independent processes with no communication:

  .. code-block:: bash

      #SBATCH --ntasks=16

- You want 16 processes spread across distinct nodes:

  .. code-block:: bash

      #SBATCH --ntasks=16
      #SBATCH --ntasks-per-node=1

  or:

  .. code-block:: bash

      #SBATCH --nodes=16
      #SBATCH --ntasks=16

- You want 10 processes spread across 5 nodes, with 2 processes per node:

  .. code-block:: bash

      #SBATCH --ntasks=10
      #SBATCH --ntasks-per-node=2

- You want 10 processes to stay on the same node:

  .. code-block:: bash

      #SBATCH --ntasks=10
      #SBATCH --ntasks-per-node=10

OpenMP (single node/shared memory parallel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- You want one process that can use 9 cores for multi-threading:

  .. code-block:: bash

      #SBATCH --ntasks=1
      #SBATCH --cpus-per-task=9

- You want 4 processes that can use 4 cores each for multi-threading:

  .. code-block:: bash

      #SBATCH --ntasks=4
      #SBATCH --cpus-per-task=4

.. note::
    
    Example cases adapted from https://support.ceci-hpc.be/doc/_contents/SubmittingJobs/SlurmFAQ.html

The diagram below shows the difference between MPI processes and OpenMP-style threading:

.. figure:: images/mpi_vs_openmp.png
    :align: center
    
    MPI vs OpenMP