.. _tutorials-example-05-mpi:

Example 5: Containerized MPI Multi-Process Job
==============================================

This example shows how to run a multi-process MPI job with Slurm and
Apptainer. It uses a small Python program with ``mpi4py`` to estimate pi using
a distributed Monte Carlo calculation.

The example follows a hybrid launch model: the host MPI launcher starts one
Apptainer container per MPI rank. The container provides Python and ``mpi4py``;
the host MPI stack coordinates the ranks across the allocated nodes.

1. Connect to the CoSTAR cluster
--------------------------------

Open a terminal and connect to the CoSTAR login node:

.. code-block:: bash

   ssh ab1234@costar-login01

Replace ``ab1234`` with your University username.

2. Create the Apptainer definition file
---------------------------------------

Download the definition file, or create a file called ``example-05-mpi.def`` in
your working directory and copy in the recipe below.

* :download:`example-05-mpi.def <source_files/example-05-mpi.def>`

.. dropdown:: example-05-mpi.def (click to collapse / view)
   :color: primary
   :open:
   :icon: file-code

   .. literalinclude:: source_files/example-05-mpi.def
      :language: text
      :linenos:

3. Create the MPI Python script
-------------------------------

Download the Python script, or create a file called ``example-05-mpi.py`` in
your working directory and copy in the script below.

* :download:`example-05-mpi.py <source_files/example-05-mpi.py>`

.. dropdown:: example-05-mpi.py (click to collapse / view)
   :color: primary
   :open:
   :icon: file-code

   .. literalinclude:: source_files/example-05-mpi.py
      :language: python
      :linenos:

4. Build the Apptainer image
----------------------------

.. note::

   On CoSTAR, Apptainer is available on compute nodes, not on the login node.
   If you want to build the image on the cluster, start an interactive Slurm
   session on a compute node first.

Start an interactive Slurm session:

.. code-block:: bash

   srun -N 1 --time=00-01:00:00 --pty bash

Once the session starts on a compute node, build the image:

.. code-block:: bash

   apptainer build example-05-mpi.sif example-05-mpi.def

When the build finishes, exit the interactive session:

.. code-block:: bash

   exit

5. Run interactively
--------------------

For a quick interactive test, first request an allocation across 2 nodes with 1
MPI rank per node:

.. code-block:: bash

   salloc -N 2 --ntasks-per-node=1 -t 02:00:00

When the allocation starts, run the MPI program inside the Apptainer image:

.. code-block:: bash

   mpirun -np 2 apptainer exec --cleanenv example-05-mpi.sif python3 example-05-mpi.py

This launches 2 MPI ranks in total. When you finish testing, leave the
interactive allocation with:

.. code-block:: bash

   exit

6. Create the batch submission script
-------------------------------------

Download the example submission script, or create a file called
``example-05-mpi.sh`` in your working directory and copy in the script below.

* :download:`example-05-mpi.sh <source_files/example-05-mpi.sh>`

.. dropdown:: example-05-mpi.sh (click to collapse / view)
   :color: primary
   :open:
   :icon: file-code

   .. literalinclude:: source_files/example-05-mpi.sh
      :language: bash
      :linenos:

The job requests 2 nodes with 1 MPI rank per node, for 2 MPI ranks in total.
The ``mpirun`` command launches those ranks and runs the Python script inside
the Apptainer image.

7. Submit as a batch job
------------------------

Submit the script to the Slurm scheduler:

.. code-block:: bash

   sbatch example-05-mpi.sh

You can check the status of your job by running:

.. code-block:: bash

   squeue --me

8. Check the output
-------------------

When the job finishes, Slurm writes the standard output and error streams to
files in the submission directory:

* ``slurm.<node>.<job_id>.out`` contains the allocated node list, MPI rank
  placement, and the final pi estimate.
* ``slurm.<node>.<job_id>.err`` contains error messages, if any were produced.

The output file should include lines similar to:

.. code-block:: text

   Nodes allocated:
   costar01
   costar02
   [INFO] World size = 2
   Rank 00 on host costar01
   Rank 01 on host costar02
   [RESULT] pi ~= 3.141820 using 2000000 samples

The rank order may differ between runs, but the world size should match the
number of MPI ranks requested by Slurm.

9. What to change next
----------------------

For your own MPI workload, adjust:

* ``--nodes`` for the number of compute nodes.
* ``--ntasks-per-node`` for the number of MPI ranks per node.
* ``IMAGE`` to point to your Apptainer image.
* ``SCRIPT`` to point to your MPI application or Python script.
* The MPI modules to match the MPI version needed by your application.