Example 2: Python/Conda Job

This example shows how to build a small Apptainer image for a CPU-only Python calculation. The image starts from a public Docker Python image, installs Miniforge, creates a Conda environment inside the image, and runs a simple prime-number calculation through Slurm.

Use this pattern when your workflow needs Python packages but does not need a GPU.

1. Connect to the CoSTAR cluster

Open a terminal and connect to the CoSTAR login node:

ssh ab1234@costar-login01

Replace ab1234 with your University username.

2. Create the Apptainer definition file

Download the definition file, or create a file called example-02-python.def in your working directory and copy in the recipe below.

• example-02-python.def

The Bootstrap: docker and From: python:3.12-slim lines tell Apptainer to build the image from the public Docker Hub Python image. The %post section installs Miniforge and creates a Conda environment called costar-example inside the image.

example-02-python.def (click to collapse / view)

Bootstrap: docker
From: python:3.12-slim

%post
    apt-get update
    apt-get install -y --no-install-recommends bzip2 ca-certificates curl
    rm -rf /var/lib/apt/lists/*

    curl -L -o /tmp/miniforge.sh \
        https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
    bash /tmp/miniforge.sh -b -p /opt/conda
    rm /tmp/miniforge.sh

    /opt/conda/bin/conda create -y -n costar-example python=3.12 numpy
    /opt/conda/bin/conda clean -afy

%environment
    export PATH=/opt/conda/envs/costar-example/bin:/opt/conda/bin:$PATH
    export PYTHONUNBUFFERED=1

%runscript
    exec /opt/conda/envs/costar-example/bin/python "$@"

%labels
    Author CoSTAR
    Description "CPU-only Python/Conda example for CoSTAR tutorials"

3. Create the Python script

Download the Python script, or create a file called example-02-python.py in your working directory and copy in the script below.

• example-02-python.py

example-02-python.py (click to collapse / view)

import sys

import numpy as np


def is_prime(number):
    if number < 2:
        return False
    if number == 2:
        return True
    if number % 2 == 0:
        return False

    limit = int(np.sqrt(number)) + 1
    for factor in range(3, limit, 2):
        if number % factor == 0:
            return False
    return True


def main():
    upper_limit = 100_000
    primes = [number for number in range(2, upper_limit + 1) if is_prime(number)]

    print("Hello from a Python/Conda environment inside Apptainer")
    print(f"Python executable: {sys.executable}")
    print(f"NumPy version: {np.__version__}")
    print(f"Prime search limit: {upper_limit}")
    print(f"Number of primes found: {len(primes)}")
    print(f"Largest prime found: {primes[-1]}")


if __name__ == "__main__":
    main()

4. Build the Apptainer image

Note

On CoSTAR, Apptainer is available on compute nodes, not on the login node. If you want to build the image on the cluster, start an interactive Slurm session on a compute node first.

Start an interactive Slurm session for one hour. This does not request a GPU and only needs minimal resources for the build, so Slurm can choose any available node on any partition:

srun -N 1 --time=00-01:00:00 --pty bash

The image is lightweight and based on python:3.12-slim, so the build may only take around 10 minutes. The one-hour allocation gives some buffer for queueing, pulling the base image, and installing the Conda packages.

To build the image use:

apptainer build example-02-python.sif example-02-python.def

When the build finishes, exit the interactive session:

exit

5. Create the submission script

Download the example submission script, or create a file called example-02-python.sh in your working directory and copy in the script below.

• example-02-python.sh

The job script runs example-02-python.py inside the Apptainer image built in step 4.

example-02-python.sh (click to collapse / view)

#!/bin/bash

#SBATCH --job-name=python-conda
#SBATCH --partition=main
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --output=slurm.%N.%j.out
#SBATCH --error=slurm.%N.%j.err

apptainer run example-02-python.sif example-02-python.py

This example uses apptainer run because the definition file contains a %runscript section:

%runscript
    exec /opt/conda/envs/costar-example/bin/python "$@"

When you run:

apptainer run example-02-python.sif example-02-python.py

Apptainer executes the image’s runscript and passes example-02-python.py to it.

You can also use apptainer exec to specify the command directly:

apptainer exec example-02-python.sif python example-02-python.py

Use apptainer run when the image defines a useful default command in %runscript. Use apptainer exec when you want to explicitly choose the command to run inside the image.

6. Submit your job

Submit the script to the Slurm scheduler:

sbatch example-02-python.sh

You can check the status of your job by running:

squeue --me

7. Check the output

When the job finishes, Slurm writes the standard output and error streams to files in the submission directory:

• slurm.<node>.<job_id>.out contains normal output from the calculation.

• slurm.<node>.<job_id>.err contains error messages, if any were produced.

The output file should show the Python executable inside the container, the NumPy version, the prime search limit, the number of primes found, and the largest prime found.

For example, the .out file should contain output similar to:

Hello from a Python/Conda environment inside Apptainer
Python executable: /opt/conda/envs/costar-example/bin/python
NumPy version: 2.4.5
Prime search limit: 100000
Number of primes found: 9592
Largest prime found: 99991