Advanced Apptainer Build Pipeline

This guide shows how to build a GPU-ready Apptainer image that recreates a known working Anaconda environment, keeps pip dependencies in a separate list for easier version changes, and compiles custom CUDA kernels for Gaussian Splatting.

The environment in this example is for the following codebase: https://github.com/tobias-kirschstein/gghead.git

For background on Apptainer, Docker image conversion, and where containers can be built on CoSTAR, see containers.

Quick highlights

  • Recreates a working Anaconda environment from a pinned export file.

  • Keeps pip dependencies in a separate requirements list for easier maintenance.

  • Allows building Gaussian Splatting CUDA kernels without a GPU present during the build by setting TORCH_CUDA_ARCH_LIST.

  • Installs code-server so the container is ready for Open OnDemand debugging.

1. Download the files

Download the Apptainer definition file and the two dependency lists.

guide-02-container.def (click to collapse / view)
 1Bootstrap: docker
 2From: ubuntu:22.04
 3
 4%files
 5    gghead-env.txt /tmp/gghead-env.txt
 6    gghead-pip.txt /tmp/gghead-pip.txt
 7
 8%post
 9    # Expected input files (copied via %files): /tmp/gghead-env.txt and /tmp/gghead-pip.txt
10    set -eu
11    export DEBIAN_FRONTEND=noninteractive
12
13    ENV_NAME="gghead"
14    ENV_SPEC="/tmp/gghead-env.txt"
15    PIP_SPEC="/tmp/gghead-pip.txt"
16    MAMBAFORGE_VERSION="23.11.0-0"
17    CODE_SERVER_VERSION="4.105.1"
18
19    # System dependencies for conda, build tools, and runtime libraries
20    apt-get update
21    apt-get install -y --no-install-recommends \
22        ca-certificates \
23        curl \
24        gdb \
25        git \
26        libcrypt-dev \
27        libglib2.0-0 \
28        libgl1-mesa-glx \
29        libtiff5 \
30        netcat \
31        ninja-build \
32        openssh-server \
33        openssl \
34        rsync \
35        tmux \
36        vim \
37        wget \
38        zlib1g-dev \
39        build-essential
40
41    # Install Mambaforge, recreate the conda env, and install pip requirements
42    wget "https://github.com/conda-forge/miniforge/releases/download/${MAMBAFORGE_VERSION}/Mambaforge-${MAMBAFORGE_VERSION}-Linux-x86_64.sh" -O /tmp/mambaforge.sh
43    bash /tmp/mambaforge.sh -b -p /opt/conda
44    /opt/conda/bin/mamba create -y --name "${ENV_NAME}" --file "${ENV_SPEC}"
45    /opt/conda/bin/conda run -n "${ENV_NAME}" pip install -r "${PIP_SPEC}"
46
47    # Build Gaussian Splatting CUDA kernels without requiring a GPU during build
48    /opt/conda/bin/conda run -n "${ENV_NAME}" \
49        TORCH_CUDA_ARCH_LIST="7.5+PTX" \
50        pip install gaussian_splatting@git+https://github.com/tobias-kirschstein/gaussian-splatting.git
51
52    # Install code-server and debugpy for Open OnDemand debugging
53    curl -fL "https://github.com/coder/code-server/releases/download/v${CODE_SERVER_VERSION}/code-server_${CODE_SERVER_VERSION}_amd64.deb" -o /tmp/code-server.deb
54    dpkg -i /tmp/code-server.deb
55    /opt/conda/bin/conda run -n "${ENV_NAME}" pip install debugpy
56
57    # Reduce final image size
58    /opt/conda/bin/conda clean -af
59    rm -f /tmp/mambaforge.sh /tmp/code-server.deb "${ENV_SPEC}" "${PIP_SPEC}"
60    rm -rf /var/lib/apt/lists/*
61
62    # Auto-activate conda env in interactive shells
63    echo "source /opt/conda/etc/profile.d/conda.sh && conda activate ${ENV_NAME}" >> /etc/profile
64
65%environment
66	export SINGULARITY_SHELL=/bin/bash
67	source /opt/conda/etc/profile.d/conda.sh
68	conda activate gghead
69
70%runscript
71	export SINGULARITY_SHELL=/bin/bash
72	source /opt/conda/etc/profile.d/conda.sh
73	conda activate gghead
74	exec /bin/bash "$@"
75
76%labels
77	Version v1.2
78	Description Apptainer image for https://github.com/tobias-kirschstein/gghead.git @ 662067055ddba11d7280a367426b56bf8fdd53f0

2. Prepare a build directory and build the image

Create a build directory, move the downloaded files into it, and rename them to match what the definition file expects:

mkdir -p ~/apptainer-builds/gghead
cd ~/apptainer-builds/gghead

mv ~/Downloads/guide-02-container.def gghead.def
mv ~/Downloads/guide-02-env.txt gghead-env.txt
mv ~/Downloads/guide-02-pip.txt gghead-pip.txt

Apptainer builds should run on a compute node. Start a small interactive session and build the .sif image:

srun -N 1 --ntasks=1 --cpus-per-task=8 --mem=32G --time=04:00:00 --pty bash
cd ~/apptainer-builds/gghead
apptainer build gghead.sif gghead.def

This should provide you with the .sif file which should be about 7.5G in size once built on the CoSTAR node.

Note

The resulting .sif file can be large. Follow the storage guidance in Containers: Docker & Apptainer and avoid filling your home quota.

3. Apptainer build file walkthrough

This definition file is organized into standard Apptainer sections. It is based on an Ubuntu 22.04 base image and uses a pinned conda export plus a separate pip list to keep versions stable and easy to update.

Key sections and what they do:

Bootstrap and base image The Bootstrap: docker and From: ubuntu:22.04 lines define the base image. This keeps the build reproducible and ensures a known starting point.

%files Copies the environment and pip requirement files into the image. The file names are standardized to gghead-env.txt and gghead-pip.txt so the build steps can rely on consistent paths.

%post Runs the build steps inside the image:

  • Installs system packages required for conda, build tools, and runtime libraries.

  • Installs Mambaforge and recreates the conda environment from the pinned export file. Mambaforge is preferred here because it ships with mamba for faster and more reliable dependency solving compared with a vanilla Miniconda install, while still being compatible with the same environment export format. Mamba is a drop in replacement for conda but is optimized for speed by using multithreading and a more optimized dependency solver.

  • Installs pip dependencies from the separate requirements list.

  • Builds the Gaussian Splatting CUDA kernels. TORCH_CUDA_ARCH_LIST is set so the kernels can be compiled without a GPU present during the build. The value 7.5 corresponds to the GPU compute capability (for example, NVIDIA T4 is 7.5). To choose the right value for your hardware, check the GPU model in your environment and look up its compute capability in NVIDIA’s GPU support table. Use the major.minor compute capability number (for example, 8.0 for A100, 8.6 for RTX 30xx). The compute capability version for H200 is 9.0 but in this example an older one is used to allow the same reproducible software environment to run on the user’s desktop GPU for benchmarking purposes and live demos. Using the latest compute capability however should result in minor performance gains in this case.

  • Installs code-server and debugpy for Open OnDemand debugging. code-server provides the web-based VS Code interface, and debugpy enables Python debugging sessions from that interface.

  • Cleans conda and apt caches to reduce the final image size.

%environment and %runscript Ensures the conda environment activates automatically for interactive shells and when the container runs.

Version pinning The definition file pins versions for Mambaforge and code-server. Pinning tool versions is a best practice for reproducibility because it reduces unexpected build failures caused by upstream changes and makes it easier to rebuild the same image later. If you do update these versions, do so intentionally and document the change alongside any environment updates.

You can adjust or re-use components of this definition file for your own projects as needed. For example, you could swap out the base image, change the Anaconda environment, or remove the code-server installation if you don’t need it.

4. Use the container in Open OnDemand

This image installs code-server and debugpy so it is ready for Open OnDemand debugging workflows. You can launch a session in Open OnDemand and point it at the container image, or use it in batch jobs with apptainer run or apptainer exec as needed.

If you need a refresher on Apptainer run and exec syntax, see the commands and examples in Containers: Docker & Apptainer.