.. _software-conda: ===== Conda ===== Conda is a tool that can help you manage different Python versions and packages in isolated environments. We recommend it for most users working with Python on the cluster. Using Conda environments makes it easier to: - Avoid conflicts between projects - Use modern and consistent Python packages - Follow research best practices around **reproducibility and portability** - Share your environment across machines or with collaborators ------------------- 1. Setting up Conda ------------------- ^^^^^^^^^^^^^^^^^^^^ Installing Miniconda ^^^^^^^^^^^^^^^^^^^^ You can install Miniconda in your home directory: .. code-block:: bash :caption: Miniconda setup # Download and install Miniconda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh # Follow the prompts, then restart your shell .. warning:: We recommend you do **not** install full Anaconda unless you have a specific reason to do so and plenty of storage space. Miniconda is much smaller and flexible. ^^^^^^^^^^^^^^^^^^^^ Finalising the setup ^^^^^^^^^^^^^^^^^^^^ To set up ``conda`` in your shell properly, it needs to be initialised **once**: .. code-block:: console [ab1234@costar-login01 ~]$ conda init bash Then log out and log back in Your shell will now support Conda properly. You will see something like: .. code-block:: console (base) [ab1234@costar-login01 ~]$ This shows that your default Conda environment (``base``) is active. ---------------------------------- 2. Creating and using environments ---------------------------------- We recommend creating your own environments, rather than using the ``base`` one. Here, we create an environment ``myenv`` using Python 3.12: .. code-block:: bash :caption: Create and use a new environment conda create -n myenv python=3.12 conda activate myenv # Search for a package conda search matplotlib # Install with conda conda install matplotlib # Or install with pip conda install pip pip install seaborn # Deactivate when done conda deactivate .. note:: By default, packages are installed from the main channel. We recommend switching to `conda-forge` for most use cases, see tips below. To install specific version, impose version number constraints, or install packages or whole environments from files, see the tabs below: .. tabs:: .. tab:: Specific versions .. code-block:: bash :caption: Install specific versions (examples) conda create -n myenv python=3.13 numpy=2.0 matplotlib>=3.0 .. tab:: From requirements.txt (pip) If you have a ``requirements.txt`` file, you can setup an environment and install the requirements like so: .. code-block:: bash :caption: Install packages from requirements.txt conda create -n myenv python=3.11 pip conda activate myenv pip install -r requirements.txt .. tab:: From environment.yml If you have an ``environment.txt`` file, you can setup an environment and install the requirements like so: .. code-block:: bash conda env create -f environment.yml # You can find the environment's name in the file conda activate myenv-name-from-yml You can export an existing environment with: .. code-block:: bash :caption: yml file export conda env export > environment.yml -------------------------- 3. Tips and best practices -------------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Project-specific environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Use one environment per project to avoid conflicts, to easily move your workloads across compute environments, and to reproduce your work or share it later. ^^^^^^^^^^^^^^^^^^^^^^^^ Use conda-forge packages ^^^^^^^^^^^^^^^^^^^^^^^^ The conda-forge channel is a community-maintained collection of packages. It is more up-to-date and avoids Anaconda's license restrictions (see `more `_). .. code-block:: bash :caption: Configure Conda to use the conda-forge channel conda config --add channels conda-forge conda config --set channel_priority strict ^^^^^^^^^^^^^^^^^ List environments ^^^^^^^^^^^^^^^^^ You can view all your conda environments with: .. code-block:: bash conda env list ^^^^^^^^^^^^^^^^^^^ Remove environments ^^^^^^^^^^^^^^^^^^^ Free up disk space by deleting unused environments: .. code-block:: bash conda remove -n myenv --all ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Export / import environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To save your environment or share it with others, export it to a .yml file: .. code-block:: bash conda env export > environment.yml You (or someone else) can recreate the environment later: .. code-block:: bash conda env create -f environment.yml ^^^^^^^^^^^^^^^^^^^^ Manage storage usage ^^^^^^^^^^^^^^^^^^^^ By default, conda stores environments in your home directory under ``~/.conda/envs``. If you create many environments or install large packages, this can quickly use up your home directory quota. """"""""""""""""""""""" Custom storage location """"""""""""""""""""""" To store environments in another location, for instance in scratch directory (replace ``ab1234`` with your user name): .. code-block:: bash :caption: Configure Conda to save environments to a custom directory cd /parallel_scratch/ab1234/ mkdir conda_envs conda config --append envs_dirs /parallel_scratch/ab1234/conda_envs Check where conda looks for environments with: .. code-block:: bash conda config --show envs_dirs .. note:: You can manually copy environments to a new location, but it is usually safer to recreate them from your ``environment.yml`` file. """"""""""""""""" Clear Cached data """"""""""""""""" To free up disk space, you can delete Conda’s package cache, which stores downloaded ``.tar.bz2`` and ``.conda`` files: .. code-block:: bash conda clean --all This removes unused packages and caches. You can also preview what will be deleted first with: .. code-block:: bash conda clean --all --dry-run ^^^^^^^^^^^^^^^^^^^^^^^^^^ Use recent Python versions ^^^^^^^^^^^^^^^^^^^^^^^^^^ We recommend selecting Python versions for your projects that offer long term security and performance updates, see the `Status of Python Versions page `_ for details. ^^^^^^^^^^^^^^^ Further reading ^^^^^^^^^^^^^^^ - `Getting started with Conda `_ - `Managing environments `_ - `Using pip in Conda environments `_ - `Conda cheat sheet (PDF) `_ - `conda-forge documentation `_