.. _cluster:

******************
The CoSTAR Cluster
******************

This section covers the core technical infrastructure of the CoSTAR cluster and how to use it.
If you are new to High-Performance Computing (HPC), it helps to think of the system in terms of the **"3 Ss"**.

* **Scheduler** sends your jobs to the right hardware.
* **Software** provides the tools and environments you need.
* **Storage** keeps your data available and secure.

Understanding how these three components work together is the key to running successful jobs.

Select a component below to read the detailed guides and configurations:

.. raw:: html

   <div style="margin-bottom: 50px;"></div>

.. grid:: 1 2 2 3
   :gutter: 3
   :margin: 1

   .. grid-item-card:: :octicon:`server` Scheduler
      :link: cluster-scheduler
      :link-type: ref

      Learn how Slurm allocates compute nodes and GPUs. Find guides on submitting batch scripts and managing your queue.

   .. grid-item-card:: :octicon:`package` Software
      :link: cluster-software
      :link-type: ref

      Find out about recommended software workflows. Learn to build containers, manage virtual environments, etc.

   .. grid-item-card:: :octicon:`database` Storage
      :link: cluster-storage
      :link-type: ref

      Understand the high performance file systems. Check your quotas and learn where to store your code versus your massive datasets.


.. _cluster-specs:

Hardware Specifications
-----------------------

The CoSTAR cluster is a heterogeneous environment featuring 10 standard compute nodes, three pilot nodes, and 100 state-of-the-art NVIDIA H200 GPUs.

Compute Nodes
~~~~~~~~~~~~~

.. list-table::
   :widths: 25 30 10 35
   :header-rows: 1
   :width: 100%

   * - Nodes
     - Processor
     - RAM
     - GPUs (per node)
   * - **10 × Standard Nodes**

       costar04 – costar13
     - 2× AMD EPYC 9535

       (128 cores total)
     - 3 TB
     - 8× H200 (141 GB HBM3e)
   * - **2 × Intel Pilot Nodes**

       costar01 – costar02
     - 2× Intel Xeon 8480+

       (112 cores total)
     - 2 TB
     - 8× H200 (141 GB HBM3e)
   * - **1 × AMD Pilot Node**

       costar03 [*]_
     - 2× AMD EPYC

       (64 cores total)
     - 1.1 TB
     - 4× H200 (141 GB HBM3e)

.. [*] **costar03** is currently unavailable due to a faulty GPU. When it returns to the production pool, it will offer additional **MIG (Multi-Instance GPU)** slices (e.g., 35GB and 71GB instances) to support smaller workloads.


Available Partitions
~~~~~~~~~~~~~~~~~~~~

The CoSTAR cluster provides several partitions (queues) for different types of workloads:

.. list-table::
   :widths: 15 15 15 55
   :header-rows: 1
   :width: 100%

   * - Partition
     - Resources
     - Max Walltime
     - Description
   * - **main**
     - 4 nodes, 32 GPUs
     - 4 days
     - Default partition for general long-running workloads.
   * - **risk**
     - 8 nodes, 64 GPUs
     - 4 days
     - Larger resource pool but preemptible (jobs may be stopped and re-queued if higher-priority work needs the resources). Only use this if your workflow implements checkpointing and can tolerate interruption.
   * - **debug**
     - 1 node, 8 GPUs
     - 4 hours
     - For short test and debugging jobs you can use this partition. It is good practice to validate job scripts before running longer running jobs.
   * - **mig**
     - 1 node, 24 MIG instances
     - 4 days
     - For smaller GPU workloads consider using MIG (Multi-Instance GPU) slices rather than full GPUs. For example a 35GB or 71GB MIG instance can be used for inference workloads which would otherwise waste resources on a full 141GB GPU.

If you are unsure which partition is best for your workload, please contact the CoSTAR Support Team.

.. [*] **Industry Users** should use the partition name provided during the onboarding process. If you are unsure, please contact the CoSTAR Support Team.


Cluster Connectivity & Storage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :widths: 25 75
   :header-rows: 0
   :width: 100%

   * - **Login Node**
     - ``costar-login01.surrey.ac.uk``
   * - **Operating System**
     - Rocky Linux 9.6
   * - **Interconnect Fabric**
     - Mellanox InfiniBand NDR (400 Gb/s)
   * - **Parallel Storage**
     - 750 TB WekaFS (NVMe-backed)
   * - **Total Production Cores**
     - 1,568
   * - **Total Production GPUs**
     - 100× NVIDIA H200 (141 GB HBM3e)