.. _cluster-scheduler:
.. _cluster-scheduler-slurm:

**************
Job  Scheduler
**************

To keep the CoSTAR cluster running efficiently, we use the **Slurm** workload manager.

Never run heavy calculations directly on the login node. Instead, submit your work to **Slurm** by requesting the compute resources you need.

**Slurm** will place your job in a queue and dispatch it to a compute node as soon as resources are available.

How to run a job
================

There are two primary ways to run workloads on the cluster:

.. grid:: 1 2 2 2
   :gutter: 3
   :margin: 1

   .. grid-item-card:: :octicon:`file-code` Batch Jobs (Automated)
      :link: slurm-batch-job
      :link-type: ref

      **Best for:** Long-running scripts, ML training, and massive data processing.


   .. grid-item-card:: :octicon:`terminal` Interactive Jobs (Live)
      :link: slurm-interactive-jobs
      :link-type: ref

      **Best for:** Debugging, compiling code, and testing software environments.


Job Management & Policies
=========================

Once your jobs are running, use these guides to manage your queue and understand cluster priorities.

.. grid:: 1 1 3 3
   :gutter: 3
   :margin: 1

   .. grid-item-card:: :octicon:`tasklist` Managing Jobs
      :link: managing-jobs
      :link-type: ref

      Check the job queue, monitor your running tasks, and cancel jobs.

   .. grid-item-card:: :octicon:`versions` MPI & Parallel Jobs
      :link: mpi
      :link-type: doc

      Structure and submit workloads that need to communicate across multiple compute nodes.

   .. grid-item-card:: :octicon:`law` Fairshare & Queues
      :link: fairshare
      :link-type: doc

      Understand site policies, queue limits, and how Slurm calculates your job priority.




Available Partitions
~~~~~~~~~~~~~~~~~~~~

The CoSTAR cluster provides several partitions (queues) for different types of workloads:

.. list-table::
   :widths: 15 15 15 55
   :header-rows: 1
   :width: 100%

   * - Partition
     - Resources
     - Max Walltime
     - Description
   * - **main**
     - 4 nodes, 32 GPUs
     - 4 days
     - Default partition for general long-running workloads.
   * - **risk**
     - 8 nodes, 64 GPUs
     - 4 days
     - Larger resource pool but preemptible (jobs may be stopped and re-queued if higher-priority work needs the resources). Only use this if your workflow implements checkpointing and can tolerate interruption.
   * - **debug**
     - 1 node, 8 GPUs
     - 4 hours
     - For short test and debugging jobs you can use this partition. It is good practice to validate job scripts before running longer running jobs.
   * - **mig**
     - 1 node, 24 MIG instances
     - 4 days
     - For smaller GPU workloads consider using MIG (Multi-Instance GPU) slices rather than full GPUs. For example a 35GB or 71GB MIG instance can be used for inference workloads which would otherwise waste resources on a full 141GB GPU.

If you are unsure which partition is best for your workload, please contact the CoSTAR Support Team.

.. [*] **Industry Users** should use the partition name provided during the onboarding process. If you are unsure, please contact the CoSTAR Support Team.




.. toctree::
   :hidden:

   submitjobs
   managingjobs
   mpi
   fairshare