Job Scheduler¶

To keep the CoSTAR cluster running efficiently, we use the Slurm workload manager.

Never run heavy calculations directly on the login node. Instead, submit your work to Slurm by requesting the compute resources you need.

Slurm will place your job in a queue and dispatch it to a compute node as soon as resources are available.

How to run a job¶

There are two primary ways to run workloads on the cluster:

Batch Jobs (Automated)

Best for: Long-running scripts, ML training, and massive data processing.

Interactive Jobs (Live)

Best for: Debugging, compiling code, and testing software environments.

Once your jobs are running, use these guides to manage your queue and understand cluster priorities.

Managing Jobs

Check the job queue, monitor your running tasks, and cancel jobs.

MPI & Parallel Jobs

Structure and submit workloads that need to communicate across multiple compute nodes.

Fairshare & Queues

Understand site policies, queue limits, and how Slurm calculates your job priority.

The CoSTAR cluster provides several partitions (queues) for different types of workloads:

Partition	Resources	Max Walltime	Description
main	4 nodes, 32 GPUs	4 days	Default partition for general long-running workloads.
risk	8 nodes, 64 GPUs	4 days	Larger resource pool but preemptible (jobs may be stopped and re-queued if higher-priority work needs the resources). Only use this if your workflow implements checkpointing and can tolerate interruption.
debug	1 node, 8 GPUs	4 hours	For short test and debugging jobs you can use this partition. It is good practice to validate job scripts before running longer running jobs.
mig	1 node, 24 MIG instances	4 days	For smaller GPU workloads consider using MIG (Multi-Instance GPU) slices rather than full GPUs. For example a 35GB or 71GB MIG instance can be used for inference workloads which would otherwise waste resources on a full 141GB GPU.

If you are unsure which partition is best for your workload, please contact the CoSTAR Support Team.