.. _managing-jobs: ************** Job Management ************** Slurm offers many different tools for checking on the queues and interacting with your jobs on the cluster. Checking the job queue (squeue) =============================== You can view your jobs in the queue using the command ``squeue`` or use the command ``squeue -u `` to see only jobs you have submitted. .. code-block:: console :caption: checking our job with squeue [abc123@costar-login01 ~]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 40143 main castepjo abc123 R 8:01 1 costar01 40072 main raccoon abc123 R 1-20:07:59 1 costar02 40145 main hello abc123 R 0:01 2 costar[01-03] <------ Here is my Job! 40116 main es254.sh abc126 R 20:49:02 1 costar01 40125 main bash abc127 R 2:06:20 1 costar01 34114 main halo_332 abc131 R 4-11:08:00 4 costar[01-03] 40090 gpu rbd_EPR_ abc140 R 1-03:18:31 1 costar03 [abc123@costar-login01 ~]$ squeue -u abc123 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 40145 main hello abc123 R 0:11 2 costar[01-03] If you have followed this example, you should have the following outputs in your directory. .. code-block:: console :caption: checking output from the job [abc123@costar-login01 ~]$ ls -ltr -rw-r--r-- 1 abc123 itsstaff 18084 Apr 17 12:48 helloworld.out -rw-r--r-- 1 abc123 itsstaff 14 Apr 17 12:48 nodes You can also query jobs to get the full information about the job using its job id number, this can be done using the command ``scontrol show job `` as shown below: .. code-block:: console :caption: using scontrol to view job information [abc123@costar-login01 ~]$ scontrol show job 40148 JobId= 40145 JobName=hello UserId=abc123(282122) GroupId=itsstaff(40000) MCS_label=N/A Priority=12940 Nice=0 Account=it QOS=normal JobState=COMPLETED Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:01 TimeLimit=00:05:00 TimeMin=N/A SubmitTime=2019-04-17T13:06:21 EligibleTime=2019-04-17T13:06:21 StartTime=2019-04-17T13:06:21 EndTime=2019-04-17T13:06:22 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=main AllocNode:Sid=login7:28627 ReqNodeList=(null) ExcNodeList=(null) NodeList=node[102-103] BatchHost=node102 NumNodes=2 NumCPUs=20 NumTasks=20 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=20,mem=4G,node=2 Socks/Node=* NtasksPerN:B:S:C=10:0:*:* CoreSpec=* MinCPUsNode=10 MinMemoryNode=2G MinTmpDiskNode=0 Features=ib DelayBoot=00:00:00 Gres=(null) Reservation=(null) OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/users/abc123/slurm_test.sh WorkDir=/users/abc123 StdErr=/users/abc123/hello.out StdIn=/dev/null StdOut=/users/abc123/hello.out Power= .. _cancel-job: Cancel a job (scancel) ====================== .. note:: You cannot delete other users' jobs; only the user who submitted the job or the HPC team can delete them. If you have submitted a job and want to delete or cancel it, you can use the command ``scancel ``. .. list-table:: :header-rows: 1 :align: left * - Command - Description * - ``scancel `` - Cancel a specific job. * - ``scancel -u `` - Cancel all your jobs. * - ``scancel -t PENDING -u `` - Cancel all your jobs in the ``PENDING`` state. * - ``scancel -t RUNNING -u `` - Cancel all your jobs in the ``RUNNING`` state. * - ``scancel _`` - Cancel a specific task in an array job. * - ``scancel _[_]`` - Cancel a range of specific tasks in an array job. View cluster information (sinfo) ================================ ``sinfo`` allows you to view information about the status of the cluster, partitions, and node usage. .. code-block:: console [abc123@costar-login01 ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST main* up 7-00:00:00 2 down* costar[101,136] main* up 7-00:00:00 3 resv costar[01,43-44] main* up 7-00:00:00 32 mix costar[02,04,06-08,15-21,32,34,36-40,45,50,109-118,140] main* up 7-00:00:00 29 alloc costar[1-13],costar[09-14,22-25,30-31,35,42,46,49] main* up 7-00:00:00 22 idle costar[03,05,26-29,33,102-108,120-121,132-135,139,141] shared up 1:00:00 1 idle costar01 debug up 1:00:00 1 idle costar03 debug up 7-00:00:00 8 idle costar[119,125-131] shared up 7-00:00:00 3 idle costar[122-124] View job stats after job completion (sacct) =========================================== ``sacct`` retrieves useful job stats from Slurm after the job has finished. **Retrieve information based on job number:** To keep the output readable, you can specify exactly which columns you want to see using the ``--format`` flag. .. code-block:: bash sacct -j --format=JobID,JobName,State,Elapsed,MaxRSS,NNodes,NodeList .. code-block:: console :caption: sacct example by job id number [abc123@costar-login01 ~]$ sacct -j 56814 --format=JobID,JobName,State,Elapsed,MaxRSS,NNodes,NodeList JobID JobName State Elapsed MaxRSS NNodes NodeList ------------ ---------- ---------- ---------- ---------- -------- --------------- 56814 Hello COMPLETED 00:00:01 4 costar[01-03] 56814.batch batch COMPLETED 00:00:01 1160K 1 node45 56814.0 pmi_proxy COMPLETED 00:00:01 828K 4 node[01-03,05] **Get information by user and time:** .. code-block:: bash sacct --starttime YYYY-MM-DD -u --format=JobID,State,Timelimit,JobName,MaxRSS,Elapsed,NodeList .. code-block:: console :caption: sacct example by job user and time [abc123@costar-login01 ~]$ sacct --starttime 2020-01-17 -u abc123 --format=JobID,State,Timelimit,JobName,MaxRSS,Elapsed,NodeList JobID State Timelimit JobName MaxRSS Elapsed NodeList ------------ ---------- ---------- ---------- ---------- ---------- --------------- 106178 COMPLETED 00:10:00 hello 00:02:00 costar01 106178.batch COMPLETED batch 1956K 00:02:00 costar03 106179 COMPLETED 00:10:00 hello 00:02:00 costar02 106179.batch COMPLETED batch 1908K 00:02:00 costar01