Job Management¶
Slurm offers many different tools for checking on the queues and interacting with your jobs on the cluster.
Checking the job queue (squeue)¶
You can view your jobs in the queue using the command squeue or
use the command squeue -u <username> to see only jobs you have submitted.
[abc123@costar-login01 ~]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
40143 main castepjo abc123 R 8:01 1 costar01
40072 main raccoon abc123 R 1-20:07:59 1 costar02
40145 main hello abc123 R 0:01 2 costar[01-03] <------ Here is my Job!
40116 main es254.sh abc126 R 20:49:02 1 costar01
40125 main bash abc127 R 2:06:20 1 costar01
34114 main halo_332 abc131 R 4-11:08:00 4 costar[01-03]
40090 gpu rbd_EPR_ abc140 R 1-03:18:31 1 costar03
[abc123@costar-login01 ~]$ squeue -u abc123
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
40145 main hello abc123 R 0:11 2 costar[01-03]
If you have followed this example, you should have the following outputs in your directory.
[abc123@costar-login01 ~]$ ls -ltr
-rw-r--r-- 1 abc123 itsstaff 18084 Apr 17 12:48 helloworld.out
-rw-r--r-- 1 abc123 itsstaff 14 Apr 17 12:48 nodes
You can also query jobs to get the full information about the job using its job id number,
this can be done using the command scontrol show job <Job Id Number> as shown below:
[abc123@costar-login01 ~]$ scontrol show job 40148
JobId= 40145 JobName=hello
UserId=abc123(282122) GroupId=itsstaff(40000) MCS_label=N/A
Priority=12940 Nice=0 Account=it QOS=normal
JobState=COMPLETED Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:01 TimeLimit=00:05:00 TimeMin=N/A
SubmitTime=2019-04-17T13:06:21 EligibleTime=2019-04-17T13:06:21
StartTime=2019-04-17T13:06:21 EndTime=2019-04-17T13:06:22 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=main AllocNode:Sid=login7:28627
ReqNodeList=(null) ExcNodeList=(null)
NodeList=node[102-103]
BatchHost=node102
NumNodes=2 NumCPUs=20 NumTasks=20 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=20,mem=4G,node=2
Socks/Node=* NtasksPerN:B:S:C=10:0:*:* CoreSpec=*
MinCPUsNode=10 MinMemoryNode=2G MinTmpDiskNode=0
Features=ib DelayBoot=00:00:00
Gres=(null) Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/users/abc123/slurm_test.sh
WorkDir=/users/abc123
StdErr=/users/abc123/hello.out
StdIn=/dev/null
StdOut=/users/abc123/hello.out
Power=
Cancel a job (scancel)¶
Note
You cannot delete other users’ jobs; only the user who submitted the job or the HPC team can delete them.
If you have submitted a job and want to delete or cancel it, you can use the command
scancel <job-id>.
Command |
Description |
|---|---|
|
Cancel a specific job. |
|
Cancel all your jobs. |
|
Cancel all your jobs in the |
|
Cancel all your jobs in the |
|
Cancel a specific task in an array job. |
|
Cancel a range of specific tasks in an array job. |
View cluster information (sinfo)¶
sinfo allows you to view information about the status of the cluster, partitions, and node usage.
[abc123@costar-login01 ~]$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
main* up 7-00:00:00 2 down* costar[101,136]
main* up 7-00:00:00 3 resv costar[01,43-44]
main* up 7-00:00:00 32 mix costar[02,04,06-08,15-21,32,34,36-40,45,50,109-118,140]
main* up 7-00:00:00 29 alloc costar[1-13],costar[09-14,22-25,30-31,35,42,46,49]
main* up 7-00:00:00 22 idle costar[03,05,26-29,33,102-108,120-121,132-135,139,141]
shared up 1:00:00 1 idle costar01
debug up 1:00:00 1 idle costar03
debug up 7-00:00:00 8 idle costar[119,125-131]
shared up 7-00:00:00 3 idle costar[122-124]
View job stats after job completion (sacct)¶
sacct retrieves useful job stats from Slurm after the job has finished.
Retrieve information based on job number:
To keep the output readable, you can specify exactly which columns you want to see using the --format flag.
sacct -j <Job Number> --format=JobID,JobName,State,Elapsed,MaxRSS,NNodes,NodeList
[abc123@costar-login01 ~]$ sacct -j 56814 --format=JobID,JobName,State,Elapsed,MaxRSS,NNodes,NodeList
JobID JobName State Elapsed MaxRSS NNodes NodeList
------------ ---------- ---------- ---------- ---------- -------- ---------------
56814 Hello COMPLETED 00:00:01 4 costar[01-03]
56814.batch batch COMPLETED 00:00:01 1160K 1 node45
56814.0 pmi_proxy COMPLETED 00:00:01 828K 4 node[01-03,05]
Get information by user and time:
sacct --starttime YYYY-MM-DD -u <username> --format=JobID,State,Timelimit,JobName,MaxRSS,Elapsed,NodeList
[abc123@costar-login01 ~]$ sacct --starttime 2020-01-17 -u abc123 --format=JobID,State,Timelimit,JobName,MaxRSS,Elapsed,NodeList
JobID State Timelimit JobName MaxRSS Elapsed NodeList
------------ ---------- ---------- ---------- ---------- ---------- ---------------
106178 COMPLETED 00:10:00 hello 00:02:00 costar01
106178.batch COMPLETED batch 1956K 00:02:00 costar03
106179 COMPLETED 00:10:00 hello 00:02:00 costar02
106179.batch COMPLETED batch 1908K 00:02:00 costar01