21.1 Canceling Jobs with Preemption
Context
CANCEL is one of the PREEMPTPOLICY types (for more information, see PREEMPTPOLICY Types). The CANCEL attribute cancels active jobs, regardless of any JOBFLAGS (such as REQUEUEABLE or SUSPENDABLE). (For more information, see Job Flags.)
For information about PREEPMPTEE and PREEMPTOR flags, see Preemption Flags
You should not allow preemption with interactive jobs unless PREEMPTPOLICY is set to CANCEL.
The following outlines some benefits of using CANCEL and also lists some things you should be aware of if you choose to use it.
Advantages:
This attribute is the easiest to configure and use.
Cautions:
Canceled jobs are not automatically restarted or requeued. Users must resubmit canceled jobs.
To preempt jobs using CANCEL
- Make the following configurations to the
- Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
- Set PREEMPTIONALGORITHM to specify how Moab handles preemption scheduling policies.
If you use JOBNODEMATCHPOLICY EXACTNODE, you must also add PREEMPTIONALGORITHM PREEMPTORCENTRIC in order for preemption to function reliably.
- Set PREEMPTPOLICY to CANCEL (for more information, see PREEMPTPOLICY Types).
- Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption Flags).
- Submit a job to the preemptee QoS (test1). For example:
- Now submit a job to the preemptor QoS (test2). For example:
For example:
GUARANTEEDPREEMPTION TRUE PREEMPTPOLICY CANCEL QOSCFG[test1] QFLAGS=PREEMPTEE MEMBERULIST=john PRIORITY=100 QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128
(Optional) Examine the following output for showq:
Moab.7
[john@g06]# showq
active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME
Moab.7 john Running 128 00:01:59 Thu Nov 10 12:28:44
1 active job 128 of 128 processors in use by local jobs (100.00%)
2 of 2 nodes active (100.00%)
eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 eligible jobs
blocked jobs-----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 blocked jobs
Total job: 1
[john@g06]$ echo sleep 120 | msub -l procs=128,walltime=120 -l qos=test2
(Optional) Examine the following output for showq:
Moab.8
[john@g06]# showq
active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME
Moab.7 john Canceling 128 00:01:56 Thu Nov 10 12:28:44
Moab.8 john Running 128 00:02:00 Thu Nov 10 12:28:48
2 active jobs 128 of 128 processors in use by local jobs (100.00%)
2 of 2 nodes active (100.00%)
eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 eligible jobs
blocked jobs-----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 blocked jobs
Total jobs: 2
Note that test1 is canceled when test2 is submitted.
(Optional) Examine the checkjob outputs for these two jobs:
[john@g06]$ checkjob Moab.9
job Moab.9
State: Removed
Completion Code: -1 Time: Thu Nov 10 12:28:48
Creds: user:john group:john qos:test1
WallTime: 00:00:02 of 00:02:00
SubmitTime: Thu Nov 10 12:28:44
(Time Queued Total: 00:00:07 Eligible: 00:00:00)
Total Requested Tasks: 128
Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2
Allocated Nodes:
node[01-02]*64
IWD: /opt/native
SubmitDir: /opt/native
Executable: /opt/native/spool/moab.job.UFe8sQ
StartCount: 1
Flags: GLOBALQUEUE,PROCSPECIFIED
Attr: PREEMPTEE
StartPriority: 100
Note that the preempted job has been removed.
[john@g06]$ checkjob Moab.10 job Moab.10 State: Running Creds: user:john group:john qos:test2 WallTime: 00:00:00 of 00:02:00 SubmitTime: Thu Nov 10 12:36:31 (Time Queued Total: 00:00:00 Eligible: 00:00:00) StartTime: Thu Nov 10 12:28:48 Total Requested Tasks: 128 Req[0] TaskCount: 128 Partition: licenses Allocated Nodes: node[01-02]*64 IWD: /opt/native SubmitDir: /opt/native Executable: /opt/native/spool/moab.job.CZavjU StartCount: 1 Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED StartPriority: 10000 Reservation 'Moab.10' (-00:00:07 -> 00:01:53 Duration: 00:02:00)
Related Topics