(Click to open topic with navigation)

23.0 Preemption > Preemption tasks > Canceling jobs with preemption

Conventions

23.1.1 Canceling jobs with preemption

ClosedContext

CANCEL is one of the PREEMPTPOLICY types (for more information, see PREEMPTPOLICY types). The CANCEL value cancels active jobs, regardless of any JOBFLAGS (such as REQUEUEABLE or SUSPENDABLE). (For more information, see Job Flags.)

For information about PREEPMPTEE and PREEMPTOR flags, see Preemption flags

You should not allow preemption with interactive jobs unless PREEMPTPOLICY is set to CANCEL.

The following outlines some benefits of using CANCEL and also lists some things you should be aware of if you choose to use it.

Advantages:

This attribute is the easiest to configure and use.

Cautions:

Canceled jobs are not automatically restarted or requeued. Users must resubmit canceled jobs.

To preempt jobs using CANCEL

  1. Make the following configurations to the moab.cfg file:
    1. Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
    2. Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and troubleshooting preemption).
    3. Set PREEMPTPOLICY to CANCEL (for more information, see PREEMPTPOLICY types).
    4. Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption flags).

    2. For example:

    GUARANTEEDPREEMPTION TRUE
PREEMPTPOLICY CANCEL
 
QOSCFG[test1] QFLAGS=PREEMPTEE MEMBERULIST=john PRIORITY=100
QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
  2. Submit a job to the preemptee QoS (test1). For example:
    3. [john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128

    (Optional) Examine the following output for showq:

    Moab.7
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS     REMAINING     STARTTIME
Moab.7    john        Running    128       00:01:59      Thu Nov 10 12:28:44
 
1 active job     128 of 128 processors in use by local jobs (100.00%)
                 2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 eligible jobs
 
blocked jobs-----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total job: 1
  3. Now submit a job to the preemptor QoS (test2). For example:
    4. [john@g06]$ echo sleep 120 | msub -l procs=128,walltime=120 -l qos=test2

    (Optional) Examine the following output for showq:

    Moab.8
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS     REMAINING     STARTTIME
Moab.7    john        Canceling  128       00:01:56      Thu Nov 10 12:28:44
Moab.8    john        Running    128       00:02:00      Thu Nov 10 12:28:48
 
2 active jobs 128 of 128 processors in use by local jobs (100.00%)
              2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 eligible jobs
 
blocked jobs-----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total jobs: 2

    Note that test1 is canceled when test2 is submitted.

    (Optional) Examine the checkjob outputs for these two jobs:

    [john@g06]$ checkjob Moab.9
job Moab.9
 
State: Removed
Completetion Code: -1 Time: Thu Nov 10 12:28:48
Creds: user:john group:john qos:test1
WallTime: 00:00:02 of 00:02:00
SubmitTime: Thu Nov 10 12:28:44
(Time Queued Total: 00:00:07 Eligible: 00:00:00)
 
Total Requested Tasks: 128
 
Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2
 
Allocated Nodes:
node[01-02]*64
 
 
IWD: /opt/native
SubmitDir: /opt/native
Executable: /opt/native/spool/moab.job.UFe8sQ

StartCount: 1
Flags: GLOBALQUEUE,PROCSPECIFIED
Attr: PREEMPTEE
StartPriority: 100

    Note that the preempted job has been removed.

    [john@g06]$ checkjob Moab.10
job Moab.10
 
State: Running
Creds: user:john group:john qos:test2
WallTime: 00:00:00 of 00:02:00
SubmitTime: Thu Nov 10 12:36:31
(Time Queued Total: 00:00:00 Eligible: 00:00:00)
 
StartTime: Thu Nov 10 12:28:48
Total Requested Tasks: 128
 
Req[0] TaskCount: 128 Partition: licenses
 
Allocated Nodes:
node[01-02]*64
 
 
IWD: /opt/native
SubmitDir: /opt/native
Executable: /opt/native/spool/moab.job.CZavjU
 
StartCount: 1
Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED
StartPriority: 10000
Reservation 'Moab.10' (-00:00:07 -> 00:01:53 Duration: 00:02:00)

Related topics