(Click to open topic with navigation)

23.0 Preemption > Preemption tasks > Requeueing jobs with preemption

Conventions

23.1.3 Requeueing jobs with preemption

ClosedContext

REQUEUE is one of the PREEMPTPOLICY types (for more information, see PREEMPTPOLICY types). The REQUEUE value terminates active jobs and returns them to the job queue in an idle state.

For information about PREEPMPTEE and PREEMPTOR flags, see Preemption flags

The following outlines some benefits of using REQUEUE and also lists some things you should be aware of if you choose to use it.

Advantages:

  • Jobs are automatically resubmitted into the job queue.

Cautions:

  • A job gets resubmitted in the job queue at the same priority it had when Moab originally started it (i.e., the job does not jump ahead in the queue).
  • Jobs start over from the beginning.

You must mark a job as RESTARTABLE if you want it to requeue. If you do not, the job will be canceled when it is preempted.

If supported by the resource manager, you can set the RESTARTABLE job flag when submitting the job by using the msub -r option. Otherwise, use the JOBFLAGS attribute of the associated class or QoS credential, as in this example:

CLASSCFG[low] JOBFLAGS=RESTARTABLE

For more information, see Job Flags.

To preempt jobs using REQUEUE

  1. Make the following configurations to the moab.cfg file:
    1. Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
    2. Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and troubleshooting preemption).
    3. Set PREEMPTPOLICY to REQUEUE (for more information, see PREEMPTPOLICY types).
    4. Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption flags).

    2. For example:

    GUARANTEEDPREEMPTION TRUE
PREEMPTPOLICY REQUEUE
 
QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100
QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
  2. Submit a job to the preemptee QoS (test1). For example:
    3. [john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128

    (Optional) Examine the following output for showq:

    Moab.1
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS     REMAINING      STARTTIME
Moab.1    john        Running    128       00:09:59       Wed Nov 9 15:56:33
 
1 active job     128 of 128 processors in use by local jobs (100.00%)
	          2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
 
0 eligible jobs
 
blocked jobs-----------------------
JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total job: 1
  3. Now submit a job to the preemptor QoS (test2). For example:
    4. [john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128

    (Optional) Examine the following output for showq and checkjob:

    Moab.2
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME     STATE      PROCS    REMAINING     STARTTIME
Moab.2    john         Running    128      00:09:59      Wed Nov 9 15:56:47
 
1 active job 128 of 128 processors in use by local jobs (100.00%)
	      2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE       PROCS    WCLIMIT       QUEUETIME
Moab.1    john        Idle        128      00:10:00      Wed Nov 9 15:56:33
 
1 eligible job
 
blocked jobs-----------------------
JOBID     USERNAME    STATE       PROCS    WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total jobs: 2
    [john@g06]# checkjob Moab.2
job Moab.2
 
State: Running
Creds: user:john group:john qos:test2
WallTime: 00:02:04 of 00:10:00
SubmitTime: Wed Nov 9 15:56:46
(Time Queued Total: 00:00:01 Eligible: 00:00:00)
 
StartTime: Wed Nov 9 15:56:47
Total Requested Tasks: 128
 
Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2
 
Allocated Nodes:
node[01-02]*64
 
 
IWD: /opt/native
SubmitDir: /opt/native
Executable: /opt/native/spool/moab.job.ELoX5Q
 
StartCount: 1
Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED
StartPriority: 10000
Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)

Related topics