Requeueing Jobs with Preemption
Context
REQUEUE is one of the PREEMPTPOLICY types (for more information, see PREEMPTPOLICY Types). The REQUEUE value terminates active jobs and returns them to the job queue in an idle state.
For information about PREEPMPTEE and PREEMPTOR flags, see Preemption Flags
The following outlines some benefits of using REQUEUE and also lists some things you should be aware of if you choose to use it.
Advantages:
- Jobs are automatically resubmitted into the job queue.
Cautions:
- A job gets resubmitted in the job queue at the same priority it had when Moab originally started it (i.e., the job does not jump ahead in the queue).
- Jobs start over from the beginning.
You must mark a job as RESTARTABLE if you want it to requeue. If you do not, the job will be canceled when it is preempted.
If supported by the resource manager, you can set the RESTARTABLE job flag when submitting the job by using the msub -r option. Otherwise, use the JOBFLAGS attribute of the associated class or QoS credential, as in this example:
CLASSCFG[low] JOBFLAGS=RESTARTABLE
For more information, see Job Flags.
To preempt jobs using REQUEUE
- Make the following configurations to the
- Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
- Set PREEMPTIONALGORITHM to specify how Moab handles preemption scheduling policies.
If you use JOBNODEMATCHPOLICY EXACTNODE, you must also add PREEMPTIONALGORITHM PREEMPTORCENTRIC in order for preemption to function reliably.
- Set PREEMPTPOLICY to REQUEUE (for more information, see PREEMPTPOLICY Types).
- Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption Flags).
- Submit a job to the preemptee QoS (test1). For example:
- Now submit a job to the preemptor QoS (test2). For example:
For example:
GUARANTEEDPREEMPTION TRUE PREEMPTPOLICY REQUEUE QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100 QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128
(Optional) Examine the following output for showq:
Moab.1 [john@g06]# showq active jobs------------------------ JOBID USERNAME STATE PROCS REMAINING STARTTIME Moab.1 john Running 128 00:09:59 Wed Nov 9 15:56:33 1 active job 128 of 128 processors in use by local jobs (100.00%) 2 of 2 nodes active (100.00%) eligible jobs---------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 eligible jobs blocked jobs----------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 blocked jobs Total job: 1
[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128
(Optional) Examine the following output for showq and checkjob:
Moab.2 [john@g06]# showq active jobs------------------------ JOBID USERNAME STATE PROCS REMAINING STARTTIME Moab.2 john Running 128 00:09:59 Wed Nov 9 15:56:47 1 active job 128 of 128 processors in use by local jobs (100.00%) 2 of 2 nodes active (100.00%) eligible jobs---------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME Moab.1 john Idle 128 00:10:00 Wed Nov 9 15:56:33 1 eligible job blocked jobs----------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 blocked jobs Total jobs: 2
[john@g06]# checkjob Moab.2 job Moab.2 State: Running Creds: user:john group:john qos:test2 WallTime: 00:02:04 of 00:10:00 SubmitTime: Wed Nov 9 15:56:46 (Time Queued Total: 00:00:01 Eligible: 00:00:00) StartTime: Wed Nov 9 15:56:47 Total Requested Tasks: 128 Req[0] TaskCount: 128 Partition: licenses NodeCount: 2 Allocated Nodes: node[01-02]*64 IWD: /opt/native SubmitDir: /opt/native Executable: /opt/native/spool/moab.job.ELoX5Q StartCount: 1 Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED StartPriority: 10000 Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)
Related Topics