The REQUEUE attribute for PREEMPTPOLICY terminates active jobs and returns them to the job queue in an idle state.

You must mark a job as RESTARTABLE if you want it to requeue. If not, the job will be canceled. If supported by the resource manager, you can set the job RESTARTABLE flag when submitting the job by using the msub -r option. Otherwise, use the JOBFLAGS attribute of the associated class or QoS credential, as in this example: CLASSCFG[low] JOBFLAGS=RESTARTABLE

The following outlines some benefits of using REQUEUE and also lists some things you should be aware of if you choose to use it.

Advantages:

• Jobs are automatically resubmitted into the job queue.

Cautions:

• A job gets resubmitted in the job queue at the same priority it had when Moab originally started it (i.e., the job does not jump ahead in the queue).
• Jobs start over from the beginning.

To use REQUEUE

The following steps explain and illustrate how to set up preemption with REQUEUE.

1. Make the following configurations to the moab.cfg file:

GUARANTEEDPREEMPTION TRUE
 
PREEMPTPOLICY REQUEUE
 
QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100 
QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000

• Set GUARANTEEDPREEMPTION to TRUE. (This locks the job on a node and keeps trying to preempt.)
• Make sure that JOBNODEMATCHPOLICY is not set to EXACTNDODE.
• Set PREEMPTPOLCY to REQUEUE.

PREEMPTPOLICY REQUEUE

• For the PREEMPTEE job, set JOBFLAGS=RESTARTABLE.

QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100

• Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job.

QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100
QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000

2. Submit a job to the PREEMPTEE QoS (test1). For example:

[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128

3. Examine the following output for showq:

Moab.1
[john@g06]# showq
 
active jobs------------------------ 
JOBID     USERNAME    STATE      PROCS    REMAINING    STARTTIME 
Moab.1    john        Running    128      00:09:59     Wed Nov 9 15:56:33
 
1 active job     128 of 128 processors in use by local jobs (100.00%) 
2 of 2 nodes active (100.00%)
 
eligible jobs---------------------- 
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 eligible jobs
 
blocked jobs----------------------- 
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total job: 1

4. Now submit a job to the PREEMPTEE QoS (test2). For example:

[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128

5. Examine the following output for showq and checkjob:

showq test2:

Moab.2
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS    REMAINING    STARTTIME
Moab.2    john         Running    128      00:09:59     Wed Nov 9 15:56:47
 
1 active job 128 of 128 processors in use by local jobs (100.00%)
2 of 2 nodes active (100.00%)
 
eligible jobs---------------------- 
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
Moab.1    john        Idle      128       00:10:00     Wed Nov 9 15:56:33
 
1 eligible job
 
blocked jobs----------------------- 
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total jobs: 2

checkjob test2:

[john@g06]# checkjob Moab.2
job Moab.2
 
State: Running
Creds: user:john group:john qos:test2
WallTime: 00:02:04 of 00:10:00
SubmitTime: Wed Nov 9 15:56:46
(Time Queued Total: 00:00:01 Eligible: 00:00:00)
StartTime: Wed Nov 9 15:56:47

Total Requested Tasks: 128

Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2

Allocated Nodes:
node[01-02]*64
 

IWD: /opt/native
SubmitDir: /opt/native
Executable: /opt/native/spool/moab.job.ELoX5Q
 
StartCount: 1 
Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED 
StartPriority: 10000 
Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)

See Also

• Choosing a PREEMPTPOLICY type