You are here: 21 Preemption > References > Simple Example of Preemption

21.8 Simple Example of Preemption

This section illustrates the process of setting up preemption on your system from beginning to end and contains examples of what actions to take and what you should see as you go.

Example scenario

For this basic setup example, we will have a user who can submit to either a "test1" or "test2" QoS. This example will use a REQUEUE preemption type.

We will go through three parts to set up this preemption:

Configuring the moab.cfg file
Submitting a job to the PREEMPTEE QoS
Submitting a job to the PREEMPTOR QoS

Okay, let's get started!

21.8.1 Configuring moab.cfg

First, you will need to make some configurations to the moab.cfg file.

Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and Troubleshooting Preemption).
Set the PREEMPTPOLICY type. In this example, PREEMPTPOLICY is set to REQUEUE. For more information, see PREEMPTPOLICY Types.
Set up QFLAGS to mark jobs as PREEMPTEE (a lower-priority job that can be preempted by a higher-priority job), or as PREEMPTOR (a higher-priority job that can preempt a lower-priority job). For more information, see Preemption Flags.

For this example, we also set JOBFLAGS=RESTARTABLE (because this example uses REQUEUE). For more information, see Requeueing Jobs with Preemption.

Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job.

Here is an example of how that would all look in a moab.cfg file (text marked red for emphasis).

GUARANTEEDPREEMPTION TRUE
#should not be JOBNODEMATCHPOLICY EXACTNODE as it causes problems when starting jobs
 
PREEMPTPOLICY REQUEUE
 
QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100
QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=1000

Now you can submit a job to the preemptee QoS (test1).

21.8.2 Submitting a Job to the Preemptee QoS

Let's submit a job to the preemptee QoS (test1), requesting all processor cores in the cluster:

[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128

Take a look at the showq and checkjob output:

Moab.1
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS     REMAINING     STARTTIME
Moab.1    john        Running    128       00:09:59      Wed Nov 9 15:56:33
 
1 active job     128 of 128 processors in use by local jobs (100.00%)
                 2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 eligible jobs
 
blocked jobs-----------------------
JOBID     USERNAME    STATE     PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total job: 1

[john@g06]# checkjob Moab.1
job Moab.1
 
State: Running
Creds: user:john group:john qos:test1
WallTime: 00:00:00 of 00:10:00
SubmitTime: Wed Nov 9 15:56:33
(Time Queued Total: 00:00:00 Eligible: 00:00:00)
 
StartTime: Wed Nov 9 15:56:33
Total Requested Tasks: 128
 
Req[0] TaskCount: 128 Partition: licenses
 
Allocated Nodes:
node[01-02]*64
 
IWD: /opt/native/
SubmitDir: /opt/native/
Executable: /opt/native/spool/moab.job.zOyf1N
 
StartCount: 1
Flags: RESTARTABLE,PREEMPTEE,GLOBALQUEUE,PROCSPECIFIED
Attr: PREEMPTEE
StartPriority: 100
Reservation 'Moab.1' (-00:00:03 -> 00:09:57 Duration: 00:10:00

21.8.3 Submitting a job to the preemptor QoS

Now we will submit a preemptor QoS job (test2) to preempt the first job (test1):

[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128

Examine the following output for showq and checkjob:

Moab.2
[john@g06]# showq
 
active jobs------------------------
JOBID     USERNAME    STATE      PROCS      REMAINING     STARTTIME
Moab.2    john        Running    128        00:09:59      Wed Nov 9 15:56:47
 
1 active job 128 of 128 processors in use by local jobs (100.00%)
             2 of 2 nodes active (100.00%)
 
eligible jobs----------------------
JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
Moab.1    john        Idle       128        00:10:00      Wed Nov 9 15:56:33
 
1 eligible job
 
blocked jobs-----------------------
JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
 
0 blocked jobs
 
Total jobs: 2

Note that the preemptor job (Moab.2) moved to Running, while the preemptee job (Moab.1) was requeued.

[john@g06]# checkjob Moab.2
job Moab.2
 
State: Running
Creds: user:john group:john qos:test2
WallTime: 00:02:04 of 00:10:00
SubmitTime: Wed Nov 9 15:56:46
(Time Queued Total: 00:00:01 Eligible: 00:00:00)
 
StartTime: Wed Nov 9 15:56:47
Total Requested Tasks: 128
 
Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2
 
Allocated Nodes:
node[01-02]*64
 
 
IWD: /opt/native/
SubmitDir: /opt/native/
Executable: /opt/native/spool/moab.job.ELoX5Q
 
StartCount: 1
Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED
StartPriority: 10000
Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)

Note the flag, HASPREEMPTED. HASPREEMPTED is set when the PREEMPTOR job has preempted the PREEMPTEE job. Also note that the preemptor job priority plays a very big role in preemption. Generally, you should assign the preemptor a higher priority than any other queued jobs so that it will move to (or near to) the top of the eligible queue.