23.1.10 Simple example of preemption
This section illustrates the process of setting up preemption on your system from beginning to end and contains examples of what actions to take and what you should see as you go.
Example scenario
For this basic setup example, we will have a user who can submit to either a "test1" or "test2" QoS. This example will use a REQUEUE preemption type.
We will go through three parts to set up this preemption:
- Configuring the moab.cfg file
- Submitting a job to the PREEMPTEE QoS
- Submitting a job to the PREEMPTOR QoS
Okay, let's get started!
Configuring moab.cfg
First, you will need to make some configurations to the moab.cfg file.
- Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
- Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and troubleshooting preemption).
- Set the PREEMPTPOLICY type. In this example, PREEMPTPOLICY is set to REQUEUE. For more information, see PREEMPTPOLICY types.
- Set up QFLAGS to mark jobs as PREEMPTEE (a lower-priority job that can be preempted by a higher-priority job), or as PREEMPTOR (a higher-priority job that can preempt a lower-priority job). For more information, see Preemption flags.
- Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job.
For this example, we also set JOBFLAGS=RESTARTABLE (because this example uses REQUEUE). For more information, see Requeueing jobs with preemption.
Here is an example of how that would all look in a moab.cfg file (text marked red for emphasis).
GUARANTEEDPREEMPTION TRUE #should not be JOBNODEMATCHPOLICY EXACTNODE as it causes problems when starting jobs PREEMPTPOLICY REQUEUE QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100 QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=1000
Now you can submit a job to the preemptee QoS (test1).
Submitting a job to the preemptee QoS
Let's submit a job to the preemptee QoS (test1), requesting all processor cores in the cluster:
[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128
Take a look at the showq and checkjob output:
Moab.1
[john@g06]# showq
active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME
Moab.1 john Running 128 00:09:59 Wed Nov 9 15:56:33
1 active job 128 of 128 processors in use by local jobs (100.00%)
2 of 2 nodes active (100.00%)
eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 eligible jobs
blocked jobs-----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 blocked jobs
Total job: 1
[john@g06]# checkjob Moab.1 job Moab.1 State: Running Creds: user:john group:john qos:test1 WallTime: 00:00:00 of 00:10:00 SubmitTime: Wed Nov 9 15:56:33 (Time Queued Total: 00:00:00 Eligible: 00:00:00) StartTime: Wed Nov 9 15:56:33 Total Requested Tasks: 128 Req[0] TaskCount: 128 Partition: licenses Allocated Nodes: node[01-02]*64 IWD: /opt/native/ SubmitDir: /opt/native/ Executable: /opt/native/spool/moab.job.zOyf1N StartCount: 1 Flags: RESTARTABLE,PREEMPTEE,GLOBALQUEUE,PROCSPECIFIED Attr: PREEMPTEE StartPriority: 100 Reservation 'Moab.1' (-00:00:03 -> 00:09:57 Duration: 00:10:00
Submitting a job to the preemptor QoS
Now we will submit a preemptor QoS job (test2) to preempt the first job (test1):
[john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128
Examine the following output for showq and checkjob:
Moab.2 [john@g06]# showq active jobs------------------------ JOBID USERNAME STATE PROCS REMAINING STARTTIME Moab.2 john Running 128 00:09:59 Wed Nov 9 15:56:47 1 active job 128 of 128 processors in use by local jobs (100.00%) 2 of 2 nodes active (100.00%) eligible jobs---------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME Moab.1 john Idle 128 00:10:00 Wed Nov 9 15:56:33 1 eligible job blocked jobs----------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 blocked jobs Total jobs: 2
Note that the preemptor job (Moab.2) moved to Running, while the preemptee job (Moab.1) was requeued.
[john@g06]# checkjob Moab.2
job Moab.2
State: Running
Creds: user:john group:john qos:test2
WallTime: 00:02:04 of 00:10:00
SubmitTime: Wed Nov 9 15:56:46
(Time Queued Total: 00:00:01 Eligible: 00:00:00)
StartTime: Wed Nov 9 15:56:47
Total Requested Tasks: 128
Req[0] TaskCount: 128 Partition: licenses
NodeCount: 2
Allocated Nodes:
node[01-02]*64
IWD: /opt/native/
SubmitDir: /opt/native/
Executable: /opt/native/spool/moab.job.ELoX5Q
StartCount: 1
Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED
StartPriority: 10000
Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)
Note the flag, HASPREEMPTED. HASPREEMPTED is set when the PREEMPTOR job has preempted the PREEMPTEE job. Also note that the preemptor job priority plays a very big role in preemption. Generally, you should assign the preemptor a higher priority than any other queued jobs so that it will move to (or near to) the top of the eligible queue.
Related topics