23.0 Preemption > Preemption tasks > Requeueing jobs with preemption

Conventions

23.1.3 Requeueing jobs with preemption

You must mark a job as RESTARTABLE if you want it to requeue. If you do not, the job will be canceled when it is preempted.

If supported by the resource manager, you can set the RESTARTABLE job flag when submitting the job by using the msub -r option. Otherwise, use the JOBFLAGS attribute of the associated class or QoS credential, as in this example:

CLASSCFG[low] JOBFLAGS=RESTARTABLE

For more information, see Job Flags.

To preempt jobs using REQUEUE

  1. Make the following configurations to the moab.cfg file:
    1. Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)
    2. Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and troubleshooting preemption).
    3. Set PREEMPTPOLICY to REQUEUE (for more information, see PREEMPTPOLICY types).
    4. Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption flags).
  2. For example:

    GUARANTEEDPREEMPTION TRUE
    PREEMPTPOLICY REQUEUE
     
    QOSCFG[test1] QFLAGS=PREEMPTEE JOBFLAGS=RESTARTABLE MEMBERULIST=john PRIORITY=100
    QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
  3. Submit a job to the preemptee QoS (test1). For example:
  4. [john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test1 -l procs=128

    (Optional) Examine the following output for showq:

    Moab.1
    [john@g06]# showq
     
    active jobs------------------------
    JOBID     USERNAME    STATE      PROCS     REMAINING      STARTTIME
    Moab.1    john        Running    128       00:09:59       Wed Nov 9 15:56:33
     
    1 active job     128 of 128 processors in use by local jobs (100.00%)
    	          2 of 2 nodes active (100.00%)
     
    eligible jobs----------------------
    JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
     
    0 eligible jobs
     
    blocked jobs-----------------------
    JOBID     USERNAME    STATE      PROCS      WCLIMIT       QUEUETIME
     
    0 blocked jobs
     
    Total job: 1
  5. Now submit a job to the preemptor QoS (test2). For example:
  6. [john@g06]# echo sleep 600 | msub -l walltime=600 -l qos=test2 -l procs=128

    (Optional) Examine the following output for showq and checkjob:

    Moab.2
    [john@g06]# showq
     
    active jobs------------------------
    JOBID     USERNAME     STATE      PROCS    REMAINING     STARTTIME
    Moab.2    john         Running    128      00:09:59      Wed Nov 9 15:56:47
     
    1 active job 128 of 128 processors in use by local jobs (100.00%)
    	      2 of 2 nodes active (100.00%)
     
    eligible jobs----------------------
    JOBID     USERNAME    STATE       PROCS    WCLIMIT       QUEUETIME
    Moab.1    john        Idle        128      00:10:00      Wed Nov 9 15:56:33
     
    1 eligible job
     
    blocked jobs-----------------------
    JOBID     USERNAME    STATE       PROCS    WCLIMIT       QUEUETIME
     
    0 blocked jobs
     
    Total jobs: 2
    [john@g06]# checkjob Moab.2
    job Moab.2
     
    State: Running
    Creds: user:john group:john qos:test2
    WallTime: 00:02:04 of 00:10:00
    SubmitTime: Wed Nov 9 15:56:46
    (Time Queued Total: 00:00:01 Eligible: 00:00:00)
     
    StartTime: Wed Nov 9 15:56:47
    Total Requested Tasks: 128
     
    Req[0] TaskCount: 128 Partition: licenses
    NodeCount: 2
     
    Allocated Nodes:
    node[01-02]*64
     
     
    IWD: /opt/native
    SubmitDir: /opt/native
    Executable: /opt/native/spool/moab.job.ELoX5Q
     
    StartCount: 1
    Flags: HASPREEMPTED,PREEMPTOR,GLOBALQUEUE,PROCSPECIFIED
    StartPriority: 10000
    Reservation 'Moab.2' (-00:02:21 -> 00:07:39 Duration: 00:10:00)

Related topics