(Click to open topic with navigation)

10.0 General Job Administration > Job Deadlines

Conventions

10.9 Job Deadlines

10.9-A Deadline Overview

Job deadlines may be specified on a per job and per credential basis and are also supported using both absolute and QoS based specifications. A job requesting a deadline is first evaluated to determine if the deadline is acceptable. If so, Moab adds it to the list of deadline jobs and allocates resources to guarantee that all accepted deadline jobs are able to complete on or before their requested deadline. Once the scheduler confirms that all deadlines can be satisfied, it then optimizes resource allocation (in priority order) attempting to execute all jobs at the earliest possible time.

10.9-B Setting Job Deadlines via QoS

Two types of job deadlines exist in Moab. The priority-based deadline linearly increases a job's priority as its deadline approaches (See Deadline (DEADLINE) Subcomponent for more information). The QoS method allows you to set a job completion time on job submission if, and only if, it requests and is allowed to access a QoS with the DEADLINE QFLAG set. This method is more powerful than the priority method, because Moab will attempt to make a reservation for the job as soon as the job enters the queue in order to meet the deadline, essentially bumping it to the front of the queue.

When a job is submitted to a QoS with the DEADLINE flag set, the job's -l deadline attribute is honored. If such QoS access is not available, or if resources do not exist at job submission time to allow the deadline to be satisfied, the job's deadline request is ignored.

Two methods exist for setting deadlines with a QoS: 

Setting Job Deadlines at Job Submission

This method of setting a job deadline allows you to specify a job deadline as you submit the job. You can set the deadline as either an exact date and time or as an amount of time after job submission (i.e. three hours after submission).

To specify a deadline on job submission

  1. In moab.cfg, reate a QoS with the DEADLINE flag enabled.
    ...
    QOSCFG[special] QFLAGS=DEADLINE

    Jobs requesting the QoS special may submit jobs with a deadline that Moab will honor.

  2. Submit a job to the QoS and set a deadline. This can be either absolute or relative.
    1. For an absolute deadline, use the format hh:mm:ss_mm/dd/yy. The following configuration sets a deadline for a job to finish by 8 a.m. on March 15th, 2013.
      msub -l qos=special deadline=08:00:00_03/15/13 job.sh

      The job must finish running by 8 A.M. on March 15, 2013.

    2. For a relative deadline, or the completion deadline of the job relative to its submission time, use the time format [[[DD:]HH:]MM:]SS.
      msub -l qos=special deadline=5:00:00 job.sh

      The job's deadline is 5 hours after its submission.

Submitting a Job to a QoS with a Preconfigured Deadline

You may also set a relative job deadline by limiting the job's queue time. This method allows you to pre-configure the deadline rather than giving the power to specify a deadline to the user submitting the job. For jobs requesting these QoSes, Moab identifies and sets job deadlines to satisfy the corresponding response time targets.

To submit a job to a QoS with a preconfigured deadline

  1. In moab.cfg, create a QoS with both the DEADLINE QFLAG and a response time target (QTTARGET). The QTTARGET is the maximum amount of time that Moab should allow the job to be idle in the queue.
    ...
    QOSCFG[special2] QFLAGS=DEADLINE QTTARGET=1:00:00

    Given this configuration, a job requesting QoS special2 must spend a maximum of one hour in the queue.

  2. Submit a job requesting the special2 quality of service.
    msub -l qos=special2 walltime=2:00:00 job.sh

    This two-hour job has a completion time deadline set to three hours after its submission (one hour of target queue time and two hours of run time).

10.9-C Job Termination Date

In addition to job completion targets, jobs may also be submitted with a TERMTIME attribute. The scheduler attempts to complete the job prior to the termination date, but if it is unsuccessful, it will terminate (cancel) the job once the termination date is reached.

10.9-D Conflict Policies

The specific policy can be configured using the DEADLINEPOLICY parameter. Moab does not have a default policy for this parameter.

Policy Description
CANCEL The job is canceled and the user is notified that the deadline could not be satisfied.
HOLD The job has a batch hold placed on it indefinitely. The administrator can then decide what action to take.
RETRY The job continually retries each iteration to meet its deadline; note that when used with QTTARGET the job's deadline continues to slide with relative time.
IGNORE The job has its request ignored and is scheduled as normal.

Deadline scheduling may not function properly with per partition scheduling enabled. Check that PARALLOCATIONPOLICY is disabled to ensure DEADLINEPOLICY will work correctly.

Related topics