(Click to open topic with navigation)
Two types of job deadlines exist in Moab. The priority-based deadline linearly increases a job's priority as its deadline approaches (See Deadline (DEADLINE) Subcomponent for more information). The QoS method allows you to set a job completion time on job submission if, and only if, it requests and is allowed to access a QoS with the DEADLINE QFLAG set. This method is more powerful than the priority method, because Moab will attempt to make a reservation for the job as soon as the job enters the queue in order to meet the deadline, essentially bumping it to the front of the queue.
When a job is submitted to a QoS with the DEADLINE flag set, the job's -l deadline attribute is honored. If such QoS access is not available, or if resources do not exist at job submission time to allow the deadline to be satisfied, the job's deadline request is ignored.
Two methods exist for setting deadlines with a QoS:
This method of setting a job deadline allows you to specify a job deadline as you submit the job. You can set the deadline as either an exact date and time or as an amount of time after job submission (i.e. three hours after submission).
To specify a deadline on job submission
Jobs requesting the QoS special may submit jobs with a deadline that Moab will honor.
msub -l qos=special deadline=08:00:00_03/15/13 job.sh
The job must finish running by 8 A.M. on March 15, 2013.
msub -l qos=special deadline=5:00:00 job.sh
The job's deadline is 5 hours after its submission.
You may also set a relative job deadline by limiting the job's queue time. This method allows you to pre-configure the deadline rather than giving the power to specify a deadline to the user submitting the job. For jobs requesting these QoSes, Moab identifies and sets job deadlines to satisfy the corresponding response time targets.
To submit a job to a QoS with a preconfigured deadline
QOSCFG[special2] QFLAGS=DEADLINE QTTARGET=1:00:00
Given this configuration, a job requesting QoS special2 must spend a maximum of one hour in the queue.
msub -l qos=special2 walltime=2:00:00 job.sh
This two-hour job has a completion time deadline set to three hours after its submission (one hour of target queue time and two hours of run time).
In addition to job completion targets, jobs may also be submitted with a TERMTIME attribute. The scheduler attempts to complete the job prior to the termination date, but if it is unsuccessful, it will terminate (cancel) the job once the termination date is reached.
The specific policy can be configured using the DEADLINEPOLICY parameter. Moab does not have a default policy for this parameter.
|CANCEL||The job is canceled and the user is notified that the deadline could not be satisfied.|
|HOLD||The job has a batch hold placed on it indefinitely. The administrator can then decide what action to take.|
|RETRY||The job continually retries each iteration to meet its deadline; note that when used with QTTARGET the job's deadline continues to slide with relative time.|
|IGNORE||The job has its request ignored and is scheduled as normal.|
Deadline scheduling may not function properly with per partition scheduling enabled. Check that PARALLOCATIONPOLICY is disabled to ensure DEADLINEPOLICY will work correctly.