Conventions

7.3 Quality of Service (QoS) Facilities

This section describes how to do the following:

It contains the following sub-sections:

7.3-A QoS Overview

Moab's QoS facility allows a site to give special treatment to various classes of jobs, users, groups, and so forth. Each QoS object can be thought of as a container of special privileges ranging from fairness policy exemptions, to special job prioritization, to special functionality access. Each QoS object also has an extensive access list of users, groups, and accounts that can access these privileges.

Sites can configure various QoSes each with its own set of priorities, policy exemptions, and special resource access settings. They can then configure user, group, account, and class access to these QoSes. A given job will have a default QoS and may have access to several additional QoSes. When the job is submitted, the submitter may request a specific QoS or just allow the default QoS to be used. Once a job is submitted, a user may adjust the QoS of the job at any time using the setqos command. The setqos command will only allow the user to modify the QoS of that user's jobs and only change the QoS to a QoS that this user has access to. Moab administrators may change the QoS of any job to any value.

Jobs can be granted access to QoS privileges if the QoS is listed in the system default configuration QDEF (QoS default) or QLIST (QoS access list), or if the QoS is specified in the QDEF or QLIST of a user, group, account, or class associated with that job. Alternatively, a user may access QoS privileges if that user is listed in the QoS's MEMBERULIST attribute.

The mdiag -q command can be used to obtain information about the current QoS configuration including specified credential access.

7.3-B QoS Enabled Privileges

The privileges enabled via QoS settings may be broken into the following categories:

All privileges are managed via the QOSCFG parameter.

Special Prioritization

Attribute name Description
FSTARGET Specifies QoS fairshare target.
FSWEIGHT Sets QoS fairshare weight offset affecting a job's fairshare priority component.
PRIORITY Assigns priority to all jobs requesting particular QoS.
QTTARGET Sets QoS queuetime target affecting a job's target priority component and QoS delivered.
QTWEIGHT Sets QoS queuetime weight offset affecting a job's service priority component.
XFTARGET Sets QoS XFactor target affecting a job's target priority component and QoS delivered.
XFWEIGHT Sets QoS XFactor weight offset affecting a job's service priority component.

Example 7-27:  

# assign priority for all QoS geo jobs 

QOSCFG[geo]  PRIORITY=10000

Service Access and Constraints

The QoS facility can be used to enable special services and to disable default services. These services are enabled/disabled by setting the QoS QFLAGS attribute.

Flag Name Description
DEADLINE Job may request an absolute or relative completion deadline and Moab will reserve resources to meet that deadline. (An alternative priority based deadline behavior is discussed in the PRIORITY FACTORS section.)
DEDICATED Moab dedicates all resources of an allocated node to the job meaning that the job will not share a node's compute resources with any other job.
ENABLEUSERRSV Allow user or personal reservations to be created and managed.
IGNALL Scheduler ignores all resource usage policies for jobs associated with this QoS.
JOBPRIOACCRUALPOLICY

Specifies how Moab should track the dynamic aspects of a job's priority. The two valid values are ACCRUE and RESET.

  • ACCRUE indicates that the job will accrue queuetime based priority from the time it is submitted unless it violates any of the policies not specified in JOBPRIOEXCEPTIONS.
  • RESET indicates that it will accrue priority from the time it is submitted unless it violates any of the JOBPRIOEXCEPTIONS. However, with RESET, if the job does violate JOBPRIOEXCEPTIONS then its queuetime based priority will be reset to 0.

JOBPRIOACCRUALPOLICY is a global parameter, but can be configured to work only in QOSCFG:

QOSCFG[arrays]  JOBPRIOACCRUALPOLICY=ACCRUE

The following old JOBPRIOACCRUALPOLICY values have been deprecated and should be adjusted to the following values:

  • QUEUEPOLICY = ACCRUE and JOBPRIOEXCEPTIONS SOFTPOLICY,HARDPOLICY
  • QUEUEPOLICYRESET = RESET and JOBPRIOEXCEPTIONS SOFTPOLICY,HARDPOLICY
  • ALWAYS = ACCRUE and JOBPRIOEXCEPTIONS ALL
  • FULLPOLICY = ACCRUE and JOBPRIOEXCEPTIONS NONE
  • FULLPOLICYRESET = RESET and JOBPRIOEXCEPTIONS NONE
JOBPRIOEXCEPTIONS

Specifies exceptions for calculating a job's dynamic priority (QUEUETIME, XFACTOR, TARGETQUEUETIME). Valid values are a comma delimited list of any of the following: DEFER, DEPENDS, SOFTPOLICY, HARDPOLICY, IDLEPOLICY, USERHOLD, BATCHHOLD, and SYSTEMHOLD (ALL or NONE can also be specified on their own).

Normally, when a job violates a policy, is placed on hold, or has an unsatisfied dependency, it will not accrue priority. Exceptions can be configured to allow a job to accrue priority in spite of any of these violations. With DEPENDS a job will increase in priority even if there exists an unsatisfied dependency. With SOFTPOLICY, HARDPOLICY, or IDLEPOLICY a job can accrue priority despite violating a specific limit. With DEFER, USERHOLD, BATCHHOLD, or SYSTEMHOLD a job can accrue priority despite being on hold.

JOBPRIOEXCEPTIONS is a global parameter, but can be configured to work only in QOSCFG:

QOSCFG[arrays]  JOBPRIOEXCEPTIONS=IDLEPOLICY
NOBF Job is not considered for backfill.
NORESERVATION Job should never reserve resources regardless of priority.
NTR

Job is prioritized as next to run (NTR) and backfill is disabled to prevent other jobs from jumping in front of ones with the NTR flag.

It is important to note that jobs marked with this flag should not be blocked. If they are, Moab will stop scheduling because if a job is marked with this flag, no other jobs will be run until the flagged NTR (Next to Run) job starts. Consider using the PRIORITY attribute of the QOSCFG[<QOSID>] parameter instead, when possible. Or, as you may encounter a scheduling delay for NTR-flagged jobs to start, consider using the RESERVATIONDEPTH[X] and RESERVATIONQOSLIST[X] parameters to provide better scheduling flow. See Reservation Policies (especially the section on Assigning Per-QoS Reservation Creation Rules) for more information.

PREEMPTCONFIG User jobs may specify options to alter how preemption impacts the job such as minpreempttime.
PREEMPTEE Job may be preempted by higher priority PREEMPTOR jobs.
PREEMPTFSV Job may be preempted by higher priority PREEMPTOR jobs if it exceeds its fairshare target when started.
PREEMPTOR Job may preempt lower priority PREEMPTEE jobs.
PREEMPTSPV Job may be preempted by higher priority PREEMPTOR jobs if it currently violates a soft usage policy limit.
PROVISION If the job cannot locate available resources with the needed OS or software, the scheduler may provision a number of nodes to meet the needed OS or software requirements.
RESERVEALWAYS Job should create resource reservation regardless of job priority.
RUNNOW Boosts a job's system priority and makes the job a preemptor.

RUNNOW overrides resource restrictions such as MAXJOB or MAXPROC.

TRIGGER The job is able to directly specify triggers.
USERESERVED[:<RSVID>] Job may only use resources within accessible reservations. If <RSVID> is specified, job may only use resources within the specified reservation.

Example 7-28: For lowprio QoS job, disable backfill and make job preemptible

QOSCFG[lowprio]  QFLAGS=NOBF,PREEMPTEE

Example 7-29: Bind all jobs to chemistry reservation

QOSCFG[chem-b]  QFLAGS=USERESERVED:chemistry

Other QoS Attributes

In addition to the flags, there are attributes that alter service access.

Attribute name Description
SYSPRIO Sets the system priority on jobs associated with this QoS.

Example 7-30: All jobs submitted under a QoS sample receive a system priority of 1

QOSCFG[sample]  SYSPRIO=1

Per QoS Required Reservations

If desired, jobs associated with a particular QoS can be locked into a reservation or reservation group using the REQRID attribute. For example, to force jobs using QoS jasper to only use the resources within the failsafe standing reservation, use the following:

QOSCFG[jasper] REQRID=failsafe
...

Setting Per QoS Network Service Levels

Minimum network service levels can be specified using the MINBANDWIDTH and MAXLATENCY attributes. If specified, Moab does not allow a job to start until these criteria can be satisfied and allocates or reserves specific resources or combinations of resources to fulfill these constraints. Further, if the Network Management System supports it, Moab dynamically creates network fabric partitions with guaranteed service levels.

QOSCFG[highspeed] MINBANDWIDTH=1GB  MAXLATENCY=10us
...

Usage Limits and Overrides

All credentials, including QoS, allow specification of job usage limits as described in the Basic Fairness Policies overview. In such cases, jobs are constrained by the most limiting of all applicable policies. With QoSes, an override limit may also be specified and with this limit, jobs are constrained by the override, regardless of other limits specified. The following attributes can override the throttling policies from other credentials:

OMAXJOB, OMAXNODE, OMAXPE, OMAXPROC, OMAXPS, OMAXJPROC, OMAXJPS, OMAXJWC, and OMAXJNODE.

(See Usage Limits/Throttling Policies Override Limits.)

Example 7-31:  

# staff QoS should have a limit of 48 jobs, ignoring the user limit 
USERCFG[DEFAULT]   MAXJOB=10
QOSCFG[staff]      OMAXJOB=48

Service Access Thresholds

Jobs can be granted access to services such as preemption and reservation creation, and they can be granted access to resource reservations. However, with QoS thresholds, this access can be made conditional on the current queuetime and XFactor metrics of an idle job. The following table lists the available QoS service thresholds:

Threshold attribute Description
PREEMPTQTTHRESHOLD A job with this QoS becomes a preemptor if the specified queuetime threshold is reached.
PREEMPTXFTHRESHOLD A job with this QoS becomes a preemptor if the specified XFactor threshold is reached.
RSVQTTHRESHOLD A job with this QoS can create a job reservation to guarantee resource access if the specified queuetime threshold is reached.
RSVXFTHRESHOLD A job with this QoS can create a job reservation to guarantee resource access if the specified XFactor threshold is reached.
ACLQTTHRESHOLD A job with this QoS can access reservations with a corresponding QoS ACL only if the specified queuetime threshold is reached.
ACLXFTHRESHOLD A job with this QoS can access reservations with a corresponding QoS ACL only if the specified XFactor threshold is reached.
TRIGGERQTTHRESHOLD If a job with this QoS fails to run before this threshold is reached, any failure triggers associated with this QoS will fire.

Preemption Management

Job preemption facilities can be controlled on a per-QoS basis using the PREEMPTEE and PREEMPTOR flags. Jobs that are preemptible can optionally be constrained to only be preempted in a particular manner by specifying the QoS PREEMPTPOLICY attribute as in the following example:

QOSCFG[special] QFLAGS=PREEMPTEE PREEMPTPOLICY=CHECKPOINT

For preemption to be effective, a job must be marked as a preemptee and must be enabled for the requested preemption type. For example, if the PREEMPTPOLICY is set to suspend, a potential target job must be both a preemptee and marked with the job flag SUSPENDABLE. (See suspension for more information.) If the target job is not suspendable, it will be either requeued or canceled. Likewise, if the PREEMPTPOLICY is set to requeue, the job will be requeued if it is marked restartable. Otherwise, it will be canceled.

The minimum time a job must run before being considered eligible for preemption can also be configured on a per-QoS basis using the PREEMPTMINTIME attribute, which is analogous to the JOBPREEMPTMINACTIVETIME. Conversely, PREEMPTMAXTIME sets a threshold for which a job is no longer eligible for preemption; see JOBPREEMPTMAXACTIVETIME for analogous details.

The PREEMPTEES attribute allows you to specify which QoSes that a job in a specific QoS is allowed to preempt. The PREEMPTEES list is a comma-delimited list of QoS IDs. When a PREEMPTEES attribute is specified, a job using that QoS can only preempt jobs using QoSes listed in the PREEMPTEES list. In turn, those QoSes must be flagged as PREEMPTEE as in the following example:

QOSCFG[a] QFLAGS=PREEMPTOR PREEMPTEES=b,c
QOSCFG[b] QFLAGS=PREEMPTEE
QOSCFG[c] QFLAGS=PREEMPTEE

In the example, jobs in the a QoS can only preempt jobs in the b and c QoSes.

7.3-C Managing QoS Access

Specifying Credential Based QoS Access

You can define the privileges allowed within a QoS by using the QOSCFG parameter; however, in most cases access to the QoS is enabled via credential specific *CFG parameters, specifically the USERCFG, GROUPCFG, ACCOUNTCFG, and CLASSCFG parameters, which allow defining QoS access lists and QoS defaults. Specify credential specific QoS access by using the QLIST and/or QDEF attributes of the associated credential parameter.

QoS Access via Logical OR

To enable QoS access, the QLIST and/or QDEF attributes of the appropriate user, group, account, or class/queue should be specified as in the following example:

# user john's jobs can access QoS geo, chem, or staff with geo as default
USERCFG[john]     QDEF=geo   QLIST=geo,chem,staff
# group system jobs can access the development QoS
GROUPCFG[systems] QDEF=development
# class batch jobs can access the normal QoS
CLASSCFG[batch]   QDEF=normal

By default, jobs may request a QoS if access to that QoS is allowed by any of the job's credentials. (In the previous example, a job from user john submitted to the class batch could request QoSes geo, chem, staff, or normal).

QoS Access via Logical AND

If desired, QoS access can be masked or logically ANDed if the QoS access list is specified with a terminating ampersand (&) as in the following example:

# user john's jobs can access QoS geo, chem, or staff with geo as default
USERCFG[john]     QDEF=geo   QLIST=geo,chem,staff
# group system jobs can access the development QoS
GROUPCFG[systems] QDEF=development
# class batch jobs can access the normal QoS
CLASSCFG[batch]   QDEF=normal
# class debug jobs can only access the development or lowpri QoSes regardless of other credentials
CLASSCFG[debug]   QLIST=development,lowpri&

Specifying QoS Based Access

QoS access may also be specified from within the QoS object using the QoS MEMBERULIST attribute as in the following example:

# define QoS premiere and grant access to users steve and john
QOSCFG[premiere]  PRIORITY=1000  QFLAGS=PREEMPTOR  MEMBERULIST=steve,john

By default, if a job requests a QoS that it cannot access, Moab places a hold on that job. The QOSREJECTPOLICY can be used to modify this behavior.

7.3-D Requesting QoS Services at Job Submission

By default, jobs inherit a default QoS based on the user, group, class, and account associated with the job. If a job has access to multiple QoS levels, the submitter can explicitly request a particular QoS using the QoS resource manager extension as in the following example:

> msub -l nodes=1,walltime=100,qos=special3 job.cmd

7.3-E Restricting Access to Special Attributes

By default, Moab allows all users access to special attributes such as node access policy. By enabling the QoS facility SPECATTRS, the access to these policies can be restricted. For example, to enable the facility, in the moab.cfg file, specify QOSCFG[DEFAULT] SPECATTRS=. Then, to allow access to the special attributes, indicate which special attributes a specific QoS may access.

QOSCFG[DEFAULT] SPECATTRS=
QOSCFG[high]  SPECATTRS=NACCESSPOLICY

Related topics