5.1.4 Common Priority Usage
- Credential Priority Factors
- Service Level Priority Factors
- Priority Factor Caps
- User Selectable Prioritization
Site administrators vary widely in their preferred manner of prioritizing jobs. Moab's scheduling hierarchy allows sites to meet job control needs without requiring adjustments to dozens of parameters. Some choose to use numerous subcomponents, others a few, and still others are content with the default FIFO behavior. Any subcomponent that is not of interest may be safely ignored.
5.1.4-A Credential Priority Factors
To help clarify the use of priority weights, a brief example may help. Suppose a site wished to maintain the FIFO behavior but also incorporate some credential based prioritization to favor a special user. Particularly, the site would like the user john to receive a higher initial priority than all other users. Configuring this behavior requires two steps. First, the user credential subcomponent must be enabled and second, john must have his relative priority specified. Take a look at the sample moab.cfg file:
USERWEIGHT 1 USERCFG[john] PRIORITY=300
The "USER" priority subcomponent was enabled by setting the USERWEIGHT parameter. In fact, the parameters used to specify the weights of all components and subcomponents follow this same "*WEIGHT" naming convention (as in RESWEIGHT and TARGETQUEUETIMEWEIGHT.
The second part of the example involves specifying the actual user priority for the user john. This is accomplished using the USERCFG parameter. Why was the priority 300 selected and not some other value? Is this value arbitrary? As in any priority system, actual priority values are meaningless, only relative values are important. In this case, we are required to balance user priorities with the default queue time based priorities. Since queuetime priority is measured in minutes queued, the user priority of 300 places a job by user john on par with a job submitted 5 minutes earlier by another user.
Is this what the site wants? Maybe, maybe not. At the onset, most sites are uncertain what they want in prioritization. Often, an estimate initiates prioritization and adjustments occur over time. Cluster resources evolve, the workload evolves, and even site policies evolve, resulting in changing priority needs over time. Anecdotal evidence indicates that most sites establish a relatively stable priority policy within a few iterations and make only occasional adjustments to priority weights from that point.
5.1.4-B Service Level Priority Factors
In another example, suppose a site administrator wants to do the following:
- favor jobs in the low, medium, and high QoSes so they will run in QoS order
- balance job expansion factor
- use job queue time to prevent jobs from starving
Under such conditions, the sample moab.cfg file might appear as follows:
QOSWEIGHT 1 XFACTORWEIGHT 1 QUEUETIMEWEIGHT 10 TARGETQUEUETIMEWEIGHT 1 QOSCFG[low] PRIORITY=1000 QOSCFG[medium] PRIORITY=10000 QOSCFG[high] PRIORITY=100000 QOSCFG[DEFAULT] QTTARGET=4:00:00
This example is a bit more complicated but is more typical of the needs of many sites. The desired QoS weightings are established by enabling the QoS subfactor using the QOSWEIGHT parameter while the various QoS priorities are specified using QOSCFG. XFACTORWEIGHT is then set as this subcomponent tends to establish a balanced distribution of expansion factors across all jobs. Next, the queuetime component is used to gradually raise the priority of all jobs based on the length of time they have been queued. Note that in this case, QUEUETIMEWEIGHT was explicitly set to 10, overriding its default value of 1. Finally, the TARGETQUEUETIMEWEIGHT parameter is used in conjunction with the USERCFG line to specify a queue time target of 4 hours.
5.1.4-C Priority Factor Caps
Assume now that the site administrator is content with this priority mix but has a problem with users submitting large numbers of very short jobs. Very short jobs would tend to have rapidly growing XFactor values and would consequently quickly jump to the head of the queue. In this case, a factor cap would be appropriate. Such caps allow a site to limit the contribution of a job's priority factor to be within a defined range. This prevents certain priority factors from swamping others. Caps can be applied to either priority components or subcomponents and are specified using the <COMPONENTNAME>CAP parameter (such as QUEUETIMECAP, RESCAP, and SERVCAP). Note that both component and subcomponent caps apply to the pre-weighted value, as in the following equation:
Priority =
C1WEIGHT * MIN(C1CAP,SUM(
S11WEIGHT * MIN(S11CAP,S11S) +
S12WEIGHT * MIN(S12CAP,S12S) +
...)) +
C2WEIGHT * MIN(C2CAP,SUM(
S21WEIGHT * MIN(S21CAP,S21S) +
S22WEIGHT * MIN(S22CAP,S22S) +
...)) +
...
Example 5-2: Priority cap
QOSWEIGHT 1 QOSCAP 10000 XFACTORWEIGHT 1 XFACTORCAP 1000 QUEUETIMEWEIGHT 10 QUEUETIMECAP 1000
5.1.4-D User Selectable Prioritization
Moab allows users to specify a job priority to jobs they own or manage. This priority may be set at job submission time or it may be dynamically modified (using setspri or mjobctl) after submitting the job. For fairness reasons, users may only apply a negative priority to their job and thus slide it further back in the queue. This enables users to allow their more important jobs to run before their less important ones without gaining unfair advantage over other users.
User priorities can be positive if ENABLEPOSUSERPRIORITY TRUE is specified in moab.cfg.
In order to set ENABLEPOSUSERPRIORITY, you must change the USERPRIOWEIGHT from its default value of 0. For example:
USERPRIOWEIGHT 100
> setspri -r 100 332411 successfully modified job priority
Specifying a user priority at job submission time is resource manager specific. See the associated resource manager documentation for more information.
User Selectable Priority w/QoS
Using the QoS facility, organizations can set up an environment in which users can more freely select the desired priority of a given job. Organizations may enable access to a number of QoSes each with its own charging rate, priority, and target service levels. Users can then assign job importance by selecting the appropriate QoS. If desired, this can allow a user to jump ahead of other users in the queue if they are willing to pay the associated costs.
Related topics