6.2 Usage Limits/Throttling Policies

A number of Moab policies allow an administrator to control job flow through the system. These throttling policies work as filters allowing or disallowing a job to be considered for scheduling by specifying limits regarding system usage for any given moment. These policies may be specified as global or specific constraints specified on a per user, group, account, QoS, or class basis.

• 6.2.1 Fairness via Throttling Policies
  • 6.2.1.1 Basic Fairness Policies
  • 6.2.1.2 Multi-Dimension Fairness Policies
• 6.2.2 Override Limits
• 6.2.3 Idle Job Limits
• 6.2.4 System Job Limits
• 6.2.5 Hard and Soft Limits
• 6.2.6 Per-partition Limits

6.2.1 Fairness via Throttling Policies

Moab allows significant flexibility with usage limits, or throttling policies. At a high level, Moab allows resource usage limits to be specified in three primary workload categories: (1) active, (2) idle, and (3) system job limits.

6.2.1.1 Basic Fairness Policies

• Active Job Limits - Constrain the total cumulative resources available to active jobs at a given time.
• Idle Job Limits - Constrain the total cumulative resources available to idle jobs at a given time.
• System Job Limits - Constrain the maximum resource requirements of any single job.

These limits can be applied to any job credential (user, group, account, QoS, and class), or on a system-wide basis. Using the keyword DEFAULT, a site may also specify the default setting for the desired user, group, account, QoS, and class. Additionally, QoS's may be configured to allow limit overrides to any particular policy.

To run, a job must meet all policy limits. Limits are applied using the *CFG set of parameters, particularly USERCFG, GROUPCFG, ACCOUNTCFG, QOSCFG, CLASSCFG, and SYSCFG. Limits are specified by associating the desired limit to the individual or default object. The usage limits currently supported are listed in the following table.

MAXJOB
Units:	# of jobs
Description:	Limits the number of jobs a credential may have active (starting or running) at any given time. MAXJOB=0 is not supported. You can, however, achieve similar results by using the HOLD attribute of the USERCFG parameter: USERCFG[john] HOLD=yes
Example:	USERCFG[DEFAULT] MAXJOB=8 GROUPCFG[staff] MAXJOB=2,4
MAXMEM
Units:	total memory in MB
Description:	Limits the total amount of dedicated memory (in MB) that can be allocated by a credential's active jobs at any given time.
Example:	ACCOUNTCFG[jasper] MAXMEM=2048
MAXNODE
Units:	# of nodes
Description:	Limits the total number of compute nodes that can be in use by active jobs at any given time. On some systems (including TORQUE/PBS), nodes have been softly defined rather than strictly defined; that is, a job may request 2 nodes but Torque will translate this request into 1 node with 2 processors. This can prevent Moab from enforcing a MAXNODE policy correctly for a single job. Correct behavior can be achieved using MAXPROC.
Example:	CLASSCFG[batch] MAXNODE=64
MAXPE
Units:	# of processor equivalents
Description:	Limits the total number of dedicated processor-equivalents that can be allocated by active jobs at any given time.
Example:	QOSCFG[base] MAXPE=128
MAXPROC
Units:	# of processors
Description:	Limits the total number of dedicated processors that can be allocated by active jobs at any given time.
Example:	CLASSCFG[debug] MAXPROC=32
MAXPS
Units:	<# of processors> * <walltime>
Description:	Limits the number of outstanding processor-seconds a credential may have allocated at any given time. For example, if a user has a 4-processor job that will complete in 1 hour and a 2-processor job that will complete in 6 hours, they have 4 * 1 * 3600 + 2 * 6 * 3600 = 16 * 3600 outstanding processor-seconds. The outstanding processor-second usage of each credential is updated each scheduling iteration, decreasing as jobs approach their completion time.
Example:	USERCFG[DEFAULT] MAXPS=720000
MAXSUBMITJOBS
Units:	# of jobs
Description:	Limits the number of jobs a credential may submit per iteration.
Example:	USERCFG[DEFAULT] MAXSUBMITJOBS=5
MAXWC
Units:	job duration [[[DD:]HH:]MM:]SS
Description:	Limits the cumulative remaining walltime a credential may have associated with active jobs. It behaves identically to the MAXPS limit (listed earlier) only lacking the processor weighting. Like MAXPS, the cumulative remaining walltime of each credential is also updated each scheduling iteration. MAXWC does not limit the maximum wallclock limit per job. For this capability, use MAX.WCLIMIT.
Example:	USERCFG[ops] MAXWC=72:00:00

The following example demonstrates a simple limit specification:

USERCFG[DEFAULT]  MAXJOB=4
USERCFG[john]     MAXJOB=8

This example allows user john to run up to 8 jobs while all other users may only run up to 4.

Simultaneous limits of different types may be applied per credential and multiple types of credentials may have limits specified. The next example demonstrates this mixing of limits and is a bit more complicated.

USERCFG[steve]    MAXJOB=2 MAXNODE=30
GROUPCFG[staff]   MAXJOB=5
CLASSCFG[DEFAULT] MAXNODE=16
CLASSCFG[batch]   MAXNODE=32

This configuration may potentially apply multiple limits to a single job. As discussed previously, a job may only run if it satisfies all applicable limits. Thus, in this example, the scheduler will be constrained to allow at most 2 simultaneous user steve jobs with an aggregate node consumption of no more than 30 nodes. However, if the job is submitted to a class other than batch, it may be limited further. Here, only 16 total nodes may be used simultaneously by jobs running in any given class with the exception of the class batch. If steve submitted a job to run in the class interactive, for example, and there were jobs already running in this class using a total of 14 nodes, his job would be blocked unless it requested 2 or fewer nodes by the default limit of 16 nodes per class.

6.2.1.2 Multi-Dimension Fairness Policies and Per Credential Overrides

Multi-dimensional fairness policies allow a site to specify policies based on combinations of job credentials. A common example might be setting a maximum number of jobs allowed per queue per user or a total number of processors per group per QoS. As with basic fairness policies, multi-dimension policies are specified using the *CFG parameters. Moab supports the most commonly used multi-dimensional fairness policies (listed in the table below) using the following format:

*CFG[X]  <LIMITTYPE>[<CRED>]=<LIMITVALUE>

The "*CFG" is one of USERCFG, GROUPCFG, ACCOUNTCFG, QOSCFG, or CLASSCFG, the <LIMITTYPE> policy is one of the policies listed in the table in section 6.2.1.1, and <CRED> is of the format <CREDTYPE>[:<VALUE>] with CREDTYPE being one of USER, GROUP, ACCT, QoS, or CLASS. The optional <VALUE> setting can be used to specify that the policy only applies to a specific credential value. For example, the following configuration sets limits on the class fast, controlling the maximum number of jobs any group can have active at any given time and the number of processors in use at any given time for user steve.

CLASSCFG[fast] MAXJOB[GROUP]=12
CLASSCFG[fast] MAXPROC[USER:steve]=50
CLASSCFG[fast] MAXIJOB[USER]=10

The following example configuration may clarify further:

# allow class batch to run up the 3 simultaneous jobs
# allow any user to use up to 8 total nodes within class
CLASSCFG[batch] MAXJOB=3 MAXNODE[USER]=8

# allow users steve and bob to use up to 3 and 4 total processors respectively within class
CLASSCFG[fast] MAXPROC[USER:steve]=3 MAXPROC[USER:bob]=4

Multi-dimensional policies cannot be applied on DEFAULT credentials.

The table below lists the 112 currently implemented, multi-dimensional usage limit permutations. The "slmt" stands for "Soft Limit" and "hlmt" stands for "Hard Limit."

6.2.2 Override Limits

Like all job credentials, the QoS object may be associated with resource usage limits. However, this credential can also be given special override limits that supersede the limits of other credentials, effectively causing all other limits of the same type (MAXJOB) to be ignored. Override limits are applied by preceding the limit specification with the capital letter O. The following configuration provides an example of this in the last line:

USERCFG[steve]    MAXJOB=2   MAXNODE=30
GROUPCFG[staff]   MAXJOB=5
CLASSCFG[DEFAULT] MAXNODE=16
CLASSCFG[batch]   MAXNODE=32
QOSCFG[hiprio]    OMAXJOB=3  OMAXNODE=64

The preceding configuration is identical to the earlier example with the exception of the final QOSCFG line. In this case, the QOSCFG parameter does two things:

• Only 3 hiprio QoS jobs may run simultaneously.
• hiprio QoS jobs may run with up to 64 nodes per credential ignoring other credential MAXNODE limits.

Given the preceding configuration, assume a job is submitted with the credentials, user steve, group staff, class batch, and QoS hiprio.

Such a job will start so long as running it does not lead to any of the following conditions:

• Total nodes used by user steve does not exceed 64.
• Total active jobs associated with user steve does not exceed 2.
• Total active jobs associated with group staff does not exceed 5.
• Total nodes dedicated to class batch does not exceed 64.
• Total active jobs associated with QoS hiprio does not exceed 3.

While the preceding example is a bit complicated for most sites, similar combinations may be required to enforce policies found on many systems.

6.2.3 Idle Job Limits

Idle (or queued) job limits control which jobs are eligible for scheduling. To be eligible for scheduling, a job must meet the following conditions:

• Be idle as far as the resource manager is concerned (no holds).
• Have all job prerequisites satisfied (no outstanding job or data dependencies).
• Meet all idle job throttling policies.

If a job fails to meet any of these conditions, it will not be considered for scheduling and will not accrue service based job prioritization. (See service component and JOBPRIOACCRUALPOLICY.) The primary purpose of idle job limits is to ensure fairness among competing users by preventing queue stuffing and other similar abuses. Queue stuffing occurs when a single entity submits large numbers of jobs, perhaps thousands, all at once so they begin accruing queuetime based priority and remain first to run despite subsequent submissions by other users.

Idle limits are specified in a manner almost identical to active job limits with the insertion of the capital letter I into the middle of the limit name. For example, to limit the number of (eligible) idle jobs a given user could have at once, the following parameter could be used:

USERCFG[DEFAULT]  MAXIJOB=20

As just shown, idle limits can constrain the total number of jobs considered to be eligible on a per credential basis. Further, like active job limits, idle job limits can also constrain eligible jobs based on aggregate requested resources. This could, for example, allow a site to indicate that for a given user, only jobs requesting up to a total of 64 processors, or 3200 processor-seconds would be considered at any given time. Which jobs to select is accomplished by prioritizing all idle jobs and then adding jobs to the eligible list one at a time in priority order until jobs can no longer be added. This eligible job selection is done only once per scheduling iteration, so, consequently, idle job limits only support a single hard limit specification. Any specified soft limit is ignored.

All single dimensional job limit types supported as active job limits are also supported as idle job limits. In addition, Moab also supports MAXIJOB[USER] and MAXIPROC[USER] policies on a per class basis. (See Basic Fairness Policies.)

Example:

USERCFG[steve]     MAXIJOB=2 MAXINODE=30
GROUPCFG[staff]   MAXIJOB=5
CLASSCFG[DEFAULT] MAXINODE=16
CLASSCFG[batch]   MAXINODE=32 MAXIJOB[USER]=2 MAXIJOB[USER:john]=6
QOSCFG[hiprio]      MAXIJOB=3 MAXINODE=64

6.2.4 System Job Limits

System job limits, also known as individual job limits, constrain the quantity of resources that may be requested by any single job. Unlike active and idle job usage limits, system job limits are specified on a global basis. The following table details the associated parameters.

6.2.5 Hard and Soft Limits

Hard and soft limit specification allows a site to balance both fairness and utilization on a given system. Typically, throttling limits are used to constrain the quantity of resources a given credential (such as user or group) is allowed to consume. These limits can be very effective in enforcing fair usage among a group of users. However, in a lightly loaded system, or one in which there are significant swings in usage from project to project, these limits can reduce system utilization by blocking jobs even when no competing jobs are queued.

Soft limits help address this problem by providing additional scheduling flexibility. They allow sites to specify two tiers of limits; the more constraining limits soft limits are in effect in heavily loaded situations and reflect tight fairness constraints. The more flexible hard limits specify how flexible the scheduler can be in selecting jobs when there are idle resources available after all jobs meeting the tighter soft limits have started. Soft and hard limits are specified in the format [<SOFTLIMIT>,]<HARDLIMIT>. For example, a given site may want to use the following configuration:

USERCFG[DEFAULT]  MAXJOB=2,8

With this configuration, the scheduler would select all jobs that meet the per user MAXJOB limit of 2. It would then attempt to start and reserve resources for all of these selected jobs. If after doing so there still remain available resources, the scheduler would then select all jobs that meet the less constraining hard per user MAXJOB limit of 8 jobs. These jobs would then be scheduled and reserved as available resources allow.

If no soft limit is specified or the soft limit is less constraining than the hard limit, the soft limit is set equal to the hard limit.

Example:

USERCFG[steve]    MAXJOB=2,4 MAXNODE=15,30
GROUPCFG[staff]   MAXJOB=2,5
CLASSCFG[DEFAULT] MAXNODE=16,32
CLASSCFG[batch]   MAXNODE=12,32
QOSCFG[hiprio]    MAXJOB=3,5 MAXNODE=32,64

Job preemption status can be adjusted based on whether the job violates a soft policy using the ENABLESPVIOLATIONPREEMPTION parameter.

6.2.6 Per-partition Limits

Per-partition scheduling can set limits and enforce credentials and polices on a per-partition basis. Configuration for per-partition scheduling is done on the grid head. In a grid, each Moab cluster is considered a partition. Per-partition scheduling is typically used in a Master/Slave grid.

To enable per-partition scheduling, add the following to moab.cfg:

PERPARTITIONSCHEDULING TRUE
JOBMIGRATEPOLICY JUSTINTIME

6.2.6.1 Per-partition Limits

You can configure per-job limits and credential usage limits on a per-partition basis in the moab.cfg file. Here is a sample configuration for partitions "g02" and "g03" in moab.cfg.

PARCFG[g02]   CONFIGFILE=/opt/moab/parg02.cfg 
PARCFG[g03]   CONFIGFILE=/opt/moab/parg03.cfg

You can then add per-partition limits in each partition configuration file:

CLASSCFG[pbatch]   MAXJOB=5 

CLASSCFG[pbatch]   MAXJOB=10 

You can configure Moab so that jobs submitted to any partition besides g02 and g03 get the default limits in moab.cfg:

CLASSCFG[DEFAULT]  MAXJOB=2

6.2.6.2 Supported Credentials and Limits

The user, group, account, QoS, and class credentials are supported in per-partition scheduling.

The following per-job limits are supported:

• MAX.NODE
• MAX.WCLIMIT
• MAX.PROC

The following credential usage limits are supported:

• MAXJOB
• MAXNODE
• MAXPROC
• MAXWC
• MAXSUBMITJOBS

Multi-dimensional limits are supported for the listed credentials and per-job limits. For example:

CLASSCFG[pbatch]   MAXJOB[user:frank]=10

See Also

• Managing Job Resource Limit Violations
• RESOURCELIMITPOLICY parameter
• FSTREE parameter (set usage limits within share tree hierarchy)
• Credential Overview