Conventions

6.2 Usage Limits/Throttling Policies

A number of Moab policies allow an administrator to control job flow through the system. These throttling policies work as filters allowing or disallowing a job to be considered for scheduling by specifying limits regarding system usage for any given moment. These policies may be specified as global or specific constraints specified on a per user, group, account, QoS, or class basis.

• Fairness via Throttling Policies
  • Basic Fairness Policies
  • Multi-Dimension Fairness Policies
• Override Limits
• Idle Job Limits
• Hard and Soft Limits
• Per-partition Limits
• Usage-based Limits
  • Configuring Actions
  • Specifying Hard and Soft Policy Violations
  • Constraining Walltime Usage

6.2-A Fairness via Throttling Policies

Moab allows sign ificant flexibility with usage limits, or throttling policies. At a high level, Moab allows resource usage limits to be specified in three primary workload categories: (1) active, (2) idle, and (3) system job limits.

Basic Fairness Policies

These limits can be applied to any job credential (user, group, account, QoS, and class), or on a system-wide basis. Using the keyword DEFAULT, a site may also specify the default setting for the desired user, group, account, QoS, and class. Additionally, you may configure QoS to allow limit overrides to any particular policy.

To run, a job must meet all policy limits. Limits are applied using the *CFG set of parameters, particularly USERCFG, GROUPCFG, ACCOUNTCFG, QOSCFG, CLASSCFG, and SYSCFG. Limits are specified by associating the desired limit to the individual or default object. The usage limits currently supported are listed in the following table.

USERCFG[gertrude] MAXARRAYJOB=10

USERCFG[joe] MAXGRES[matlab]=2 
USERCFG[jim] MAXGRES[matlab]=2,4

MAXJOB
Units	# of jobs
Description	Limits the number of jobs a credential may have active (starting or running) at any given time. Moab places a hold on all new jobs submitted by that credential once it has reached its maximum number of allowable jobs. MAXJOB=0 is not supported. You can, however, achieve similar results by using the HOLD attribute of the USERCFG parameter: USERCFG[john] HOLD=yes
Example	USERCFG[DEFAULT] MAXJOB=8 GROUPCFG[staff] MAXJOB=2,4

ACCOUNTCFG[jasper] MAXMEM=2048

MAXNODE
Units	# of nodes
Description	Limits the total number of compute nodes that can be in use by active jobs at any given time. On some systems (including TORQUE/PBS), nodes have been softly defined rather than strictly defined; that is, a job may request 2 nodes but TORQUE will translate this request into 1 node with 2 processors. This can prevent Moab from enforcing a MAXNODE policy correctly for a single job. Correct behavior can be achieved using MAXPROC.
Example	CLASSCFG[batch] MAXNODE=64

QOSCFG[base] MAXPE=128

CLASSCFG[debug] MAXPROC=32

USERCFG[DEFAULT] MAXPS=720000

USERCFG[DEFAULT] MAXSUBMITJOBS=5							

MAXWC
Units	job duration [[[DD:]HH:]MM:]SS
Description	Limits the cumulative remaining walltime a credential may have associated with active jobs. It behaves identically to the MAXPS limit (listed earlier) only lacking the processor weighting. Like MAXPS, the cumulative remaining walltime of each credential is also updated each scheduling iteration. MAXWC does not limit the maximum wallclock limit per job. For this capability, use MAX.WCLIMIT.
Example	USERCFG[ops] MAXWC=72:00:00

The following example demonstrates a simple limit specification:

USERCFG[DEFAULT]  MAXJOB=4
USERCFG[john]     MAXJOB=8

Simultaneous limits of different types may be applied per credential and multiple types of credentials may have limits specified. The next example demonstrates this mixing of limits and is a bit more complicated.

USERCFG[steve]    MAXJOB=2 MAXNODE=30
GROUPCFG[staff]   MAXJOB=5
CLASSCFG[DEFAULT] MAXNODE=16
CLASSCFG[batch]   MAXNODE=32

This configuration may potentially apply multiple limits to a single job. As discussed previously, a job may only run if it satisfies all applicable limits. Thus, in this example, the scheduler will be constrained to allow at most 2 simultaneous user steve jobs with an aggregate node consumption of no more than 30 nodes. However, if the job is submitted to a class other than batch, it may be limited further. Here, only 16 total nodes may be used simultaneously by jobs running in any given class with the exception of the class batch. If steve submitted a job to run in the class interactive, for example, and there were jobs already running in this class using a total of 14 nodes, his job would be blocked unless it requested 2 or fewer nodes by the default limit of 16 nodes per class.

Multi-Dimension Fairness Policies and Per Credential Overrides

Multi-dimensional fairness policies allow a site to specify policies based on combinations of job credentials. A common example might be setting a maximum number of jobs allowed per queue per user or a total number of processors per group per QoS. As with basic fairness policies, multi-dimension policies are specified using the *CFG parameters or through the identity manager interface. Moab supports the most commonly used multi-dimensional fairness policies (listed in the table below) using the following format:

*CFG[X] <LIMITTYPE>[<CRED>]=<LIMITVALUE>

*CFG is one of USERCFG, GROUPCFG, ACCOUNTCFG, QOSCFG, or CLASSCFG, the <LIMITTYPE> policy is one of the policies listed in the table in section 6.2.1.1, and <CRED> is of the format <CREDTYPE>[:<VALUE>] with CREDTYPE being one of USER, GROUP, ACCT, QoS, or CLASS. The optional <VALUE> setting can be used to specify that the policy only applies to a specific credential value. For example, the following configuration sets limits on the class fast, controlling the maximum number of jobs any group can have active at any given time and the number of processors in use at any given time for user steve.

CLASSCFG[fast] MAXJOB[GROUP]=12
CLASSCFG[fast] MAXPROC[USER:steve]=50
CLASSCFG[fast] MAXIJOB[USER]=10

The following example configuration may clarify further:

# allow class batch to run up the 3 simultaneous jobs
# allow any user to use up to 8 total nodes within class
CLASSCFG[batch] MAXJOB=3 MAXNODE[USER]=8
# allow users steve and bob to use up to 3 and 4 total processors respectively within class
CLASSCFG[fast] MAXPROC[USER:steve]=3 MAXPROC[USER:bob]=4

Multi-dimensional policies cannot be applied on DEFAULT credentials.

The table below lists the currently implemented, multi-dimensional usage limit permutations. The "slmt" stands for "Soft Limit" and "hlmt" stands for "Hard Limit."

6.2-B Override Limits

Like all job credentials, the QoS object may be associated with resource usage limits. However, this credential can also be given special override limits that supersede the limits of other credentials, effectively causing all other limits of the same type to be ignored. See QoS Usage Limits and Overrides for a complete list of policies that can be overridden. The following configuration provides an example of this in the last line:

USERCFG[steve]    MAXJOB=2   MAXNODE=30
GROUPCFG[staff]   MAXJOB=5
CLASSCFG[DEFAULT] MAXNODE=16
CLASSCFG[batch]   MAXNODE=32
QOSCFG[hiprio]    OMAXJOB=3  OMAXNODE=64

Given the preceding configuration, assume a job is submitted with the credentials, user steve, group staff, class batch, and QoS hiprio.

Such a job will start so long as running it does not lead to any of the following conditions:

• Total nodes used by user steve does not exceed 64.
• Total active jobs associated with user steve does not exceed 2.
• Total active jobs associated with group staff does not exceed 5.
• Total nodes dedicated to class batch does not exceed 64.
• Total active jobs associated with QoS hiprio does not exceed 3.

While the preceding example is a bit complicated for most sites, similar combinations may be required to enforce policies found on many systems.

6.2-C Idle Job Limits

Idle (or queued) job limits control which jobs are eligible for scheduling. To be eligible for scheduling, a job must meet the following conditions:

• Be idle as far as the resource manager is concerned (no holds).
• Have all job prerequisites satisfied (no outstanding job or data dependencies).
• Meet all idle job throttling policies.

If a job fails to meet any of these conditions, it will not be considered for scheduling and will not accrue service based job prioritization. (See service component and JOBPRIOACCRUALPOLICY.) The primary purpose of idle job limits is to ensure fairness among competing users by preventing queue stuffing and other similar abuses. Queue stuffing occurs when a single entity submits large numbers of jobs, perhaps thousands, all at once so they begin accruing queue time based priority and remain first to run despite subsequent submissions by other users.

Idle limits are specified in a manner almost identical to active job limits with the insertion of the capital letter I into the middle of the limit name. Below are examples of the MAXIJOB and MAXINODE limits, which are idle limit equivalents to the MAXJOB and MAXNODE limits:

USERCFG[DEFAULT]  MAXIJOB=8

GROUPCFG[staff]   MAXIJOB=2,4

USERCFG[DEFAULT]  MAXINODE=2

Idle limits can constrain the total number of jobs considered to be eligible on a per credential basis. Further, like active job limits, idle job limits can also constrain eligible jobs based on aggregate requested resources. This could, for example, allow a site to indicate that for a given user, only jobs requesting up to a total of 64 processors, or 3200 processor-seconds would be considered at any given time. Which jobs to select is accomplished by prioritizing all idle jobs and then adding jobs to the eligible list one at a time in priority order until jobs can no longer be added. This eligible job selection is done only once per scheduling iteration, so, consequently, idle job limits only support a single hard limit specification. Any specified soft limit is ignored.

All single dimensional job limit types supported as active job limits are also supported as idle job limits. In addition, Moab also supports MAXIJOB[USER] and MAXIPROC[USER] policies on a per class basis. (See Basic Fairness Policies.)

Example:

USERCFG[steve]    MAXIJOB=2
GROUPCFG[staff]   MAXIJOB=5
CLASSCFG[batch]   MAXIJOB[USER]=2 MAXIJOB[USER:john]=6
QOSCFG[hiprio]    MAXIJOB=3

6.2-D Hard and Soft Limits

Hard and soft limit specification allows a site to balance both fairness and utilization on a given system. Typically, throttling limits are used to constrain the quantity of resources a given credential (such as user or group) is allowed to consume. These limits can be very effective in enforcing fair usage among a group of users. However, in a lightly loaded system, or one in which there are significant swings in usage from project to project, these limits can reduce system utilization by blocking jobs even when no competing jobs are queued.

Soft limits help address this problem by providing additional scheduling flexibility. They allow sites to specify two tiers of limits; the more constraining limits soft limits are in effect in heavily loaded situations and reflect tight fairness constraints. The more flexible hard limits specify how flexible the scheduler can be in selecting jobs when there are idle resources available after all jobs meeting the tighter soft limits have started. Soft and hard limits are specified in the format [<SOFTLIMIT>,]<HARDLIMIT>. For example, a given site may want to use the following configuration:

USERCFG[DEFAULT]  MAXJOB=2,8

Example:

USERCFG[steve]    MAXJOB=2,4 MAXNODE=15,30
GROUPCFG[staff]   MAXJOB=2,5
CLASSCFG[DEFAULT] MAXNODE=16,32
CLASSCFG[batch]   MAXNODE=12,32
QOSCFG[hiprio]    MAXJOB=3,5 MAXNODE=32,64

Job preemption status can be adjusted based on whether the job violates a soft policy using the ENABLESPVIOLATIONPREEMPTION parameter.

6.2-E Per-partition Limits

Per-partition scheduling can set limits and enforce credentials and polices on a per-partition basis.

To enable per-partition scheduling, add the following to moab.cfg:

PERPARTITIONSCHEDULING TRUE
JOBMIGRATEPOLICY JUSTINTIME

With per-partition scheduling, it is recommended that limits go on the specific partitions and not on the global level. If limits are specified on both levels, Moab will take the more constricting of the limits. Also, please note that a DEFAULT policy on the global partition is not overridden by any policy on a specific partition.

Per-partition Limits

You can configure per-job limits and credential usage limits on a per-partition basis in the moab.cfg file. Here is a sample configuration for partitions g02 and g03 in moab.cfg.

PARCFG[g02]   CONFIGFILE=/opt/moab/parg02.cfg
PARCFG[g03]   CONFIGFILE=/opt/moab/parg03.cfg

You can then add per-partition limits in each partition configuration file:

# /opt/moab/parg02.cfg
CLASSCFG[pbatch]   MAXJOB=5 

# /opt/moab/parg03.cfg
CLASSCFG[pbatch]   MAXJOB=10 

You can configure Moab so that jobs submitted to any partition besides g02and g03 get the default limits in moab.cfg:

stl

CLASSCFG[pbatch]  MAXJOB=2

Supported Credentials and Limits

The user, group, account, QoS, and class credentials are supported in per-partition scheduling.

The following per-job limits are supported:

• MAX.NODE
• MAX.WCLIMIT
• MAX.PROC

The following credential usage limits are supported:

• MAXJOB
• MAXNODE
• MAXPROC
• MAXWC
• MAXSUBMITJOBS

Multi-dimensional limits are supported for the listed credentials and per-job limits. For example:

CLASSCFG[pbatch]   MAXJOB[user:frank]=10

6.2-F Usage-based Limits

Resource usage limits constrain the amount of resources a given job may consume. These limits are generally proportional to the resources requested and may include walltime, any standard resource, or any specified generic resource. The parameter RESOURCELIMITPOLICY controls which resources are limited, what limit policy is enforced per resource, and what actions the scheduler should take in the event of a policy violation.

Configuring Actions

The RESOURCELIMITPOLICY parameter accepts a number of policies, resources, and actions using the format and values defined below.

If walltime is the resource to be limited, be sure that the resource manager is configured to not interfere if a job surpasses its given walltime. For TORQUE, this is done by using $ignwalltime in the configuration on each MOM node.

Format

RESOURCELIMITPOLICY<RESOURCE>:[<SPOLICY>,]<HPOLICY>:[<SACTION>,]<HACTION>[:[<SVIOLATIONTIME>,]<HVIOLATIONTIME>]...

Resource	Description
CPUTIME	Maximum total job proc-seconds used by any single job (allows scheduler enforcement of cpulimit).
DISK	Local disk space (in MB) used by any single job task.
JOBMEM	Maximum real memory/RAM (in MB) used by any single job. JOBMEM will only work with the MAXMEM flag.
JOBPROC	Maximum processor load associated with any single job. You must set MAXPROC to use JOBPROC.
MEM	Maximum real memory/RAM (in MB) used by any single job task.
MINJOBPROC	Minimum processor load associated with any single job (action taken if job is using 5% or less of potential CPU usage).
NETWORK	Maximum network load associated with any single job task.
PROC	Maximum processor load associated with any single job task.
SWAP	Maximum virtual memory/SWAP (in MB) used by any single job task.
WALLTIME	Requested job walltime.

Example 6-1: Notify and then cancel job if requested memory is exceeded

# if job exceeds memory usage, immediately notify owner
# if job exceeds memory usage for more than 5 minutes, cancel the job
RESOURCELIMITPOLICY MEM:ALWAYS,EXTENDEDVIOLATION:NOTIFY,CANCEL:00:05:00

Example 6-2: Checkpoint job on walltime violations

# if job exceeds requested walltime, checkpoint job
RESOURCELIMITPOLICY WALLTIME:ALWAYS:CHECKPOINT
# when checkpointing, send term signal, followed by kill 1 minute later
RMCFG[base] TYPE=PBS CHECKPOINTTIMEOUT=00:01:00 CHECKPOINTSIG=SIGTERM

Example 6-3: Cancel jobs that use 5% or less of potential CPU usage for more than 5 minutes

RESOURCELIMITPOLICY MINJOBPROC:EXTENDEDVIOLATION:CANCEL:5:00

Example 6-4: Migrating a job when it blocks other workload

RESOURCELIMITPOLICY JOBPROC:BLOCKEDWORKLOADONLY:MIGRATE

Specifying Hard and Soft Policy Violations

Moab is able to perform different actions for both hard and soft policy violations. In most resource management systems, a mechanism does not exist to allow the user to specify both hard and soft limits. To address this, Moab provides the RESOURCELIMITMULTIPLIER parameter that allows per partition and per resource multiplier factors to be specified to generate the actual hard and soft limits to be used. If the factor is less than one, the soft limit will be lower than the specified value and a Moab action will be taken before the specified limit is reached. If the factor is greater than one, the hard limit will be set higher than the specified limit allowing a buffer space before the hard limit action is taken.

In the following example, job owners will be notified by email when their memory reaches 100% of the target, and the job will be canceled if it reaches 125% of the target. For wallclock usage, the job will be requeued when it reaches 90% of the specified limit if another job is waiting for its resources, and it will be checkpointed when it reaches the full limit.

RESOURCELIMITPOLICY       MEM:ALWAYS,ALWAYS:NOTIFY,CANCEL
RESOURCELIMITPOLICY       WALLTIME:BLOCKEDWORKLOADONLY,ALWAYS:REQUEUE,CHECKPOINT
RESOURCELIMITMULTIPLIER   MEM:1.25,WALLTIME:0.9

Constraining Walltime Usage

While Moab constrains walltime using the parameter RESOURCELIMITPOLICY like other resources, it also allows walltime exception policies which are not available with other resources. In particular, Moab allows jobs to exceed the requested wallclock limit by an amount specified on a global basis using the JOBMAXOVERRUN parameter or on a per credential basis using the OVERRUN attribute of the CLASSCFG parameter.

JOBMAXOVERRUN    00:10:00
CLASSCFG[debug]  overrun=00:00:30  

Related topics 

• RESOURCELIMITPOLICY parameter
• FSTREE parameter (set usage limits within share tree hierarchy)
• Credential Overview
• JOBMAXOVERRUN parameter
• WCVIOLATIONACTION parameter
• RESOURCELIMITMULTIPLIER parameter