17.9 Grid Usage Policies
- 17.9.1 Grid Usage Policy Overview
- 17.9.2 Peer Job Resource Limits
- 17.9.3 Usage Limits via Peer Credentials
- 17.9.4 Using General Policies in a Grid Environment
17.9.1 Grid Usage Policy Overview
Moab allows extensive control over how peers interact. These controls allow the following:
- Limiting which remote users, group, and accounts can utilize local compute resources
- Limiting the total quantity of local resources made available to remote jobs at any given time
- Limiting remote resource access to a specific subset of resources
- Limiting timeframes during which local resources will be made available to remote jobs
- Limiting the types of remote jobs which will be allowed to execute
17.9.2 Peer Job Resource Limits
Both source and destination peers can limit the types of jobs they will allow in terms of resources requested, services provided, job duration, applications used, etc using Moab's job template feature. Using this method, one or more job profiles can be created on either the source or destination side, and Moab can be configured to allow or reject jobs based on whether or not the jobs meet the specified job profiles.
When using the ALLOWJOBLIST and REJECTJOBLIST attributes, the following rules apply:
- All jobs that meet the job templates listed by ALLOWJOBLIST are allowed.
- All jobs that do not meet ALLOWJOBLIST job templates and which do meet REJECTJOBLIST job templates are rejected.
- All jobs that meet no job templates in either list are allowed.
17.9.3 Usage Limits via Peer Credentials
With peer interfaces, destination clusters willing to accept remote jobs can map these jobs onto a select subset of users, accounts, QoS's, and queues. With the ability to lock these jobs into certain credentials comes the ability to apply any arbitrary credential constraints, priority adjustments, and resource limitations normally available within cluster management. Specifically, the following can be accomplished:
- limit number of active jobs simultaneously allowed
- limit quantity of allocated compute resources simultaneously allowed
- adjust job priority
- control access to specific scheduling features (deadlines, reservations, preemption, etc)
- adjust fairshare targets
- limit resource access
17.9.4 Using General Policies in a Grid Environment
While Moab does provide a number of unique grid-based policies for use in a grid environment, the vast majority of available management tools come from the transparent application of cluster policies. Cluster-level policies such as job prioritization, node allocation, fairshare, usage limits, reservations, preemption, and allocation management all just work and can be applied in a grid in exactly the same manner.
The one key concept to understand that is in a centralized based grid, these policies apply across the entire grid, in a peer-based grid, these policies apply only to local workload and resources.
17.9.4.1 Source Cluster Policies
In many cases, organizations are interested in treating jobs differently based on their point of origin. This can be accomplished by assigning and/or keying off of a unique credential associated with the remote workload. For example, a site may wish to constrain jobs from a remote cluster to only a portion of the total available cluster cycles. This could be accomplished using usage limits, fairshare targets, fairshare caps, reservations, or allocation management based policies.
The examples below show three different approaches for constraining remote resource access.
Example 1: Constraining Remote Resource Access via Fairshare Caps
# define peer relationship and map all incoming jobs to orion account RMCFG[orion.INBOUND] SET.JOB=orion.set JOBCFG[orion.set] ACCOUNT=orion # configure basic fairshare for 7 one day intervals FSPOLICY DEDICATEDPS FSINTERVAL 24:00:00 FSDEPTH 7 FSUSERWEIGHT 100 # use fairshare cap to limit jobs from orion to 10% of cycles ACCOUNTCFG[orion] FSCAP=10%
Example 2: Constraining Remote Resource Access via Fairshare Targets and Preemption
# define peer relationship and map all incoming jobs to orion account RMCFG[orion.INBOUND] SET.JOB=orion.set JOBCFG[orion.set] ACCOUNT=orion # local cluster can preempt jobs from orion USERCFG[DEFAULT] JOBFLAGS=PREEMPTOR PREEMPTPOLICY CANCEL # configure basic fairshare for 7 one day intervals FSPOLICY DEDICATEDPS FSINTERVAL 24:00:00 FSDEPTH 7 FSUSERWEIGHT 100 # decrease priority of remote jobs and force jobs exceeding 10% usage to be preemptible ACCOUNTCFG[orion] FSTARGET=10- ENABLEFSVIOLATIONPREEMPTION TRUE
Example 3: Constraining Remote Resource Access via Priority and Usage Limits
# define peer relationship and map all incoming jobs to orion account RMCFG[orion.INBOUND] SET.JOB=orion.set JOBCFG[orion.set] QOS=orion USERCFG[DEFAULT] QDEF=orion # local cluster can preempt jobs from orion USERCFG[DEFAULT] JOBFLAGS=PREEMPTOR PREEMPTPOLICY CANCEL # adjust remote jobs to have reduced priority QOSCFG[orion] PRIORITY=-1000 # allow remote jobs to use up to 64 procs without being preemptible and up to 96 as preemptees QOSCFG[orion] MAXPROC=64,96 ENABLESPVIOLATIONPREEMPTION TRUE
See Also
- Grid Sandbox - control grid resource access