11.10 Job Templates
- 11.10.1 Job Template Overview
- 11.10.2 Applying Job Templates
- 11.10.3 Job Template Extension Attributes
- 11.10.4 Resource Manager Based Templates
- 11.10.5 Job Template Examples
- 11.10.6 Managing Job Templates
11.10.1 Job Template Overview
Job templates are used for two primary purposes: (1) to provide a means of generically matching and categorizing jobs, and (2) to provide a means of setting arbitrary default or forced attributes for certain jobs. Job templates can be used in many aspects of scheduling but they are most commonly applied in the area of Peer Based Grid usage policy. Job templates are defined using the JOBCFG configuration parameter.
11.10.2 Applying Job Templates
11.10.2.1 Matching
The JOBMATCHCFG parameter allows relationships to be established between a number of job templates. JMAX and JMIN function as filters to determine whether a job is eligible for a subsequent template to be applied to the job. If a job is eligible, JDEF and JSET templates apply attributes to the job. The table in section 11.10.3 Job Template Extension Attributes indicates which job template types are compatible with which job template extension attributes. The following types of templates can be specified with the JOBMATCHCFG parameter:
 |
For JMAX, a job template can specify only positive non-zero numbers as maximum limits for generic resources. If a job requests a generic resource that is not limited by the template, then the template can still be used. |
# limit all users to a total of two non-interactive jobs USERCFG[DEFAULT] MAXJOB=2 # reservation configuration SRCFG[test] DESCRIPTION="compute pool for interactive and short duration jobs" SRCFG[test] JOBATTRLIST=INTERACTIVE SRCFG[test] MAXTIME=1:00:00 SRCFG[test] HOSTLIST=R:atl[16-63] # job template configuration JOBCFG[inter.min] FLAGS=interactive JOBCFG[inter.set] FLAGS=ignpolicies JOBMATCHCFG[interactive] JMIN=inter.min JSET=inter.set
In the preceding example, a reservation called test is created. This reservation is only accessible by interactive jobs and jobs that will take less than one hour to complete. Specifying MAXJOB=2 means that each user on the system is only allowed to run two jobs simultaneously.
The job template configuration is signified by the JOBCFG parameter. The inter.min template (JMIN) determines that if a job is marked as interactive it is eligible to have inter.set template applied to it. The JOBMATCHCFG parameter establishes the relationship between inter.min and inter.set. The inter.set template forces the ignpolicies flag on the job, which allows the job to ignore the MAXJOB policy. Also, the job automatically requests the reservation named test. Any user submitting a job may have as many interactive jobs running as resources permit, and the job runs inside the test reservation, which constrains the jobs to run on nodes at1[16-63]. With this template, users do not need to know to which nodes to send interactive jobs; Moab automatically sends interactive jobs to the nodes specified in the reservation.
11.10.2.2 Selecting
Rather than matching a job automatically based on job attributes, a specific template can be directly requested by setting the SELECT attribute to TRUE. Attributes specified in the job template are applied to the job.
JOBCFG[inter.set] FLAGS=ignpolicies SELECT=true
In the preceding example, instead of the interactive jobs automatically going to the reservation named test (as in the Matching example), users can request the inter.set template for the ignpolicies flag and the requested reservation test to be applied to the job.
Use the following to request the job template:
msub -l template=inter.set myjob.shSelecting a specific reservation is often used in creating workflows. See 11.10.7 Creating Workflows with Job Templates for more information.
11.10.3 Job Template Extension Attributes
When creating a job template, any attribute acceptable within the WIKI workload query data format can be used. In addition, job templates can use any of the extension attributes in the following table. Note that a checkmark in the Template Type (JMIN, JMAX, JDEF, JSET) row indicates compatibility with the associated attribute.
| ACCOUNT | |||||||||
| Format: | <ACCOUNT>[,<ACCOUNT>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | Account credentials associated with job. Used in job template matching. | ||||||||
| Example: |
JOBCFG[public] FLAGS=preemptee JOBCFG[public.min] ACCOUNT=public_acct JOBMATCHCFG[public] JMIN=public.min JSET=public |
||||||||
| ACTION | |||||||||
| Format: | HOLD or CANCEL | ||||||||
| Template Type: |
|
||||||||
| Description: | Describes an action that is done to the job if the set template is applied. | ||||||||
| Example: | JOBCFG[test.min] QOS=high JOBCFG[test.set] ACTION=CANCEL JOBMATCHCFG[test] JMIN=test.min JSET=test.set |
||||||||
| ALLOCSIZE | |||||||||
| Format: | <ALLOCSIZE>[,<DEALLOCSIZE>] | ||||||||
| Type: |
|
||||||||
| Description: | Number of application tasks to allocate/deallocate at each allocation adjustment. | ||||||||
| Example: | JOBCFG[webdb] ALLOCSIZE=4 |
||||||||
| ALLOCDELAY | |||||||||
| Format: |
|
||||||||
| Template Type: |
|
||||||||
| Description: |
|
||||||||
| Example: | JOBCFG[webdb] ALLOCDELAY=1:00 |
||||||||
| ALLOCSYNCTIME | |||||||||
| Format: | [[[DD]:HH]:MM]:SS | ||||||||
| Template Type: |
|
||||||||
| Description: | Time indicating maximum duration Moab waits before forcing allocation decisions via provisioning or other steps.
|
||||||||
| Example: |
Moab forces the deallocation of resources assigned to webdb if the application does not release them within four minutes of a deallocation request. For the sirius application, Moab forces deallocation if not released by the application within twenty seconds of a deallocation request. |
||||||||
| CLASS | |||||||||
| Format: |
|
||||||||
| Template Type: |
|
||||||||
| Description: | Class credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[night] FLAGS=preemptor JOBCFG[night.min] CLASS=night_class JOBMATCHCFG[night] JMIN=night.min JSET=night |
||||||||
| CPULIMIT | |||||||||
| Format: |
|
||||||||
| Template Type: |
|
||||||||
| Description: |
|
||||||||
| Example: | JOBCFG[job.min] CPULIMIT=1:00:00:00 JOBCFG[job.max] CPULIMIT=2:00:00:00 |
||||||||
| DESCRIPTION | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Description of the job. | ||||||||
| Example: | JOBCFG[webdb] DESCRIPTION="Template job" |
||||||||
| DPROCS | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Number of processors dedicated per task. Default is 1. | ||||||||
| Example: | JOBCFG[job.min] DPROCS=2 JOBCFG[job.max] DPROCS=4 |
||||||||
| EUSER | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | In a job template, EUSER is the effective user, meaning that when a job starts it will change the user running the job to the one on the template. | ||||||||
| Example: | JOBCFG[batch] EUSER=render |
||||||||
| EXEC | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Sets what the job runs, regardless of what the user sets. | ||||||||
| Example: | JOBCFG[setup.pre] EXEC=nfs/tools/setup.pre.sh |
||||||||
| FLAGS | |||||||||
| Format: | <JOBFLAG>[,<JOBFLAG>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | One or more legal job flag values. | ||||||||
| Example: | JOBCFG[webdb] FLAGS=NORMSTART |
||||||||
| GNAME | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Group credential associated with job. | ||||||||
| Example: | JOBCFG[webserv] GNAME=service
|
||||||||
| GRES | |||||||||
| Format: | <generic resource>[:<COUNT>] | ||||||||
| Template Type: |
|
||||||||
| Description: | Consumable generic attributes associated with individual nodes or the special pseudo-node global, which provides shared cluster (floating) consumable resources. Use the NODECFG parameter to configure such resources. | ||||||||
| Example: | JOBCFG[gres.set] GRES=abaqus:2 In this example, the gres.set template applies two Abaqus licenses per task to a matched job. |
||||||||
| GROUP | |||||||||
| Format: | <GROUP>[,<GROUP>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | Group credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[webserv] GROUP=service
|
||||||||
| MEM | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Maximum amount of physical memory per task used by the job. See Requesting Resources for more information. | ||||||||
| Example: | JOBCFG[smalljobs] MEM=25 |
||||||||
| NODEACCESSPOLICY | |||||||||
| Format: | One of the following: SHARED, SHAREDONLY, SINGLEJOB, SINGLETASK , SINGLEUSER, or UNIQUEUSER | ||||||||
| Template Type: |
|
||||||||
| Description: | Specifies how node resources will be shared by a job. See the Node Access Overview for more information. | ||||||||
| Example: | JOBCFG[serverapp] NODEACCESSPOLICY=SINGLEJOB |
||||||||
| NODERANGE | |||||||||
| Format: | <MIN>[,<MAX>] | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum nodes allowed to be allocated to job. | ||||||||
| Example: | JOBCFG[vizserver] NODERANGE=1,16 |
||||||||
| NODES | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Number of nodes required by job. Default is 1. See Node Definition for more information. | ||||||||
| Example: | JOBCFG[job.min] NODES=2 JOBCFG[job.max] NODES=4 |
||||||||
| PARTITION | |||||||||
| Format: | <PARTITION>[:<PARTITION>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | Specifies the partition (or partitions) in which a job must run. | ||||||||
| Example: | JOBCFG[meis] PARTITION=math:geology |
||||||||
| PREF | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Specifies which node features are preferred by the job and should be allocated if available. See PREF for more information. | ||||||||
| Example: | JOBCFG[meis] PREF=bigmem |
||||||||
| PRIORITY | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Relative job priority. | ||||||||
| Example: | JOBCFG[meis] PRIORITY=25000 |
||||||||
| PROCRANGE | |||||||||
| Format: | <MIN>[,<MAX>] | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum processors allowed to be allocated to job. | ||||||||
| Example: | JOBCFG[meis] PROCRANGE=2,64 |
||||||||
| QOS | |||||||||
| Format: | <QOS>[,<QOS>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | QoS credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[admin] RFEATURES=bigmem JOBCFG[admin.min] QOS=admin_qos JOBMATCHCFG[admin] JMIN=admin.min JSET=admin |
||||||||
| RARCH | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Architecture required by job. | ||||||||
| Example: | JOBCFG[servapp] RARCH=i386 |
||||||||
| RFEATURES | |||||||||
| Format: | <FEATURE>[,<FEATURE>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | List of features required by job. | ||||||||
| Example: | JOBCFG[servapp] RFEATURES=fast,bigmem |
||||||||
| RM | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Destination resource manager to be associated with job. | ||||||||
| Example: | JOBCFG[webdb] RM=slurm |
||||||||
| ROPSYS | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | Operating system required by job. | ||||||||
| Example: | JOBCFG[test.set] ROPSYS=windows |
||||||||
| SELECT | |||||||||
| Format: | <BOOLEAN> : TRUE | FALSE | ||||||||
| Template Type: |
|
||||||||
| Description: | Job template can be directly requested by job at submission. | ||||||||
| Example: | JOBCFG[servapp] SELECT=TRUE |
||||||||
| SOFTWARE | |||||||||
| Format: | <RESTYPE>[{+|:}<COUNT>][@<TIMEFRAME>] | ||||||||
| Template Type: |
|
||||||||
| Description: | Indicates generic resources required by the job. See SOFTWARE for more information. | ||||||||
| Example: | JOBCFG[servapp] SOFTWARE=matlab:2 |
||||||||
| SYSTEMJOBTYPE | |||||||||
| Format: | |||||||||
| Template Type: |
|
||||||||
| Description: | System job type (ex. vmcreate). | ||||||||
| Example: | JOBCFG[vmcreate.min] SYSTEMJOBTYPE=vmcreate |
||||||||
| TARGETBACKLOG | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum backlog for application within job. In the case of dynamic jobs, Moab allocates/deallocates resources as needed to keep the job within the target range. | ||||||||
| Example: | JOBCFG[pdb] TARGETBACKLOG=0.5,2.0 |
||||||||
| TARGETLOAD | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum load for application within job. In the case of dynamic jobs, Moab allocates/deallocates resources as needed to keep the job within the target range. | ||||||||
| Example: | JOBCFG[pdb] TARGETLOAD=0.5,2.0 |
||||||||
| TARGETRESPONSETIME | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum response time in seconds for application within job. In the case of dynamic jobs, Moab allocates/deallocates resources as needed to keep the job within the target range. | ||||||||
| Example: | JOBCFG[pdb] TARGETRESPONSETIME=0.5,2.0 |
||||||||
| TARGETTHROUGHPUT | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
|
||||||||
| Description: | Minimum and maximum throughput for application within job. In the case of dynamic jobs, Moab allocates/deallocates resources as needed to keep the job within the target range. | ||||||||
| Example: | JOBCFG[pdb] TARGETTHROUGHPUT=0.5,2.0 |
||||||||
| TASKS | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Number of tasks required by job. Default is 1. See Task Definition for more information. | ||||||||
| Example: | JOBCFG[job.min] TASKS=4 JOBCFG[job.max] TASKS=8 |
||||||||
| TASKPERNODE | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
|
||||||||
| Description: | Exact number of tasks required per node. Default is 0. | ||||||||
| Example: | JOBCFG[job.min] TASKPERNODE=2 JOBCFG[job.max] TASKPERNODE=4 |
||||||||
| TEMPLATEDEPEND | |||||||||
| Format: | <TYPE>:<TEMPLATE_NAME> | ||||||||
| Template Type: |
|
||||||||
| Description: | Create another job from the TEMPLATE_NAME job template, on which any jobs using this template will depend. This is used for dynamically creating workflows. See Job Dependencies for more information. | ||||||||
| Example: | JOBCFG[engineering] TEMPLATEDEPEND=AFTER:setup.pre JOBCFG[setup.pre] SELECT=TRUE EXEC=/tools/setup.pre.sh |
||||||||
| UNAME | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
|
||||||||
| Description: | User credential associated with job. | ||||||||
| Example: | JOBCFG[webserv] UNAME=service
|
||||||||
| USER | |||||||||
| Format: | <USER>[,<USER>]... | ||||||||
| Template Type: |
|
||||||||
| Description: | User credentials associated with job. | ||||||||
| Example: | JOBCFG[webserv] USER=service
|
||||||||
| VARIABLE | |||||||||
| Format: | <NAME>[:<VAL>] | ||||||||
| Template Type: |
|
||||||||
| Description: | Variables attached to the job template. | ||||||||
| Example: | JOBCFG[this] VARIABLE=var1:1 VARIABLE=var2:1
|
||||||||
| WCLIMIT | |||||||||
| Format: | [[HH:]MM:]SS | ||||||||
| Template Type: |
|
||||||||
| Description: | Walltime required by job. Default is 864000 (100 hours). | ||||||||
| Example: | JOBCFG[job.min] WCLIMIT=2:00:00 JOBCFG[job.max] WCLIMIT=12:00:00 |
||||||||
| WORK | |||||||||
| Format: | <DOUBLE> | ||||||||
| Template Type: |
|
||||||||
| Description: | Unitless measure of work accomplished per task. | ||||||||
| Example: | JOBCFG[webserv] WORK=10.5 |
||||||||
11.10.4 Resource Manager Templates
Job templates can also be applied to resource managers modifying attributes of jobs submitted to, migrated to, or routed through specified resource manager interfaces. In particular the RMCFG attributes MAX.JOB, MIN.JOB, SET.JOB, and DEFAULT.JOB can be used. However, the meanings of resource manager job templates are slightly different than their JOBMATCHCFG counterparts as described in the following table.
11.10.5 Job Template Examples
Job templates can be used for a wide range of purposes including enabling automated learning, setting up custom application environments, imposing special account constraints, and applying group default settings. The following examples highlight some of these uses:
11.10.5.1 Example 1: Setting Up Application-Specific Environments
JOBCFG[xxx] EXEC=*app* JOBPROLOG=/usr/local/appprolog.x
11.10.5.2 Example 2: Applying Job Preferences and Defaults
JOBCFG[xxx] CLASS=appq EXEC=*app* PREF=clearspeed NODEALLOCATIONPOLICY PRIORITY NODECFG[DEFAULT] PRIOF=5.0*PREF
11.10.5.3 Example 3: Applying Resource Constraints to Fuzzy Collections
In the following example, a job template match is set up. Using the JOBMATCHCFG parameter, Moab is configured to apply all attributes of the inter.set job template to all jobs that match the constraints of the inter.min job template. In this example, all interactive jobs are assigned the ignpolicies flag that allows them to ignore active, idle, system, and partition level policies. Interactive jobs are also locked into the test standing reservation and thus only allowed to run on the associated nodes.
# limit all users to a total of two non-interactive jobs USERCFG[DEFAULT] MAXJOB=2 SRCFG[test] DESCRIPTION="compute pool for interactive and short duration jobs" SRCFG[test] JOBATTRLIST=INTERACTIVE SRCFG[test] MAXTIME=1:00:00 SRCFG[test] HOSTLIST=R:atl[16-63] JOBCFG[inter.min] FLAGS=interactive JOBCFG[inter.set] FLAGS=ignpolicies JOBMATCHCFG[interactive] JMIN=inter.min JSET=inter.set
11.10.5.4 Example 4: Resource Manager Templates
In the following example, interactive jobs are not allowed to enter through this resource manager and any job that does route in from this resource manager interface has the preemptee flag set.
JOBCFG[no_inter] FLAGS=interactive JOBCFG[preempt_job] FLAGS=preemptee RMCFG[gridA.in] MAX.JOB=no_inter SET.JOB=preempt_job
11.10.6 Managing Job Templates
11.10.6.1 Dynamically Creating Job Templates
Job templates can be dynamically created while Moab is running using the mschedctl -m command.
|
Dynamically created job templates will only survive a Moab restart if the --flags=pers[istent] flag is used. |
> mschedctl -m --flags=pers "JOBCFG[preempt] FLAGS=PREEMPTOR" > mschedctl -m --flags=pers "JOBCFG[preempt] SELECT=TRUE" > msub -l template=preempt mysjob.sh
11.10.6.2 Removing Job Templates
Job templates cannot be dynamically removed. You can only remove job templates by commenting out (or removing) the template definition (lines) in the moab.cfg file; then you must restart Moab.11.10.6.3 Dynamically Modifying Job Templates
Job templates can be dynamically modified by using the mschedctl -m command.
|
Dynamic changes do not persist after a shutdown of the Moab server unless the --flags=pers[istent] flag is used. |
> mschedctl -m "JOBCFG[single] NODEACCESSPOLICY=SINGLEJOB"
11.10.6.4 Viewing Job Templates
Job templates can be viewed by specifying "template:" before the template name in checkjob and also by using the command mdiag -j as in the following example:
> mdiag -j --flags=policy > checkjob -v template:match.min
See Also