11.10 Job Templates |
|
11.10 Job Templates |
|
Job templates are used for two primary purposes: (1) to provide a means of generically matching and categorizing jobs, and (2) to provide a means of setting arbitrary default or forced attributes for certain jobs. Job templates can be used in many aspects of scheduling but they are most commonly applied in the areas of Peer-Based Grid usage policies. Job templates are defined using the JOBCFG configuration parameter.
The JOBMATCHCFG parameter allows relationships to be established between a number of job templates. JMAX and JMIN function as filters to determine whether a job is eligible for a subsequent template to be applied to the job. If a job is eligible, JDEF and JSET templates apply attributes to the job. The table in section Job Template Extension Attributes indicates which job template types are compatible with which job template extension attributes. The following types of templates can be specified with the JOBMATCHCFG parameter:
 | For JMAX, a job template can specify only positive non-zero numbers as maximum limits for generic resources. If a job requests a generic resource that is not limited by the template, then the template can still be used. |
# limit all users to a total of two non-interactive jobs USERCFG[DEFAULT] MAXJOB=2 # reservation configuration SRCFG[test] DESCRIPTION="compute pool for interactive and short duration jobs" SRCFG[test] JOBATTRLIST=INTERACTIVE SRCFG[test] MAXTIME=1:00:00 SRCFG[test] HOSTLIST=R:atl[16-63] # job template configuration JOBCFG[inter.min] FLAGS=interactive JOBCFG[inter.set] FLAGS=ignpolicies JOBMATCHCFG[interactive] JMIN=inter.min JSET=inter.set
In the preceding example, a reservation called test is created. This reservation is only accessible by interactive jobs and jobs that will take less than one hour to complete. Specifying MAXJOB=2 means that each user on the system is only allowed to run two jobs simultaneously.
The job template configuration is signified by the JOBCFG parameter. The inter.min template (JMIN) determines that if a job is marked as interactive it is eligible to have inter.settemplate applied to it. The JOBMATCHCFG parameter establishes the relationship between inter.min and inter.set. The inter.set template forces the ignpolicies flag on the job, which allows the job to ignore the MAXJOB policy. Also, the job automatically requests the reservation named test. Any user submitting a job may have as many interactive jobs running as resources permit, and the job runs inside the test reservation, which constrains the jobs to run on nodes at1[16-63]. With this template, users do not need to know to which nodes to send interactive jobs; Moab automatically sends interactive jobs to the nodes specified in the reservation.
Rather than matching a job automatically based on job attributes, a specific template can be directly requested by setting the SELECT attribute to TRUE. Attributes specified in the job template are applied to the job.
JOBCFG[inter.set] FLAGS=ignpolicies SELECT=true
In the preceding example, instead of the interactive jobs automatically going to the reservation named test (as in the Matching example), users can request the inter.set template for the ignpolicies flag and the requested reservation test to be applied to the job.
Use the following to request the job template:
Selecting a specific reservation is often used in creating workflows. See Creating Workflows with Job Templates for more information.
msub -l template=inter.set myjob.sh | Attributes set in a template are evaluated as if they were part of the original job submission. They are still subject to all the same ACLs and policies. |
When creating a job template, any attribute acceptable within the WIKI workload query data format can be used. In addition, job templates can use any of the extension attributes in the following table. Note that a checkmark in the Template Type (JMIN, JMAX, JDEF, JSET) row indicates compatibility with the associated attribute.
| While JMIN and JMAX both work with numerical values, only JMIN can be used to test text values. |
| ACCOUNT | |||||||||
| Format: | <ACCOUNT>[,<ACCOUNT>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | Account credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[public] FLAGS=preemptee JOBCFG[public.min] ACCOUNT=public_acct JOBMATCHCFG[public] JMIN=public.min JSET=public | ||||||||
| ACTION | |||||||||
| Format: | HOLD or CANCEL | ||||||||
| Template Type: |
| ||||||||
| Description: | Describes an action that is done to the job if the set template is applied. | ||||||||
| Example: | JOBCFG[test.min] QOS=high JOBCFG[test.set] ACTION=CANCEL JOBMATCHCFG[test] JMIN=test.min JSET=test.set | ||||||||
| CLASS | |||||||||
| Format: |
| ||||||||
| Template Type: |
| ||||||||
| Description: | Class credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[night] FLAGS=preemptor JOBCFG[night.min] CLASS=night_class JOBMATCHCFG[night] JMIN=night.min JSET=night | ||||||||
| CPULIMIT | |||||||||
| Format: |
| ||||||||
| Template Type: |
| ||||||||
| Description: |
| ||||||||
| Example: | JOBCFG[job.min] CPULIMIT=1:00:00:00 JOBCFG[job.max] CPULIMIT=2:00:00:00 | ||||||||
| DESCRIPTION | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Description of the job. When you run the checkjob command, the description appears as Reason. | ||||||||
| Example: | JOBCFG[webdb] DESCRIPTION="Template job" | ||||||||
| DPROCS | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Number of processors dedicated per task. Default is 1. | ||||||||
| Example: | JOBCFG[job.min] DPROCS=2 JOBCFG[job.max] DPROCS=4 | ||||||||
| EUSER | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | In a job template, EUSER is the effective user, meaning that when a job starts it will change the user running the job to the one on the template. | ||||||||
| Example: | JOBCFG[batch] EUSER=render | ||||||||
| EXEC | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Sets what the job runs, regardless of what the user sets. | ||||||||
| Example: | JOBCFG[setup.pre] EXEC=nfs/tools/setup.pre.sh | ||||||||
| FLAGS | |||||||||
| Format: | <JOBFLAG>[,<JOBFLAG>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | One or more legal job flag values. | ||||||||
| Example: | JOBCFG[webdb] FLAGS=NORMSTART | ||||||||
| GNAME | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Group credential associated with job. | ||||||||
| Example: | JOBCFG[webserv] GNAME=service
| ||||||||
| GRES | |||||||||
| Format: | <generic resource>[:<COUNT>] | ||||||||
| Template Type: |
| ||||||||
| Description: | Consumable generic attributes associated with individual nodes or the special pseudo-node global, which provides shared cluster (floating) consumable resources. Use the NODECFG parameter to configure such resources. | ||||||||
| Example: | JOBCFG[gres.set] GRES=abaqus:2 In this example, the gres.set template applies two Abaqus licenses per task to a matched job. | ||||||||
| GROUP | |||||||||
| Format: | <GROUP>[,<GROUP>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | Group credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[webserv] GROUP=service
| ||||||||
| MEM | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Maximum amount of physical memory per task used by the job. See Requesting Resources for more information. | ||||||||
| Example: | JOBCFG[smalljobs] MEM=25 | ||||||||
| NODEACCESSPOLICY | |||||||||
| Format: | One of the following:SHARED, SHAREDONLY, SINGLEJOB, SINGLETASK , SINGLEUSER, or UNIQUEUSER | ||||||||
| Template Type: |
| ||||||||
| Description: | Specifies how node resources will be shared by a job. See the Node Access Overview for more information. | ||||||||
| Example: | JOBCFG[serverapp] NODEACCESSPOLICY=SINGLEJOB | ||||||||
| NODERANGE | |||||||||
| Format: | <MIN>[,<MAX>] | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum nodes allowed to be allocated to job. | ||||||||
| Example: | JOBCFG[vizserver] NODERANGE=1,16 | ||||||||
| NODES | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Number of nodes required by job. Default is 1. See Node Definition for more information. | ||||||||
| Example: | JOBCFG[job.min] NODES=2 JOBCFG[job.max] NODES=4 | ||||||||
| NODESET | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | See Node Set Overview for more information. | ||||||||
| Example: | JOBCFG[medium.set] NODESET=ONEOF:FEATURE:fast,slow | ||||||||
| PARTITION | |||||||||
| Format: | <PARTITION>[:<PARTITION>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | Specifies the partition (or partitions) in which a job must run. | ||||||||
| Example: | JOBCFG[meis] PARTITION=math:geology | ||||||||
| PREF | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Specifies which node features are preferred by the job and should be allocated if available. See PREF for more information. | ||||||||
| Example: | JOBCFG[meis] PREF=bigmem | ||||||||
| PRIORITY | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Relative job priority.
| ||||||||
| Example: | JOBCFG[meis] PRIORITY=25000 | ||||||||
| PROCRANGE | |||||||||
| Format: | <MIN>[,<MAX>] | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum processors allowed to be allocated to job. | ||||||||
| Example: | JOBCFG[meis] PROCRANGE=2,64 | ||||||||
| QOS | |||||||||
| Format: | <QOS>[,<QOS>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | QoS credentials associated with job. Used in job template matching. | ||||||||
| Example: | JOBCFG[admin] RFEATURES=bigmem JOBCFG[admin.min] QOS=admin_qos JOBMATCHCFG[admin] JMIN=admin.min JSET=admin | ||||||||
| RARCH | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Architecture required by job. | ||||||||
| Example: | JOBCFG[servapp] RARCH=i386 | ||||||||
| RFEATURES | |||||||||
| Format: | <FEATURE>[,<FEATURE>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | List of features required by job. | ||||||||
| Example: | JOBCFG[servapp] RFEATURES=fast,bigmem | ||||||||
| RM | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Destination resource manager to be associated with job. | ||||||||
| Example: | JOBCFG[webdb] RM=slurm | ||||||||
| ROPSYS | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | Operating system required by job. | ||||||||
| Example: | JOBCFG[test.set] ROPSYS=windows | ||||||||
| SELECT | |||||||||
| Format: | <BOOLEAN> : TRUE | FALSE | ||||||||
| Template Type: |
| ||||||||
| Description: | Job template can be directly requested by job at submission. | ||||||||
| Example: | JOBCFG[servapp] SELECT=TRUE | ||||||||
| SOFTWARE | |||||||||
| Format: | <RESTYPE>[{+|:}<COUNT>][@<TIMEFRAME>] | ||||||||
| Template Type: |
| ||||||||
| Description: | Indicates generic resources required by the job. See SOFTWARE for more information. | ||||||||
| Example: | JOBCFG[servapp] SOFTWARE=matlab:2 | ||||||||
| SYSTEMJOBTYPE | |||||||||
| Format: | |||||||||
| Template Type: |
| ||||||||
| Description: | System job type (ex. vmcreate). | ||||||||
| Example: | JOBCFG[vmcreate.min] SYSTEMJOBTYPE=vmcreate | ||||||||
| TARGETBACKLOG | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum backlog for application within job. | ||||||||
| Example: | JOBCFG[pdb] TARGETBACKLOG=0.5,2.0 | ||||||||
| TARGETLOAD | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum load for application within job. | ||||||||
| Example: | JOBCFG[pdb] TARGETLOAD=0.5,2.0 | ||||||||
| TARGETRESPONSETIME | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum response time in seconds for application within job. | ||||||||
| Example: | JOBCFG[pdb] TARGETRESPONSETIME=0.5,2.0 | ||||||||
| TARGETTHROUGHPUT | |||||||||
| Format: | [<MIN>,]<MAX> | ||||||||
| Template Type: |
| ||||||||
| Description: | Minimum and maximum throughput for application within job. | ||||||||
| Example: | JOBCFG[pdb] TARGETTHROUGHPUT=0.5,2.0 | ||||||||
| TASKS | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Number of tasks required by job. Default is 1. See Task Definition for more information. | ||||||||
| Example: | JOBCFG[job.min] TASKS=4 JOBCFG[job.max] TASKS=8 | ||||||||
| TASKPERNODE | |||||||||
| Format: | <INTEGER> | ||||||||
| Template Type: |
| ||||||||
| Description: | Exact number of tasks required per node. Default is 0.
| ||||||||
| Example: | JOBCFG[job.min] TASKPERNODE=2 JOBCFG[job.max] TASKPERNODE=4 | ||||||||
| TEMPLATEDEPEND | |||||||||
| Format: | <TYPE>:<TEMPLATE_NAME> | ||||||||
| Template Type: |
| ||||||||
| Description: | Create another job from the TEMPLATE_NAME job template, on which any jobs using this template will depend. This is used for dynamically creating workflows. See Job Dependencies for more information. | ||||||||
| Example: | JOBCFG[engineering] TEMPLATEDEPEND=AFTER:setup.pre JOBCFG[setup.pre] SELECT=TRUE EXEC=/tools/setup.pre.sh | ||||||||
| UNAME | |||||||||
| Format: | <STRING> | ||||||||
| Template Type: |
| ||||||||
| Description: | User credential associated with job. | ||||||||
| Example: | JOBCFG[webserv] UNAME=service
| ||||||||
| USER | |||||||||
| Format: | <USER>[,<USER>]... | ||||||||
| Template Type: |
| ||||||||
| Description: | User credentials associated with job. | ||||||||
| Example: | JOBCFG[webserv] USER=service
| ||||||||
| VARIABLE | |||||||||
| Format: | <NAME>[:<VAL>] | ||||||||
| Template Type: |
| ||||||||
| Description: | Variables attached to the job template. | ||||||||
| Example: | JOBCFG[this] VARIABLE=var1:1 VARIABLE=var2:1
| ||||||||
| WCLIMIT | |||||||||
| Format: | [[HH:]MM:]SS | ||||||||
| Template Type: |
| ||||||||
| Description: | Walltime required by job. Default is 8640000 (100 days). | ||||||||
| Example: | JOBCFG[job.min] WCLIMIT=2:00:00 JOBCFG[job.max] WCLIMIT=12:00:00 | ||||||||
Job templates can also be applied to resource managers modifying attributes of jobs submitted to, migrated to, or routed through specified resource manager interfaces. In particular the RMCFG attributes MAX.JOB, MIN.JOB, SET.JOB, and DEFAULT.JOB can be used. However, the meanings of resource manager job templates are slightly different than their JOBMATCHCFG counterparts as described in the following table.
Job templates can be used for a wide range of purposes including enabling automated learning, setting up custom application environments, imposing special account constraints, and applying group default settings. The following examples highlight some of these uses:
JOBCFG[xxx] EXEC=*app* JOBPROLOG=/usr/local/appprolog.x
JOBCFG[xxx] CLASS=appq EXEC=*app* PREF=clearspeed NODEALLOCATIONPOLICY PRIORITY NODECFG[DEFAULT] PRIOF=5.0*PREF
In the following example, a job template match is set up. Using the JOBMATCHCFG parameter, Moab is configured to apply all attributes of the inter.set job template to all jobs that match the constraints of the inter.min job template. In this example, all interactive jobs are assigned the ignpolicies flag that allows them to ignore active, idle, system, and partition level policies. Interactive jobs are also locked into the teststanding reservation and thus only allowed to run on the associated nodes.
# limit all users to a total of two non-interactive jobs USERCFG[DEFAULT] MAXJOB=2 SRCFG[test] DESCRIPTION="compute pool for interactive and short duration jobs" SRCFG[test] JOBATTRLIST=INTERACTIVE SRCFG[test] MAXTIME=1:00:00 SRCFG[test] HOSTLIST=R:atl[16-63] JOBCFG[inter.min] FLAGS=interactive JOBCFG[inter.set] FLAGS=ignpolicies JOBMATCHCFG[interactive] JMIN=inter.min JSET=inter.set
In the following example, interactive jobs are not allowed to enter through this resource manager and any job that does route in from this resource manager interface has the preemptee flag set.
JOBCFG[no_inter] FLAGS=interactive JOBCFG[preempt_job] FLAGS=preemptee RMCFG[gridA.in] MAX.JOB=no_inter SET.JOB=preempt_job
Job templates can be dynamically created while Moab is running using the mschedctl -m command.
| Dynamically created job templates will only survive a Moab restart if the --flags=pers[istent] flag is used. |
> mschedctl -m --flags=pers "JOBCFG[preempt] FLAGS=PREEMPTOR" > mschedctl -m --flags=pers "JOBCFG[preempt] SELECT=TRUE" > msub -l template=preempt mysjob.sh
Job templates can be dynamically modified by using the mschedctl -m command.
| Dynamic changes do not persist after a shutdown of the Moab server unless the --flags=pers[istent] flag is used. |
> mschedctl -m "JOBCFG[single] NODEACCESSPOLICY=SINGLEJOB"
Job templates can be viewed by specifying "template:" before the template name in checkjob and also by using the command mdiag -j as in the following example:
> mdiag -j --flags=policy > checkjob -v template:match.min
Job templates can be requested directly by setting the SELECT flag on the job template to TRUE as in the following example:
> JOBCFG[medium.set] NODESET=ONEOF:FEATURE:fast,slow SELECT=true > msub -l template=medium.set
11.10.7 Creating Workflows with Job Templates |  |
Moab can create workflows from individual jobs using job templates. The Moab naming convention for jobs created with job templates is <moabid>.<templatename>. By default, when Moab submits jobs to only one resource manager, the job IDs are synchronized with the resource manger's job IDs. For consistency, the parameter USEMOABJOBID may be used so that a template-created job is easily associated with its parent job (such as moab.1, moab.1.setup.pre).
| JSETs and JDEFs have only been tested using msub as the job submission command. |
In this example, creating a job in the engineering account generates a workflow based on a two-step setup. The job is set to depend on the completion of this workflow.
JOBMATCHCFG[engineering.job] JMIN=engineering.match JSET=engineering JOBCFG[engineering.match] ACCOUNT=engineering JOBCFG[engineering] TEMPLATEDEPEND=AFTER:setup.pre2 JOBCFG[setup.pre2] TEMPLATEDEPEND=AFTER:setup.pre SELECT=TRUE EXEC=/nfs/tools/setup.pre2.sh JOBCFG[setup.pre] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/setup.pre.sh STAGEIN=TRUE
In the previous example, a job is submitted with the account name engineering. Moab then matches the submitted job with the engineering template. The engineering template states that the job must run after the setup.pre2 template job runs. Then the setup.pre2 template creates a job from the specified EXEC string; the setup.pre2 job must run after the setup.pre template job runs. The setup.pre template creates a job from the specified EXEC string. The new setup.pre job is specified for two processors and has a wallclock time of one minute.
Note that since the setup.pre.sh script is not on an NFS mount, it must be staged in. STAGEIN has a value of TRUE or FALSE. If set to TRUE, Moab ensures the executable is staged out to the compute host where the dependent job runs. If set to FALSE, Moab assumes the executable is already available on the compute host.
In this example the job will depend on the completion of two other jobs Moab creates. Both jobs execute at the same time.
# Engineering2 JOBCFG[engineering2] TEMPLATEDEPEND=AFTER:engineering2.pre2 TEMPLATEDEPEND=AFTER:engineering2.pre JOBCFG[engineering2.pre2] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/engineering2.pre2.sh JOBCFG[engineering2.pre] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/engineering2.pre.sh JOBCFG[engineering2.match] ACCOUNT=engineering2 JOBMATCHCFG[engineering2.job] JMIN=engineering2.match JSET=engineering2
Three additional jobs are created that depend on the submitted job.
# Workflow 2 JOBCFG[workflow2] TEMPLATEDEPEND=BEFORE:workflow2.post1 TEMPLATEDEPEND=BEFORE:workflow2.post2 TEMPLATEDEPEND=BEFORE:workflow2.post3 JOBCFG[workflow2.post1] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow2.post1.sh JOBCFG[workflow2.post2] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow2.post2.sh JOBCFG[workflow2.post3] TASKS=2 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow2.post3.sh JOBCFG[workflow2.match] ACCOUNT=workflow2 JOBMATCHCFG[workflow2.job] JMIN=workflow2.match JSET=workflow2
A complex workflow that handles failures.
# Workflow 4 JOBCFG[workflow4.step1] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step1.sh TEMPLATEDEPEND=BEFOREFAIL:workflow4.fail1 JOBCFG[workflow4.fail1] TASKS=1 WCLIMIT=00:00:30 SELECT=TRUE EXEC=/usr/tools/workflow.fail.1.sh TEMPLATEDEPEND=BEFOREANY:workflow4.fail2 JOBCFG[workflow4.fail2] TASKS=1 WCLIMIT=00:00:30 SELECT=TRUE EXEC=/usr/tools/workflow.fail.2.sh # Submission job JOBCFG[workflow4.step2] TEMPLATEDEPEND=AFTEROK:workflow4.step1 TEMPLATEDEPEND=BEFOREOK:workflow4.step3.1 TEMPLATEDEPEND=BEFOREOK:workflow4.step3.2 JOBCFG[workflow4.step3.1] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step3.1.sh JOBCFG[workflow4.step3.2] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step3.2.sh TEMPLATEDEPEND=BEFOREOK:workflow4.step4 JOBCFG[workflow4.step4] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step4.sh JOBCFG[workflow4.step4] TEMPLATEDEPEND=BEFOREOK:workflow4.step5.1 TEMPLATEDEPEND=BEFOREOK:workflow4.step5.2 TEMPLATEDEPEND=BEFORENOTOK:workflow4.step5.3 JOBCFG[workflow4.step5.1] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step5.1.sh JOBCFG[workflow4.step5.2] TASKS=1 WCLIMIT=00:01:00 SELECT=TRUE EXEC=/usr/tools/workflow.step5.2.sh JOBCFG[workflow4.step5.3] TASKS=1 WCLIMIT=00:00:30 SELECT=TRUE EXEC=/usr/tools/workflow.step5.3.sh JOBCFG[workflow4.match] ACCOUNT=workflow4 JOBMATCHCFG[workflow4.job] JMIN=workflow4.match JSET=workflow4.step2
Copyright © 2012 Adaptive Computing Enterprises, Inc.®
