2.1 Job Submission

• 2.1.1 Multiple Jobs Submission
• 2.1.2 Requesting Resources
• 2.1.3 Requesting Generic Resources
• 2.1.4 Requesting Floating Resources
• 2.1.5 Requesting Other Resources
• 2.1.6 Exported Batch Environment Variables
• 2.1.7 Enabling Trusted Submit Hosts
• 2.1.8 Example Submit Scripts

Job submission is accomplished using the qsub command, which takes a number of command line arguments and integrates such into the specified PBS command file. The PBS command file may be specified as a filename on the qsub command line or may be entered via STDIN.

• The PBS command file does not need to be executable.
• The PBS command file may be piped into qsub (i.e., cat pbs.cmd | qsub)
• In the case of parallel jobs, the PBS command file is staged to, and executed on, the first allocated compute node only. (Use pbsdsh to run actions on multiple nodes.)
• The command script is executed from the user's home directory in all cases. (The script may determine the submission directory by using the $PBS_O_WORKDIR environment variable)
• The command script will be executed using the default set of user environment variables unless the -V or -v flags are specified to include aspects of the job submission environment.

By default, job submission is allowed only on the TORQUE server host (host on which pbs_server is running). Enablement of job submission from other hosts is documented in Configuring Job Submit Hosts.

Versions of TORQUE earlier than 2.4.5 attempted to apply queue and server defaults to a job that didn't have defaults specified. If a setting still did not have a value after that, TORQUE applied the queue and server maximum values to a job (meaning, the maximum values for an applicable setting were applied to jobs that had no specified or default value). In TORQUE 2.4.5 and later, the queue and server maximum values are no longer used as a value for missing settings.

2.1.1 Multiple Jobs Submission

Sometimes users will want to submit large numbers of jobs based on the same job script. Rather than using a script to repeatedly call qsub, a feature known as job arrays now exists to allow the creation of multiple jobs with one qsub command. Additionally, this feature includes a new job naming convention that allows users to reference the entire set of jobs as a unit, or to reference one particular job from the set.

Job arrays are submitted through the -t option to qsub, or by using #PBS -t in your batch script. This option takes a comma-separated list consisting of either a single job ID number, or a pair of numbers separated by a dash. Each of these jobs created will use the same script and will be running in a nearly identical environment.

> qstat -t 0-4 job_script
1098[].hostname

> qstat
1098[0].hostname ...
1098[1].hostname ...
1098[2].hostname ...
1098[3].hostname ...
1098[4].hostname ...

Versions of TORQUE earlier than 2.3 had different semantics for the -t argument. In these versions, -t took a single integer number—a count of the number of jobs to be created.

Each 1098[x] job has an environment variable called PBS_ARRAYID, which is set to the value of the array index of the job, so 1098[0].hostname would have PBS_ARRAYID set to 0. This allows you to create job arrays where each job in the array performs slightly different actions based on the value of this variable, such as performing the same tasks on different input files. One other difference in the environment between jobs in the same array is the value of the PBS_JOBNAME variable.

# These two examples are equivalent in TORQUE 2.2
> qsub -t 0-99
> qsub -t 100

# You can also pass comma delimited lists of ids and ranges:
> qsub -t 0,10,20,30,40
> qsub -t 0-50,60,70,80

Running qstat displays a job summary, which provides an overview of the array's state. To see each job in the array, run qstat -t.

The qalter, qdel, qhold, and qrls commands can operate on arrays—either the entire array or a range of that array. Additionally, any job in the array may be accessed normally by using that job's ID, just as you would with any other job. For example, running the following command would run only the specified job:

qrun 1098[0].hostname

2.1.1.1 Slot Limit

The slot limit is a way for administrators to limit the number of jobs from a job array that can be eligible for scheduling at the same time. When a slot limit is used, TORQUE puts a hold on all jobs in the array that exceed the slot limit. When an eligible job in the array completes, TORQUE removes the hold flag from the next job in the array. Slot limits can be declared globally with the max_slot_limit paramter, or on a per-job basis with qsub -t.

2.1.2 Requesting Resources

Various resources can be requested at the time of job submission. A job can request a particular node, a particular node attribute, or even a number of nodes with particular attributes. Either native TORQUE resources, or external scheduler resource extensions may be specified. The native TORQUE resources are listed in the following table:

> qsub -l nodes=3 -1 procs=2

The size format specifies the maximum amount in terms of bytes or words. It is expressed in the form integer[suffix]. The suffix is a multiplier defined in the following table ('b' means bytes (the default) and 'w' means words). The size of a word is calculated on the execution server as its word size.

Example 1 (qsub -l nodes)

> qsub -l nodes=12

> qsub -l nodes=2:server+14

> qsub -l nodes=server:hippi+10:noserver+3:bigmem:hippi

> qsub -l nodes=b2005+b1803+b1813

> qsub -l nodes=4:ppn=2

> qsub -l nodes=1:ppn=4

> qsub -l nodes=2:blue:ppn=2+red:ppn=3+b1014

Example 2

> qsub -l mem=200mb /home/user/script.sh

Example 3

> qsub -l nodes=node01,mem=200mb /home/user/script.sh

2.1.3 Requesting Generic Resources

When generic resources have been assigned to nodes using the server's nodes file, these resources can be requested at the time of job submission using the other field. (See Consumable Generic Resources in the Moab Workload Manager Administrator's Guide for details on configuration within Moab).

Example 1

> qsub -l other=matlab /home/user/script.sh

This can also be requested at the time of job submission using the -W x=GRES:matlab flag.

2.1.4 Requesting Floating Resources

When floating resources have been set up inside Moab, they can be requested in the same way as generic resources. Moab will automatically understand that these resources are floating and will schedule the job accordingly. (See Floating Generic Resources in the Moab Workload Manager Administrator's Guide for details on configuration within Moab.)

Example 2

> qsub -l other=matlab /home/user/script.sh

This can also be requested at the time of job submission using the -W x=GRES:matlab flag.

2.1.5 Requesting Other Resources

Many other resources can be requested at the time of job submission using the Moab Workload Manager. See Resource Manager Extensions in the Moab Workload Manager Administrator's Guide for a list of these supported requests and correct syntax.

2.1.6 Exported Batch Environment Variables

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. These variables are listed in the following table:

2.1.7 Enabling Trusted Submit Hosts

By default, only the node running the pbs_server daemon is allowed to submit jobs. Additional nodes can be trusted as submit hosts by taking any of the following steps:

• Set the allow_node_submit server parameter.
  • Allows any host trusted as a compute host to also be trusted as a submit host.
• Set the submit_hosts server parameter (comma-delimited).
  • Allows specified hosts to be trusted as a submit host.
• Use .rhosts to enable ruserok() based authentication.

See Job Submission Host Advanced Config for more information.

If allow_node_submit is set, the parameter allow_proxy_user must be set to allow user proxying when submitting/running jobs.

2.1.8 Example Submit Scripts

The following is an example job test script:

#!/bin/sh
#
#This is an example script example.sh
#
#These commands set up the Grid Environment for your job:
#PBS -N ExampleJob
#PBS -l nodes=1,walltime=00:01:00
#PBS -q np_workq
#PBS -M YOURUNIQNAME@umich.edu
#PBS -m abe

#print the time and date
date

#wait 10 seconds
sleep 10

#print the time and date again
date

See Also

• Maui Documentation
• http://www.lunarc.lu.se/Support/SpecialTopics/ExampleScripts/MatlabScripts/MatlabScript
• http://www.clusters.umaine.edu/wiki/index.php/Example_Submission_Scripts
• qsub wrapper - Allow local checking and modification of submitted jobs