TORQUE Resource Manager
2.1 Job Submission

2.1 Job Submission

  • 2.1.1 Multiple Jobs Submission
  • 2.1.2 Requesting Resources
  • 2.1.3 Requesting Generic Resources
  • 2.1.4 Requesting Floating Resources
  • 2.1.5 Requesting Other Resources
  • 2.1.6 Exported Batch Environment Variables
  • 2.1.7 Enabling Trusted Submit Hosts
  • 2.1.8 Example Submit Scripts

Job submission is accomplished using the qsub command, which takes a number of command line arguments and integrates such into the specified PBS command file. The PBS command file may be specified as a filename on the qsub command line or may be entered via STDIN.

  • The PBS command file does not need to be executable.
  • The PBS command file may be piped into qsub (i.e., cat pbs.cmd | qsub)
  • In the case of parallel jobs, the PBS command file is staged to, and executed on, the first allocated compute node only. (Use pbsdsh to run actions on multiple nodes.)
  • The command script is executed from the user's home directory in all cases. (The script may determine the submission directory by using the $PBS_O_WORKDIR environment variable)
  • The command script will be executed using the default set of user environment variables unless the -V or -v flags are specified to include aspects of the job submission environment.

Note By default, job submission is allowed only on the TORQUE server host (host on which pbs_server is running). Enablement of job submission from other hosts is documented in Configuring Job Submit Hosts.

Note

Versions of TORQUE earlier than 2.4.5 attempted to apply queue and server defaults to a job that didn't have defaults specified. If a setting still did not have a value after that, TORQUE applied the queue and server maximum values to a job (meaning, the maximum values for an applicable setting were applied to jobs that had no specified or default value).

In TORQUE 2.4.5 and later, the queue and server maximum values are no longer used as a value for missing settings.

2.1.1 Multiple Jobs Submission

Sometimes users will want to submit large numbers of jobs based on the same job script. Rather than using a script to repeatedly call qsub, a feature known as job arrays now exists to allow the creation of multiple jobs with one qsub command. Additionally, this feature includes a new job naming convention that allows users to reference the entire set of jobs as a unit, or to reference one particular job from the set.

Job arrays are submitted through the -t option to qsub, or by using #PBS -t in your batch script. This option takes a comma-separated list consisting of either a single job ID number, or a pair of numbers separated by a dash. Each of these jobs created will use the same script and will be running in a nearly identical environment.

> qstat -t 0-4 job_script
1098[].hostname

> qstat
1098[0].hostname ...
1098[1].hostname ...
1098[2].hostname ...
1098[3].hostname ...
1098[4].hostname ...

Note Versions of TORQUE earlier than 2.3 had different semantics for the -t argument. In these versions, -t took a single integer number—a count of the number of jobs to be created.

Each 1098[x] job has an environment variable called PBS_ARRAYID, which is set to the value of the array index of the job, so 1098[0].hostname would have PBS_ARRAYID set to 0. This allows you to create job arrays where each job in the array performs slightly different actions based on the value of this variable, such as performing the same tasks on different input files. One other difference in the environment between jobs in the same array is the value of the PBS_JOBNAME variable.

# These two examples are equivalent in TORQUE 2.2
> qsub -t 0-99
> qsub -t 100

# You can also pass comma delimited lists of ids and ranges:
> qsub -t 0,10,20,30,40
> qsub -t 0-50,60,70,80

Running qstat displays a job summary, which provides an overview of the array's state. To see each job in the array, run qstat -t.

The qalter, qdel, qhold, and qrls commands can operate on arrays—either the entire array or a range of that array. Additionally, any job in the array may be accessed normally by using that job's ID, just as you would with any other job. For example, running the following command would run only the specified job:

qrun 1098[0].hostname

2.1.1.1 Slot Limit

The slot limit is a way for administrators to limit the number of jobs from a job array that can be eligible for scheduling at the same time. When a slot limit is used, TORQUE puts a hold on all jobs in the array that exceed the slot limit. When an eligible job in the array completes, TORQUE removes the hold flag from the next job in the array. Slot limits can be declared globally with the max_slot_limit paramter, or on a per-job basis with qsub -t.

2.1.2 Requesting Resources

Various resources can be requested at the time of job submission. A job can request a particular node, a particular node attribute, or even a number of nodes with particular attributes. Either native TORQUE resources, or external scheduler resource extensions may be specified. The native TORQUE resources are listed in the following table:

Resource Format Description
string Specifies the administrator defined system architecture required. This defaults to whatever the PBS_MACH string is set to in "local.mk".
seconds, or [[HH:]MM:]SS Maximum amount of CPU time used by all processes in the job.
string Specifies a user owned epilogue script which will be run before the system epilogue and epilogue.user scripts at the completion of a job. The syntax is epilogue=<file>. The file can be designated with an absolute or relative path.
size* The amount of total disk requested for the job. (Ignored on Unicos.)
string Name of the host on which the job should be run. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent.
size* Maximum amount of physical memory used by the job. (Ignored on Darwin, Digital Unix, Free BSD, HPUX 11, IRIX, NetBSD, and SunOS. Also ignored on Linux if number of nodes is not 1. Not implemented on AIX and HPUX 10.)
integer Number between -20 (highest priority) and 19 (lowest priority). Adjust the process execution priority.
{<node_count> | <hostname>} [:ppn=<ppn>][:gpus=<gpu>][:<property>[:<property>]...] [+ ...] Number and/or type of nodes to be reserved for exclusive use by the job. The value is one or more node_specs joined with the + (plus) character: node_spec[+node_spec...]. Each node_spec is a number of nodes required of the type declared in the node_spec and a name of one or more properties desired for the nodes. The number, the name, and each property in the node_spec are separated by a : (colon). If no number is specified, one (1) is assumed.

The name of a node is its hostname. The properties of nodes are:

  • ppn=# - Specify the number of virtual processors per node requested for this job.

    The number of virtual processors available on a node by default is 1, but it can be configured in the $TORQUE_HOME/server_priv/nodes file using the np attribute. The virtual processor can relate to a physical core on the node or it can be interpreted as an "execution slot" such as on sites that set the node np value greater than the number of physical cores (or hyper-thread contexts). The ppn value is a characteristic of the hardware, system, and site, and its value is to be determined by the administrator.

  • gpus=# - Specify the number of GPUs per node requested for this job.

    The number of GPUs available on a node can be configured in the $TORQUE_HOME/server_priv/nodes file using the gpu attribute. The GPU value is a characteristic of the hardware, system, and site, and its value is to be determined by the administrator.

  • property - A string assigned by the system administrator specifying a node's features. Check with your administrator as to the node names and properties available to you.
See Example 1 (-l nodes) for examples.

Note By default, the node resource is mapped to a virtual node (that is, directly to a processor, not a full physical compute node). This behavior can be changed within Maui or Moab by setting the JOBNODEMATCHPOLICY parameter. See Appendix F of the Moab Workload Manager Administrator's Guide for more information.

string Specifies the administrator defined operating system as defined in the MOM configuration file.
string Allows a user to specify site specific information. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent.

Note This does not work for msub using Moab and Maui.

seconds, or [[HH:]MM:]SS Maximum amount of CPU time used by any single process in the job.
[numa=X][[:]sockets=Y][:usethreads] Specifies the task placement of jobs.
size* Maximum amount of physical memory used by any single process of the job. (Ignored on Fujitsu. Not implemented on Digital Unix and HPUX.)
procs=<integer> (Applicable in version 2.5.0 and later.) The number of processors to be allocated to a job. The processors can come from one or more qualified node(s). Only one procs declaration may be used per submitted qsub command.
> qsub -l nodes=3 -1 procs=2
string A string made up of 1's and 0's in reverse order of the processor cores requested. A procs_bitmap=1110 means the job requests a node that has four available cores, but the job runs exclusively on cores two, three, and four. With this bitmap, core one is not used.
string Specifies a user owned prologue script which will be run after the system prologue and prologue.user scripts at the beginning of a job. The syntax is prologue=<file>. The file can be designated with an absolute or relative path.
size* Maximum amount of virtual memory used by any single process in the job. (Ignored on Unicos.)
integer For TORQUE, this resource has no meaning. It is passed on to the scheduler for interpretation. In the Moab scheduler, the size resource is intended for use in Cray installations only.
string Allows a user to specify software required by the job. This is useful if certain software packages are only available on certain systems in the site. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent. (See Scheduler License Management in the Moab Workload Manager Administrator's Guide for more information.)
size* Maximum amount of virtual memory used by all concurrent processes in the job. (Ignored on Unicos.)
seconds, or [[HH:]MM:]SS Maximum amount of real time during which the job can be in the running state.
*size format:

The size format specifies the maximum amount in terms of bytes or words. It is expressed in the form integer[suffix]. The suffix is a multiplier defined in the following table ('b' means bytes (the default) and 'w' means words). The size of a word is calculated on the execution server as its word size.

Suffix Multiplier
b w 1
kb kw 1024
mb mw 1,048,576
gb gw 1,073,741,824
tb tw 1,099,511,627,776

Example 1 (qsub -l nodes)

Usage Description
> qsub -l nodes=12
request 12 nodes of any type
> qsub -l nodes=2:server+14
request 2 "server" nodes and 14 other nodes (a total of 16) - this specifies two node_specs, "2:server" and "14"
> qsub -l nodes=server:hippi+10:noserver+3:bigmem:hippi
request (a) 1 node that is a "server" and has a "hippi" interface, (b) 10 nodes that are not servers, and (c) 3 nodes that have a large amount of memory and have hippi
> qsub -l nodes=b2005+b1803+b1813
request 3 specific nodes by hostname
> qsub -l nodes=4:ppn=2
request 2 processors on each of four nodes
> qsub -l nodes=1:ppn=4
request 4 processors on one node
> qsub -l nodes=2:blue:ppn=2+red:ppn=3+b1014
request 2 processors on each of two blue nodes, three processors on one red node, and the compute node "b1014"

Example 2

> qsub -l mem=200mb /home/user/script.sh
This job requests a node with 200 MB of available memory.

Example 3

> qsub -l nodes=node01,mem=200mb /home/user/script.sh
This job will wait until node01 is free with 200 MB of available memory.

2.1.3 Requesting Generic Resources

When generic resources have been assigned to nodes using the server's nodes file, these resources can be requested at the time of job submission using the other field. (See Consumable Generic Resources in the Moab Workload Manager Administrator's Guide for details on configuration within Moab).

Example 1

> qsub -l other=matlab /home/user/script.sh
This job will run on any node that has the generic resource matlab.

Note This can also be requested at the time of job submission using the -W x=GRES:matlab flag.

2.1.4 Requesting Floating Resources

When floating resources have been set up inside Moab, they can be requested in the same way as generic resources. Moab will automatically understand that these resources are floating and will schedule the job accordingly. (See Floating Generic Resources in the Moab Workload Manager Administrator's Guide for details on configuration within Moab.)

Example 2

> qsub -l other=matlab /home/user/script.sh
This job will run on any node when there are enough floating resources available.

Note This can also be requested at the time of job submission using the -W x=GRES:matlab flag.

2.1.5 Requesting Other Resources

Many other resources can be requested at the time of job submission using the Moab Workload Manager. See Resource Manager Extensions in the Moab Workload Manager Administrator's Guide for a list of these supported requests and correct syntax.

2.1.6 Exported Batch Environment Variables

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. These variables are listed in the following table:

Variable Description
user specified jobname
zero-based value of job array index for this job (in version 2.2.0 and later)
job's submission directory
N/A
number of tasks requested
home directory of submitting user
active port for MOM daemon
name of submitting user
language variable for job
job cookie
node offset number
script shell
unique pbs job id
host on which job script is currently running
job queue
file containing line delimited list on nodes allocated to the job
path variable used to locate executables within job script

2.1.7 Enabling Trusted Submit Hosts

By default, only the node running the pbs_server daemon is allowed to submit jobs. Additional nodes can be trusted as submit hosts by taking any of the following steps:

  • Set the allow_node_submit server parameter.
    • Allows any host trusted as a compute host to also be trusted as a submit host.
  • Set the submit_hosts server parameter (comma-delimited).
    • Allows specified hosts to be trusted as a submit host.
  • Use .rhosts to enable ruserok() based authentication.

See Job Submission Host Advanced Config for more information.

Note If allow_node_submit is set, the parameter allow_proxy_user must be set to allow user proxying when submitting/running jobs.

2.1.8 Example Submit Scripts

The following is an example job test script:

#!/bin/sh
#
#This is an example script example.sh
#
#These commands set up the Grid Environment for your job:
#PBS -N ExampleJob
#PBS -l nodes=1,walltime=00:01:00
#PBS -q np_workq
#PBS -M YOURUNIQNAME@umich.edu
#PBS -m abe

#print the time and date
date

#wait 10 seconds
sleep 10

#print the time and date again
date

See Also