qsub

Declares the time after which the job is eligible for execution.

The date_time argument is in the form:

[[[[CC]YY]MM]DD]hhmm[.SS]

where CC is the first two digits of the year (the century), YY is the second two digits of the year, MM is the two digits for the month, DD is the day of the month, hh is the hour, mm is the minute, and the optional SS is the seconds.

If the month (MM) is not specified, it will default to the current month if the specified day (DD) is in the future. Otherwise, the month will be set to next month. Likewise, if the day (DD) is not specified, it will default to today if the time (hhmm) is in the future. Otherwise, the day will be set to tomorrow.

For example, if you submit a job at 11:15 am with a time of -a 1110, the job will be eligible to run at 11:10 am tomorrow.

Defines the maximum number of seconds qsub will block attempting to contact pbs_server. If pbs_server is down, or for a variety of communication failures, qsub will continually retry connecting to pbs_server for job submission.

This value overrides the CLIENTRETRY parameter in torque.cfg. This is a non-portable Torque extension. Portability-minded users can use the PBS_CLIENTRETRY environmental variable. A negative value is interpreted as infinity. The default is 0.

Defines the options that will apply to the job. If the job executes upon a host which does not support checkpoint, these options will be ignored.

Valid checkpoint options are:

• none – No checkpointing is to be performed.
• enabled – Specify that checkpointing is allowed but must be explicitly invoked by either the qhold or qchkpt commands.
• shutdown – Specify that checkpointing is to be done on a job at pbs_mom shutdown.
• periodic – Specify that periodic checkpointing is enabled. The default interval is 10 minutes and can be changed by the $checkpoint_interval option in the MOM config file or by specifying an interval when the job is submitted
• interval=minutes – Checkpointing is to be performed at an interval of minutes, which is the integer number of minutes of wall time used by the job. This value must be greater than zero.
• depth=number – Specify a number (depth) of checkpoint images to be kept in the checkpoint directory.
• dir=path – Specify a checkpoint directory (default is /var/spool/torque/checkpoint).

Defines the prefix that declares a directive to the qsub command within the script file. (See the paragraph on script directives under Extended description.)

If the -C option is presented with a directive_prefix argument that is the null string, qsub will not scan the script file for directives.

Defines the path to be used for the standard error stream of the batch job. The path argument is of the form:

[hostname:]path_name

where hostname is the name of a host to which the file will be returned, and path_name is the path name on that host in the syntax recognized by POSIX.

When specifying a directory for the location you need to include a trailing slash.

The argument will be interpreted as follows:

• path_name – where path_name is not an absolute path name, then the qsub command will expand the path name relative to the current working directory of the command. The command will supply the name of the host upon which it is executing for the hostname component.
• hostname:path_name – where path_name is not an absolute path name, then the qsub command will not expand the path name relative to the current working directory of the command. On delivery of the standard error, the path name will be expanded relative to the user's home directory on the hostname system.
• path_name – where path_name specifies an absolute path name, then the qsub will supply the name of the host on which it is executing for the hostname.
• hostname:path_name – where path_name specifies an absolute path name, the path will be used as specified.

If the -e option is not specified, the default file name for the standard error stream will be used. The default name has the following form:

• job_name.esequence_number – where job_name is the name of the job (see the -n name option) and sequence_number is the job number assigned when the job is submitted.

Job is made fault tolerant. Jobs running on multiple nodes are periodically polled by mother superior. If one of the nodes fails to report, the job is canceled by mother superior and a failure is reported. If a job is fault tolerant, it will not be canceled based on failed polling (no matter how many nodes fail to report). This may be desirable if transient network failures are causing large jobs not to complete, where ignoring one failed polling attempt can be corrected at the next polling attempt.

If Torque is compiled with PBS_NO_POSIX_VIOLATION (there is no config option for this), you have to use -W fault_tolerant=true to mark the job as fault tolerant.

Specifies the arguments that will be passed to the job script when the script is launched. The accepted syntax is:

qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript2.sh

Quotation marks are required. qsub will fail with an error message if the argument following -F is not a quoted value. The pbs_mom server will pass the quoted value as arguments to the job script when it launches the script.

Sets an idle slot limit for the job array being submitted. If this parameter is set for a non-array job, it will be rejected. Additionally, if the user requests an idle slot limit that exceeds the server parameter's default, the job will be rejected. See also the idle_slot_limit server parameter.

$ qsub -t 0-99 -i 10 script.sh

Declares if the standard error stream of the job will be merged with the standard output stream of the job.

An option argument value of oe directs that the two streams will be merged, intermixed, as standard output. An option argument value of eo directs that the two streams will be merged, intermixed, as standard error.

If the join argument is n or the option is not specified, the two streams will be two separate files.

If using either the -e or the -o option and the -j eo|oe option, the -j option takes precedence and all standard error and output messages go to the chosen output file.

Defines which (if either) of standard output or standard error will be retained on the execution host. If set for a stream, this option overrides the path name for that stream. If not set, neither stream is retained on the execution host.

The argument is either the single letter "e" or "o", or the letters "e" and "o" combined in either order. Or the argument is the letter "n".

• e – The standard error stream is to be retained on the execution host. The stream will be placed in the home directory of the user under whose user id the job executed. The file name will be the default file name given by:

job_name.esequence

where job_name is the name specified for the job, and sequence is the sequence number component of the job identifier.

• o – The standard output stream is to be retained on the execution host. The stream will be placed in the home directory of the user under whose user id the job executed. The file name will be the default file name given by:

job_name.osequence

where job_name is the name specified for the job, and sequence is the sequence number component of the job identifier.

• eo – Both the standard output and standard error streams will be retained.
• oe – Both the standard output and standard error streams will be retained.
• n – Neither stream is retained.

When set on a job, overrides server and queue kill_delay settings. The kill_delay value is a positive integer. The default is 0. See kill_delay for more information.

resource_list

Defines the resources that are required by the job and establishes a limit to the amount of resource that can be consumed. See 5.681 Requesting Resources for more information.

If not set for a generally available resource, such as CPU time, the limit is infinite. The resource_list argument is of the form:

resource_name[=[value]][,resource_name[=[value]],...]

In this situation, you should request the more inclusive resource first. For example, a request for procs should come before a gres request.

In Torque 3.0.2 or later, qsub supports the mapping of -l gpus=X to -l gres=gpus:X. This allows users who are using NUMA systems to make requests such as -l ncpus=20:gpus=5 indicating they are not concerned with the GPUs in relation to the NUMA nodes they request, they only want a total of 20 cores and 5 GPUs.

-l supports some Moab-only extensions. See 5.681 Requesting Resources for more information on native Torque resources. qsub -W x= is recommended instead (supports more options). See -W for more information.

For information on specifying multiple types of resources for allocation, see Multi-Req Support in the Moab Workload Manager Administrator Guide.

req_information

Available with Torque 6.0 and later. This uses a different syntax than the -l resource_list option.

Defines the NUMA-aware resource requests for NUMA hardware. This option will work with non-NUMA hardware.

See 5.590 -L NUMA Resource Request for the syntax and valid values.

Defines the set of conditions under which the execution server will send a mail message about the job. The mail_options argument is a string which consists of either the single character "n" or "p", or one or more of the characters "a", "b", "e", and "f".

If the character "n" is specified, no normal mail is sent. Mail for job cancels and other events outside of normal job processing are still sent.

If the character "p" is specified, mail will never be sent for the job.

For the characters"a", "b", "e" and "f":

• a – Mail is sent when the job is aborted by the batch system.
• b – Mail is sent when the job begins execution.
• e – Mail is sent when the job terminates.
• f – Mail is sent when the job terminates with a non-zero exit code.

If the -m option is not specified, mail will be sent if the job is aborted.

Declares the list of users to whom mail is sent by the execution server when it sends mail about the job.

The user_list argument is of the form:

user[@host][,user[@host],...]

If unset, the list defaults to the submitting user at the qsub host, i.e. the job owner.

Allows a user to specify an exclusive-node access/allocation request for the job. This will set node_exclusive = True in the output of qstat -f <job ID>.

For Moab, the following options are equivalent to "-n":

> qsub -l naccesspolicy=singlejob jobscript.sh

# OR

> qsub -W x=naccesspolicy:singlejob jobscript.sh

By default, this only applies for cpusets, and only for compatible schedulers (see 5.695 Linux Cpuset Support).

For systems that use Moab and have cgroups enabled, the recommended manner for assigning all cores is to use NUMA syntax: "-L tasks=<count>:lprocs=all:place=node".

With cgroups, the ("-l") syntax (lowercase L) will, by default, restrict to the number of cores requested, or to the resources_default.procs value (i.e., 1 core, typically). In order to override this behavior and have Moab assign all the cores on a node while using "-l...singlejob" and/or "-n" (in other words, without "-L ...lprocs=all..."), you must also set RMCFG[<torque>] FLAGS=MigrateAllJobAttributes in moab.cfg.

Declares a name for the job. The name specified may be an unlimited number of characters in length. It must consist of printable, nonwhite space characters with the first character alphabetic.

If the -N option is not specified, the job name will be the base name of the job script file specified on the command line. If no script file name was specified and the script was read from the standard input, then the job name will be set to STDIN.

Defines the path to be used for the standard output stream of the batch job. The path argument is of the form:

[hostname:]path_name

where hostname is the name of a host to which the file will be returned, and path_name is the path name on that host in the syntax recognized by POSIX.

When specifying a directory for the location you need to include a trailing slash.

The argument will be interpreted as follows:

• path_name – where path_name is not an absolute path name, then the qsub command will expand the path name relative to the current working directory of the command. The command will supply the name of the host upon which it is executing for the hostname component.
• hostname:path_name – where path_name is not an absolute path name, then the qsub command will not expand the path name relative to the current working directory of the command. On delivery of the standard output, the path name will be expanded relative to the user's home directory on the hostname system.
• path_name – where path_name specifies an absolute path name, then the qsub will supply the name of the host on which it is executing for the hostname.
• hostname:path_namewhere path_name specifies an absolute path name, the path will be used as specified.

If the -o option is not specified, the default file name for the standard output stream will be used. The default name has the following form:

• job_name.osequence_number – where job_name is the name of the job (see the -n name option) and sequence_number is the job number assigned when the job is submitted.

Defines the destination of the job. The destination names a queue, a server, or a queue at a server.

The qsub command will submit the script to the server defined by the destination argument. If the destination is a routing queue, the job may be routed by the server to a new destination.

If the -q option is not specified, the qsub command will submit the script to the default server. (See Environment variables and the PBS ERS section 2.7.4, "Default Server".)

If the -q option is specified, it is in one of the following three forms:

• queue
• @server
• queue@server

If the destination argument names a queue and does not name a server, the job will be submitted to the named queue at the default server.

If the destination argument names a server and does not name a queue, the job will be submitted to the default queue at the named server.

If the destination argument names both a queue and a server, the job will be submitted to the named queue at the named server.

Declares whether the job is rerunable (see the qrerun command). The option argument is a single character, either y or n.

If the argument is "y", the job is rerunable. If the argument is "n", the job is not rerunable. The default value is y, rerunable.

Declares the path to the desires shell for this job.

qsub script.sh -S /bin/tcsh

If the shell path is different on different compute nodes, use the following syntax:

path[@host][,path[@host],...]
qsub script.sh -S /bin/tcsh@node1,/usr/bin/tcsh@node2

Only one path may be specified for any host named. Only one path may be specified without the corresponding host name. The path selected will be the one with the host name that matched the name of the execution host. If no matching host is found, then the path specified without a host will be selected, if present.

If the -S option is not specified, the option argument is the null string, or no entry from the path_list is selected, the execution will use the user's login shell on the execution host.

Specifies the task ids of a job array. Single task arrays are allowed.

The array_request argument is an integer id or a range of integers. Multiple ids or id ranges can be combined in a comma delimited list. Examples: -t 1-100 or -t 1,10,50-100

An optional slot limit can be specified to limit the amount of jobs that can run concurrently in the job array. The default value is unlimited. The slot limit must be the last thing specified in the array_request and is delimited from the array by a percent sign (%).

This sets the slot limit to 5. Only 5 jobs from this array can run at the same time.

You can use qalter to modify slot limits on an array. The server parameter max_slot_limit can be used to set a global slot limit policy.

Defines the user name under which the job is to run on the execution system.

The user_list argument is of the form:

user[@host][,user[@host],...]

Only one user name may be given per specified host. Only one of the user specifications may be supplied without the corresponding host specification. That user name will used for execution on any host not named in the argument list. If unset, the user list defaults to the user who is running qsub.

Expands the list of environment variables that are exported to the job.

In addition to the variables described in the "Description" section above, variable_list names environment variables from the qsub command environment which are made available to the job when it executes. The variable_list is a comma separated list of strings of the form variable or variable=value. These variables and their values are passed to the job. Note that -v has a higher precedence than -V, so identically named variables specified via -v will provide the final value for an environment variable in the job.

Use "-W x=" as pass-through for scheduler-only job extensions. See Resource Manager Extensions in the Moab Workload ManagerAdministrator Guide for a list of scheduler-only job extensions.

For legacy purposes, qsub -l will continue to support some scheduler-only job extensions. However, when in doubt, use "-W x=".

The -W option allows for the specification of additional job attributes. The general syntax of -W is in the form:

-W attr_name=attr_value.

You can use multiple -W options with this syntax:

-W attr_name1=attr_value1 -W attr_name2=attr_value2.

If white space occurs anywhere within the option argument string or the equal sign, "=", occurs within an attribute_value string, then the string must be enclosed with either single or double quote marks.

PBS currently supports the following attributes within the -W option:

• depend=dependency_list – Defines the dependency between this and other jobs. The dependency_list is in the form:

type[:argument[:argument...][,type:argument...]

The argument is either a numeric count or a PBS job id according to type. If argument is a count, it must be greater than 0. If it is a job id and not fully specified in the form seq_number.server.name, it will be expanded according to the default server rules which apply to job IDs on most commands. If argument is null (the preceding colon need not be specified), the dependency of the corresponding type is cleared (unset). For more information, see depend=dependency_list valid dependencies.

• group_list=g_list – Defines the group name under which the job is to run on the execution system. The g_list argument is of the form:

group[@host][,group[@host],...]

Only one group name may be given per specified host. Only one of the group specifications may be supplied without the corresponding host specification. That group name will used for execution on any host not named in the argument list. If not set, the group_list defaults to the primary group of the user under which the job will be run.

• interactive=true – If the interactive attribute is specified, the job is an interactive job. The -I option is an alternative method of specifying this attribute.
• job_radix=<int> – To be used with parallel jobs. It directs the Mother Superior of the job to create a distribution radix of size <int> between sisters. See Managing Multi-Node Jobs.
• stagein=file_list
• stageout=file_list – Specifies which files are staged (copied) in before job start or staged out after the job completes execution. On completion of the job, all staged-in and staged-out files are removed from the execution system. The file_list is in the form:

local_file@hostname:remote_file[,...]

regardless of the direction of the copy. The name local_file is the name of the file on the system where the job executed. It may be an absolute path or relative to the home directory of the user. The name remote_file is the destination name on the host specified by hostname. The name may be absolute or relative to the user's home directory on the destination host. The use of wildcards in the file name is not recommended. The file names map to a remote copy program (rcp) call on the execution system in the follow manner:

• For stagein: rcp hostname:remote_file local_file
• For stageout: rcp local_file hostname:remote_file

Data staging examples:

-W stagein=/tmp/input.txt@headnode:/home/user/input.txt

-W stageout=/tmp/output.txt@headnode:/home/user/output.txt

If Torque has been compiled with wordexp support, then variables can be used in the specified paths. Currently only $PBS_JOBID, $HOME, and $TMPDIR are supported for stagein.

• umask=XXX – Sets umask used to create stdout and stderr spool files in pbs_mom spool directory. Values starting with 0 are treated as octal values, otherwise the value is treated as a decimal umask value.

By default, if you submit an interactive job with a script, the script will be parsed for PBS directives but the rest of the script will be ignored since it's an interactive job. The -x option allows the script to be executed in the interactive job and then the job completes. For example:

script.sh
#!/bin/bash
ls
---end script---
qsub -I script.sh
qsub: waiting for job 5.napali to start
dbeer@napali:#
<displays the contents of the directory, because of the ls command>
qsub: job 5.napali completed

When this job terminates execution, jobs jobid... may begin.

If any of the before forms are used, the jobs referenced by jobid must have been submitted with a dependency type of on.

If any of the before forms are used, the jobs referenced by jobid must have the same owner as the job being submitted. Otherwise, the dependency is ignored.

When this job terminates execution, jobs in arrayid may begin.

If any of the before forms are used, the jobs referenced by arrayid must have been submitted with a dependency type of on.

If any of the before forms are used, the jobs referenced by arrayid must have the same owner as the job being submitted. Otherwise, the dependency is ignored.