5.18 msub

5.18.1 Synopsis

msub [-a datetime][-A account][-c interval][-C directive_prefix][-d path] [-e path][-E][-F][-h][-I][-j join][-k keep][-K][-l resourcelist][-L NUMA_resourcelist][-m mailoptions] [-M user_list][-n node_exclusive][-N name][-o path][-p priority][-P <user>[:<group>]][-q destination][-r] [-S pathlist][-t jobarrays][-u userlist][-v variablelist][-V][-w <path>] [-W additionalattributes][-x][-z][--stagein][--stageout][--stageinfile][--stageoutfile][--stageinsize][--stageoutsize][--workflowjobids][script]

5.18.2 Overview

msub allows users to submit jobs directly to Moab. When a job is submitted directly to a resource manager (such as Torque), it is constrained to run on only those nodes that the resource manager is directly monitoring. In many instances, a site may be controlling multiple resource managers. When a job is submitted to Moab rather than to a specific resource manager, it is not constrained as to what nodes it is executed on. msub can accept command line arguments (with the same syntax as qsub), job scripts (in either PBS or LoadLeveler syntax), or the SSS Job XML specification.

Moab must run as a root user in order for msub submissions to work. Workload submitted via msub when Moab is running as a non-root user fail immediately.

Submitted jobs can then be viewed and controlled via the mjobctl command.

Flags specified in the following table are not necessarily supported by all resource managers.

5.18.3 Access

When Moab is configured to run as root, any user may submit jobs via msub.

5.18.4 Options

-a
Name	Eligible Date
Format	[[[[CC]YY]MM]DD]hhmm[.SS]
Description	Declares the time after which the job is eligible for execution.
Example	> msub -a 12041300 cmd.pbs Moab will not schedule the job until 1:00 pm on December 4, of the current year.

-A
Name	Account
Format	<ACCOUNT NAME>
Description	Defines the account associated with the job.
Example	> msub -A research cmd.pbs Moab will associate this job with account research.

-c
Name	Checkpoint Interval
Format	[n\|s\|c\|c=<minutes>]
Description	Checkpoint of the will occur at the specified interval. n — No Checkpoint is to be performed. s — Checkpointing is to be performed only when the server executing the job is shut down. c — Checkpoint is to be performed at the default minimum time for the server executing the job. c=<minutes> — Checkpoint is to be performed at an interval of minutes.
Example	> msub -c c=12 cmd.pbs The job will be checkpointed every 12 minutes.

-C
Name	Directive Prefix
Format	'<PREFIX NAME>'
Default	First known prefix (#PBS, #@, #BSUB, #!, #MOAB, #MSUB)
Description	Specifies which directive prefix should be used from a job script. It is best to submit with single quotes. '#PBS' An empty prefix will cause Moab to not search for any prefix. -C '' Command line arguments have precedence over script arguments. Custom prefixes can be used with the -C flag. -C '#MYPREFIX' Custom directive prefixes must use PBS syntax. If the -C flag is not given, Moab will take the first default prefix found. Once a directive is found, others are ignored.
Example	> msub -C '#MYPREFIX' cmd.pbs #MYPREFIX -l walltime=5:00:00 (in cmd.pbs) Moab will use the #MYPREFIX directive specified in cmd.pbs, setting the wallclock limit to five hours.

-d
Name	Initial Working Directory
Format	<path>
Default	Depends on the RM being used. If using Torque, the default is $HOME. If using Slurm, the default is the submission directory.
Description	Specifies which directory the job should execute in.
Example	> msub -d /home/test/job12 cmd.pbs The job will begin execution in the /home/test/job12 directory.

-e
Name	Error Path
Format	[<hostname>:]<path>
Default	$SUBMISSIONDIR/$JOBNAME.e$JOBID
Description	Defines the path to be used for the standard error stream of the batch job.
Example	> msub -e test12/stderr.txt The STDERR stream of the job will be placed in the relative (to execution) directory specified.

-E
Name	Environment Variables
Description	Moab adds the following variables, if populated, to the job's environment: MOAB_ACCOUNT — Account name. MOAB_BATCH — Set if a batch job (non-interactive). MOAB_CLASS — Class name. MOAB_DEPEND — Job dependency string. MOAB_GROUP — Group name. MOAB_JOBARRAYINDEX —For a job in an array, the index of the job. MOAB_JOBARRAYRANGE — For a system with job arrays, the range of all job arrays. MOAB_JOBID — Job ID. MOAB_JOBNAME — Job name. MOAB_MACHINE — Name of the machine (i.e. Destination RM) that the job is running on. MOAB_NODECOUNT — Number of nodes allocated to job. MOAB_NODELIST — Comma-separated list of nodes (listed singly with no ppn info). MOAB_PARTITION — Partition name the job is running in. MOAB_PROCCOUNT — Number of processors allocated to job. MOAB_QOS — QOS name. MOAB_SUBMITDIR — Directory from which the job was submitted. MOAB_TASKMAP — Node list with procs per node listed. <nodename>.<procs> MOAB_USER — User name. In Slurm environments, not all variables will be populated since the variables are added at submission (such as NODELIST). With Torque/PBS, the variables are added just before the job is started. This feature only works with Slurm and Torque/PBS.
Example:	> msub -E mySim.cmd The job mySim will be submitted with extra environment variables.

-E

Name

Environment Variables

Description

Moab adds the following variables, if populated, to the job's environment:

MOAB_ACCOUNT — Account name.
MOAB_BATCH — Set if a batch job (non-interactive).
MOAB_CLASS — Class name.
MOAB_DEPEND — Job dependency string.
MOAB_GROUP — Group name.
MOAB_JOBARRAYINDEX —For a job in an array, the index of the job.
MOAB_JOBARRAYRANGE — For a system with job arrays, the range of all job arrays.
MOAB_JOBID — Job ID.
MOAB_JOBNAME — Job name.
MOAB_MACHINE — Name of the machine (i.e. Destination RM) that the job is running on.
MOAB_NODECOUNT — Number of nodes allocated to job.
MOAB_NODELIST — Comma-separated list of nodes (listed singly with no ppn info).
MOAB_PARTITION — Partition name the job is running in.
MOAB_PROCCOUNT — Number of processors allocated to job.
MOAB_QOS — QOS name.
MOAB_SUBMITDIR — Directory from which the job was submitted.
MOAB_TASKMAP — Node list with procs per node listed. <nodename>.<procs>
MOAB_USER — User name.

In Slurm environments, not all variables will be populated since the variables are added at submission (such as NODELIST). With Torque/PBS, the variables are added just before the job is started.

This feature only works with Slurm and Torque/PBS.

Example:

> msub -E mySim.cmd

The job mySim will be submitted with extra environment variables.

-F
Name	Script Flags
Format	"<STRING>"
Description	Specifies the flags Torque will pass to the job script at execution time. The -F flag is only compatible with Torque resource managers.
Example	> msub -F "arg1 arg2" -1 nodes=1,walltime=60 files/job.sh Torque will pass parameters arg1 and arg2 to the job.sh script when the job executes.

-h
Name	Hold
Description	Specifies that a user hold be applied to the job at submission time.
Example	> msub -h cmd.ll The job will be submitted with a user hold on it.

-I
Name	Interactive
Description	Declares the job is to be run interactively. qsub must exist on the same host as msub if the interactive job is destined for a Torque cluster, because the interactive msub request will be converted to a qsub -I request.
Example	> msub -I job117.sh The job will be submitted in interactive mode.

-j
Name	Join
Format	[eo\|oe\|n]
Default	n (not merged)
Description	If eo is specified, the error and output streams are merged into the error stream. If oe is specified, the error and output streams will be merged into the output stream. If using either the -e or the -o option and the -j eo\|oe option, the -j option takes precedence and all standard error and output messages go to the chosen output file.
Example	> msub -j oe cmd.sh STDOUT and STDERR will be merged into one file.

-k
Name	Keep
Format	[e\|o\|eo\|oe\|n]
Default	n (not retained)
Description	Defines which (if either) of output and error streams will be retained on the execution host (overrides path for stream).
Example	> msub -k oe myjob.sh STDOUT and STDERR for the job will be retained on the execution host.

-K
Name	Continue Running
Format	N/A
Description	Tells the client to continue running until the submitted job is completed. The client will query the status of the job every 5 seconds. The time interval between queries can be specified or disabled via MSUBQUERYINTERVAL. Use the -K option sparingly (if at all) as it slows down the Moab scheduler with frequent queries. Running ten jobs with the -K option creates an additional fifty queries per minute for the scheduler.
Example	> msub -K newjob.sh 3 Job 3 completed* *Only shows up after job completion.

-l
Name	Resource List
Format	<STRING> -l [BANDWIDTH\|DDISK\|DEADLINE\|DEPEND\|DMEM\|EXCLUDENODES\|FEATURE...\|] Additional options can be referenced on the resource manager extensions page.
Description	Defines the resources that are required by the job and establishes a limit to the amount of resource that can be consumed. Resources native to the resource manager, scheduler resource manager extensions, or job flags may be specified. Note that resource lists are dependent on the resource manager in use. For information on specifying multiple types of resources for allocation, see Multi-Req Support. Moab does not support the combination of msub -l excludenodes and ENABLEHIGHTHROUGHPUT TRUE.
Example	> msub -l nodes=32:ppn=2,pmem=1800mb,walltime=3600,VAR=testvar:myvalue cmd.sh > msub -l nodes=32:ppn=2,pmem=1800mb,walltime=3600,VAR=testvar: myvalue cmd.sh The job requires 32 nodes with 2 processors each, 1800 MB per task, a walltime of 3600 seconds, and a variable named testvar with a value of myvalue. If JOBNODEMATCHPOLICY is not set, Moab does not reserve the requested number of processors on the requested number of nodes. It reserves the total number of requested processors (nodes x ppn) on any number of nodes. Rather than setting nodes=<value>:ppn=<value>, set procs=<value>, replacing <value> with the total number of processors the job requires. Note that JOBNODEMATCHPOLICY is not set by default. > msub -l nodes=32:ppn=2 -l advres=!<resvid> This entry would tell Moab to only consider resources other than the specified <reservation id>.

-L
Name	NUMA req_information
Description	Available with Moab 9.0 or later with Torque 6.0 or later. This uses a different syntax than the -l resource_list option. Defines the NUMA-aware resource requests for NUMA hardware. This option will work with non-NUMA hardware. See 5.218 -L NUMA Resource Request for the syntax and valid values.

-L

Name

NUMA req_information

Description

Available with Moab 9.0 or later with Torque 6.0 or later. This uses a different syntax than the -l resource_list option.

Defines the NUMA-aware resource requests for NUMA hardware. This option will work with non-NUMA hardware.

See 5.218 -L NUMA Resource Request for the syntax and valid values.

-m
Name	Mail Options
Format	<STRING> (either n or one or more of the characters a, b, and e)
Description	Defines the set of conditions (abort,begin,end) when the server will send a mail message about the job to the user.
Example	> msub -m be cmd.sh Mail notifications will be sent when the job begins and ends.

-M
Name	Mail List
Format	<user>[@<host>][,<user>[@<host>],...]
Default	$JOBOWNER
Description	Specifies the list of users to whom mail is sent by the execution server. Overrides the EMAILADDRESS specified on the USERCFG credential.
Example	> msub -M jon@node01,bill@node01,jill@node02 cmd.sh Mail will be sent to the specified users if the job is aborted.

-n
Name	Node Exclusive
Description	Allows a user to specify an exclusive-node access/allocation request for the job. See SINGLEJOB for more information.
Example	> msub -n job1187.sh Job will have exclusive access to each node on which it runs.

-N
Name	Name
Format	<STRING>
Default	STDIN or name of job script
Description	Specifies the user-specified job name attribute.
Example	> msub -N chemjob3 cmd.sh Job will be associated with the name chemjob3.

-o
Name	Output Path
Format	[<hostname>:]<path> - %J and %I are acceptable variables. %J is the master array name and %I is the array member index in the array.
Default	$SUBMISSIONDIR/$JOBNAME.o$JOBID
Description:	Defines the path to be used for the standard output stream of the batch job. More variables are allowed when they are used in the job script instead of msub -o. In the job script, specify a #PBS -o line and input your desired variables. The allowable variables are: OID OTYPE USER OWNER JOBID JOBNAME Submitting a job script that has the line #PBS -o $(USER)_$(JOBID)_$(JOBNAME).txt results in a file called <username>_<jobID>_<jobName>.txt. Do not use msub -o when submitting a job script that has a #PBS -o line defined.
Example	> msub -o test12/stdout.txt The STDOUT stream of the job will be placed in the relative (to execution) directory specified. > msub -t 1-2 -o /home/jsmith/simulations/%J-%I.out ~/sim5.sh A job array is submitted and the name of the output files includes the master array index and the array member index.

-p
Name	Priority
Format	<INTEGER> (between -1024 and 0)
Default	0
Description	Defines the priority of the job. To enable priority range from -1024 to +1023, see ENABLEPOSUSERPRIORITY.
Example	> msub -p 25 cmd.sh The job will have a user priority of 25.

-P
Name	Proxy User
Format	<user>[:<group>]
Description	Allows a root user or manager to submit a job as another user. Moab treats proxy jobs as though the jobs were submitted by the supplied username. This option can only be used by users in the ADMINCFG[1] security level.
Example	msub -P user1 cmd.pbs

-q
Name	Destination Queue (Class)
Format	[<queue>][@<server>]
Default	[<DEFAULT>]
Description	Defines the destination of the job. If no destination queue is specified and the environment variable MOAB_DEFAULTQUEUE is present, msub will use the environment variable when submitting the job.
Example	> msub -q priority cmd.sh The job will be submitted to the priority queue.

-r
Name	Rerunable
Format	[y\|n]
Default	n
Description:	Declares whether the job is rerunable.
Example	> msub -r n cmd.sh The job cannot be rerun. The default for qsub -r is 'y' (yes), which is opposite from msub -r. For better clarity, use the following instead. msub -l [flags\|jobflags]=restartable]

-S
Name	Shell
Format	<path>[@<host>][,<path>[@<host>],...]
Default	$SHELL
Description	Declares the shell that interprets the job script.
Example	> msub -S /bin/bash The job script will be interpreted by the /bin/bash shell.

-t
Name	Job Arrays
Format	<name>[<indexlist>]%<limit>
Description	Starts a job array with the jobs in the index list. The limit variable specifies how many jobs may run at a time. For more information, see Submitting Job Arrays. Moab enforces an internal limit of 100,000 subjobs that a single array job submission can specify.
Example	> msub -t myarray[1-1000]%4

-u
Name	User List
Format	<user>[@<host>[,<user>[@<host>],...]
Default	UID of msub command
Description	Defines the user name under which the job is to run on the execution system.
Example	> msub -u bill@node01 cmd.sh On node01 the job will run under Bill's UID, if permitted.

-v
Name	Variable List
Format	<string>[,<string>,...]
Description	Expands the list the environment variables that are exported to the job (taken from the msub command environment).
Example	> msub -v DEBUG cmd.sh The DEBUG environment variable will be defined for the job.

-V
Name	All Variables
Description	Declares that all environment variables in the msub environment are exported to the batch job
Example	> msub -V cmd.sh All environment variables will be exported to the job.

-w
Name	Working Directory
Format	<path>
Description	Defines the working directory path to be used for the job. If the -w option is not specified, the default working directory is the current directory. This option sets the environment variable PBS_O_WORKDIR.
Example	> msub -l -w /tmp

-W
Name	Additional Attributes
Format	<string>
Description	Allows for specification of additional job attributes (See Resource Manager Extension)
Example	> msub -W x=GRES:matlab:1 cmd.sh The job requires one resource of matlab. This flag can be used to set a filter for what namespaces will be passed from a job to a trigger using a comma-delimited list. This limits the trigger's action to objects contained in certain workflows. For more information, see Requesting Name Space Variables. > msub -W x="trigns=vc1,vc2" The job passes namespaces vc1 and vc2 to triggers.

-x
Format	<script> or <command>
Description	When running an interactive job, the -x flag makes it so that the corresponding script won't be parsed for PBS directives, but is instead a command that is launched once the interactive job has started. The job terminates at the completion of this command. This option works only when using Torque. The -x option for msub differs from qsub in that qsub does not require the script name to come directly after the flag. The msub command requires a script or command immediately after the -x declaration.
Example	> msub -I -x ./script.pl > msub -I -x /tmp/command

-z
Name	Silent Mode
Description	The job's identifier will not be printed to stdout upon submission.
Example	> msub -z cmd.sh No job identifier will be printout the stdout upon successful submission.

5.18.4.A Staging data

Data staging, or the ability to copy data required for a job from one location to another or to copy resulting data to a new location (See About Data Staging for more information), must be specified at job submission. To stage data in, you would use the msub --stagein and/or --stageinfile option, optionally with --stageinsize. You would use similar options the same way for staging out: --stageout, --stageoutfile, and --stageoutsize. --stagein and --stageout, which you can use multiple times in the same msub command, allow you to specify a single file or directory to stage in or out. --stageinfile and --stageoutfile allow you to specify a text file that lists the files to stage in or out. The --stageinsize and [--stageoutsize] options allow you to estimate the total size of the files and directories that you want to stage in or out, which can help Moab make an intelligent guess about how long it will take to stage the data in or out, thus ensuring that the job can start as soon as possible after the staging has occurred.

Staging a file or directory

The --stagein and --stageout options use the same format.

--<stagein|stageout><=| ><source>%<destination>

Where <source> and <destination> take on the following format:

[<user>@]<host>:/<path>[/<fileName>]

Specifying a user and file name are optional. If you do not specify a file name, Moab will assume a directory.

> msub ... --stagein=student@biology:/stats/file001%admin@moab:/tmp/staging <jobScript>

This msub commands tells Moab that the job requires file001 from student's stats directory on the biology server to be staged to admin's staging directory on the moab server prior to the job's starting.

You can specify the option multiple times for the same msub command; however, staging large number of files is easier with --stageinfile or --stageoutfile.

You can also use #MSUB or #PBS within a job script to specify data staging options. For example:

#MSUB --stageinsize=1gb

#MSUB --stagein=...

See Sample User Job Script for more information. Note that the data staging options are not compatible with qsub.

Staging multiple files or directories

The --stageinfile and --stageoutfile options use the same format. You must include the path to a text file that lists each file to stage in or out on its own line. Each file specification follows the same format as a --stagein or --stageout specification as described above. The format of the command options looks like this:

--<stageinfile|stageoutfile><=| ><path>/<fileName>

The file contains multiple lines with the following format:

[<user>@]<host>:/<path>[/<fileName>]%[<user>@]<host>:/<path>[/<fileName>]

...

Moab ignores blank lines in the file. You can comment out lines by preceding them with a pound sign (#). The following examples demonstrate what the --stageinfile option looks like on the command line and what the file it specifies might look like.

> msub ... --stageinfile=/tmp/myStagingFile <jobScript>

/tmp/myStagingFile:

student@biology:/stats/file001%moab:/tmp/staging

student@biology:/stats/file002%moab:/tmp/staging

student@biology:/stats/file003%moab:/tmp/staging

#student@biology:/stats/file004%moab:/tmp/staging

student@biology:/stats/file005%moab:/tmp/staging

student@biology:/stats/file006%moab:/tmp/staging

student@biology:/stats/file007%moab:/tmp/staging

student@biology:/stats/file008%moab:/tmp/staging

student@biology:/stats/file009%moab:/tmp/staging

student@biology:/stats/file010%moab:/tmp/staging

Moab stages in each file listed in myStagingFile to the /tmp/staging directory. Each file resides on the biology host as the student user. Moab ignores the blank line and the line specifying file004.

Stage in or out file size

The optional --stageinsize and --stageoutsize options give you the opportunity to estimate the size of the file(s) or directory(-ies) being staged to aid Moab in choosing an appropriate start time. Both options use the same format:

--<stageinsize|stageoutsize>=<integer>[unit]

The integer indicates the size of the file(s) and directory(-ies) in megabytes unless you specify a different unit. Moab accepts the follow case-insensitive suffixes: KB, MB, GB, or TB.

> msub --stageinfile=/stats/file003 --stageinsize=100 <jobScript>

Moab copies the /davidharris/research/recordlist file, which is approximately 100 megabytes, from the biology node to the host where the job will run prior to job start.

> msub --stageinfile=/stats/file002 --stageinsize=1gb <jobScript>

Moab copies all files specified in the /davidharris/research/recordlist file, which add up to approximately 1 gigabyte, to the host where the job will run prior to job start.

5.18.4.B Return all the job IDs in the workflow at submission time

By default, msub will print the job ID to stdout at the time of submission. If you want msub to print all of the jobs that are created as part of the workflow template, you can use the msub --workflowjobids option to show all the job IDs at submission time:

$ echo sleep 60 | msub -l walltime=15 --workflowjobids

MoabA.3.dsin MoabA.3 MoabA.3.dsout

5.18.4.C Job Script

The msub command supports job scripts written in any one of the following languages:

Language	Notes
PBS/Torque Job Submission Language	---
SSS XML Job Object Specification	---

5.18.4.D Low Latency

The msub can be configured to return a job ID very quickly by eliminating the processing of some job attributes, filters, remap classes, job arrays, templates, workflows, limits and other information when a job is submitted. This can be done globally by configuring DISPLAYFLAGS USENOBLOCKMSUB or on the individual job submission by appending "--noblock" to the command line.

It is recommended that when using a non-blocking msub that JOBIDFORMAT be configured (and PROXYJOBSUBMISSION if desired).

5.18.5 /etc/msubrc

Sites that wish to automatically add parameters to every job submission can populate the file /etc/msubrc with global parameters that every job submission will inherit.

For example, if a site wished every job to request a particular generic resource they could use the following /etc/msubrc:

-W x=GRES:matlab:2

5.18.6 Usage Notes

msub is designed to be as flexible as possible, allowing users accustomed to PBS, LSF, or LoadLeveler syntax, to continue submitting jobs as they normally would. It is not recommended that different styles be mixed together in the same msub command.

When only one resource manager is configured inside of Moab, all jobs are immediately staged to the only resource manager available. However, when multiple resource managers are configured Moab will determine which resource manager can run the job soonest. Once this has been determined, Moab will stage the job to the resource manager.

It is possible to have Moab take a best effort approach at submission time using the forward flag. When this flag is specified, Moab will do a quick check and make an intelligent guess as to which resource manager can run the job soonest and then immediately stage the job.

Moab can be configured to instantly stage a job to the underlying resource manager (like Torque/LOADLEVELER) through the parameter INSTANTSTAGE. When set inside moab.cfg, Moab will migrate the job instantly to an appropriate resource manager. Once migrated, Moab will destroy all knowledge of the job and refresh itself based on the information given to it from the underlying resource manager.

In most instances Moab can determine what syntax style the job belongs to (PBS or LoadLeveler); if Moab is unable to make a guess, it will default the style to whatever resource manager was configured at compile time. If LoadLeveler and PBS were both compiled then LoadLeveler takes precedence.

Moab can translate a subset of job attributes from one syntax to another. It is therefore possible to submit a PBS style job to a LoadLeveler resource manager, and vice versa, though not all job attributes will be translated.

5.18.7 Examples

Example 5-4:

> msub -l nodes=3:ppn=2,walltime=1:00:00,pmem=100kb script2.pbs.cmd
4364.orion

Example 5-5:

This example is the XML-formatted version of the above example. See Submitting Jobs via msub in XML for more information.

<job>
  <InitialWorkingDirectory>/home/user/test/perlAPI
  </InitialWorkingDirectory>
  <Executable>/home/user/test/perlAPI/script2.pbs.cmd
  </Executable>
  <SubmitLanguage>PBS</SubmitLanguage>
  <Requested>
    <Feature>ppn2</Feature>
    <Processors>3</Processors>
    <WallclockDuration>3600</WallclockDuration>
  </Requested>
</job>