Moab Workload Manager

13.2 Resource Manager Configuration

13.2.1 Defining and Configuring Resource Manager Interfaces

Moab resource manager interfaces are defined using the RMCFG parameter. This parameter allows specification of key aspects of the interface. In most cases, only the TYPE attribute needs to be specified and Moab determines the needed defaults required to activate and use the selected interface. In the following example, an interface to a Loadleveler resource manager is defined.

RMCFG[orion] TYPE=LL
...

Note that the resource manager is given a label of orion. This label can be any arbitrary site-selected string and is for local usage only. For sites with multiple active resource managers, the labels can be used to distinguish between them for resource manager specific queries and commands.

13.2.1.1 Resource Manager Attributes

The following table lists the possible resource manager attributes that can be configured.

ADMINEXEC
"jobsubmit"
NONE
Normally, when the JOBSUBMITURL is executed, Moab will drop to the UID and GID of the user submitting the job. Specifying an ADMINEXEC of jobsubmit causes Moab to use its own UID and GID instead (usually root). This is useful for some native resource managers where the JOBSUBMITURL is not a user command (such as qsub) but a script that interfaces directly with the resource manager.
RMCFG[base] ADMINEXEC=jobsubmit
Moab will not use the user's UID and GID for executing the JOBSUBMITURL.
   
AUTHTYPE
one of CHECKSUM, OTHER, PKI, SECUREPORT, or NONE.
CHECKSUM
Specifies the security protocol to be used in scheduler-resource manager communication.
Note Only valid with WIKI based interfaces.
RMCFG[base] AUTHTYPE=CHECKSUM

Moab requires a secret key based checksum associated with each resource manager message.

   
BANDWIDTH
<FLOAT>[{M|G|T}]
-1 (unlimited)
Specifies the maximum deliverable bandwidth between the Moab server and the resource manager for staging jobs and data. Bandwidth is specified in units per second and defaults to a unit of MB/s. If a unit modifier is specified, the value is interpreted accordingly (M - megabytes/sec, G - gigabytes/sec, T - terabytes/sec).
RMCFG[base] BANDWIDTH=340G
Moab will reserve up to 340 GB of network bandwidth when scheduling job and data staging operations to and from this resource manager.
   
CHECKPOINTSIG
one of suspend, <INTEGER>, or SIG<X>
---
Specifies what signal to send the resource manager when a job is checkpointed. (See Checkpoint Overview.)
RMCFG[base] CHECKPOINTSIG=SIGKILL

Moab routes the signal SIGKILL through the resource manager to the job when a job is checkpointed.

   
CHECKPOINTTIMEOUT
[[[DD:]HH:]MM:]SS
0 (no timeout)
Specifies how long Moab waits for a job to checkpoint before canceling it. If set to 0, Moab does not cancel the job if it fails to checkpoint. (See Checkpoint Overview.)
RMCFG[base] CHECKPOINTTIMEOUT=5:00

Moab cancels any job that has not exited 5 minutes after receiving a checkpoint request.

   
CLIENT
<PEER>
use name of resource manager for peer client lookup
If specified, the resource manager will use the peer value to authenticate remote connections. (See configuring peers). If not specified, the resource manager will search for a CLIENTCFG entry of RM:<RMNAME> in the moab-private.cfg file.
RMCFG[clusterBI] CLIENT=clusterB

Moab will look up and use information for peer clusterB when authenticating the clusterBI resource manager.

   
CLUSTERQUERYURL
[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
---
Specifies how Moab queries the resource manager. (See Native RM, URL Notes, and interface details.)
RMCFG[base] CLUSTERQUERYURL=file:///tmp/cluster.config
Moab reads /tmp/cluster.config when it queries base resource manager.
   
CONFIGFILE
<STRING>
N/A
Specifies the resource manager specific configuration file that must be used to enable correct API communication.
Note Only valid with LL- and SLURM-based interfaces.
RMCFG[base] TYPE=LL CONFIGFILE=/home/loadl/loadl_config

The scheduler uses the specified file when establishing the resource manager/scheduler interface connection.

   
DATARM
<RM NAME>
N/A
If specified, the resource manager uses the given storage resource manager to handle staging data in and out.
RMCFG[clusterB] DATARM=clusterB_storage

When data staging is required by jobs starting/completing on clusterB, Moab uses the storage interface defined by clusterB_storage to stage and monitor the data.

   
DEFAULTCLASS
<STRING>
N/A
Specifies the class to use if jobs submitted via this resource manager interface do not have an associated class.
RMCFG[internal] DEFAULTCLASS=batch
Moab assigns the class batch to all jobs from the resource manager internal that do not have a class assigned.
Note If you are using PBS as the resource manager, a job will never come from PBS without a class, and the default will never apply.
   
DEFAULT.JOB
<STRING>
N/A
Specifies the job template to use to set various job attributes that are not specified by the submitter.
RMCFG[base] DEFAULT.JOB=defjob
Moab uses the defjob job template to identify and apply job attribute defaults.
   
DEFAULTHIGHSPEEDADAPTER
<STRING>
sn0
Specifies the default high speed switch adapter to use when starting LoadLeveler jobs (supported in version 4.2.2 and higher of Moab and 3.2 of LoadLeveler).
RMCFG[base]     DEFAULTHIGHSPEEDADAPTER=sn1
The scheduler will start jobs requesting a high speed adapter on sn1.
   
DESCRIPTION
<STRING>
N/A
Specifies the human-readable description for the resource manager interface. If white space is used, the description should be quoted.
RMCFG[ganglia] TYPE=NATIVE:ganglia DESCRIPTION='resource monitor providing extended resource utilization stats'
Moab annotates the ganglia resource manager accordingly.
   
ENV
Semi-colon delimited (;) list of <KEY>=<VALUE> pairs
MOABHOMEDIR=<MOABHOMEDIR>
Specifies a list of environment variables that will be passed to URL's of type 'exec://' for that resource manager.
RMCFG[base] ENV=HOST=node001;RETRYTIME=50
RMCFG[base] CLUSTERQUERYURL=exec:///opt/moab/tools/cluster.query.pl
RMCFG[base] WORKLOADQUERYURL=exec:///opt/moab/tools/workload.query.pl

The environment variables HOST and RETRYTIME (with values 'node001' and '50' respectively) are passed to the /opt/moab/tools/cluster.query.pl and /opt/moab/tools/workload.query.pl when they are executed.

   
EPORT
<INTEGER>
N/A
Specifies the event port to use to receive resource manager based scheduling events.
RMCFG[base] EPORT=15017
The scheduler will look for scheduling events from the resource manager host at port 15017.
   
FAILTIME
<[[[DD:]HH:]MM:]SS
N/A
Specifies how long a resource manager must be down before any failure triggers associated with the resource manager fire.
RMCFG[base] FAILTIME=3:00
If the base resource manager is down for three minutes, any resource manager failure triggers fire.
   
FLAGS
comma delimited list of zero or more of the following: asyncstart, autostart, autosync, client, fullcp, executionServer, grid, hostingCenter, ignqueuestate, private, pushslavejobupdates, report, shared, slavepeer or static
N/A
Specifies various attributes of the resource manager. See Flag Details for more information.
RMCFG[base] FLAGS=static,slavepeer
Moab uses this resource manager to perform a single update of node and job objects reported elsewhere.
   
FLOWINTERVAL
[[[DD:]HH:]MM:]SS
01:00:00 (one hour)
Specifies the duration of the flow control sliding window.
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FLOWLIMIT
<INTEGER>
1
Specifies the limit of flow metric consumption allowed within the sliding window.
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FLOWMETRIC
one of jobs, procs, nodes, procseconds, or peseconds
jobs
Specifies the metric of consumption of the flow control sliding window.
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FNLIST
comma delimited list of zero or more of the following: clusterquery, jobcancel, jobrequeue, jobresume, jobstart, jobsuspend, queuequery, resourcequery or workloadquery
N/A
By default, a resource manager utilizes all functions supported to query and control batch objects. If this parameter is specified, only the listed functions are used.
RMCFG[base] FNLIST=queuequery
Moab only uses this resource manager interface to load queue configuration information.
   
HOST
<STRING>
localhost
The host name of the machine on which the resource manager server is running.
RMCFG[base] host=server1
   
IGNHNODES
<BOOLEAN>
FALSE
Specifies whether to read in the PBSPro host nodes. This parameter is used in conjunction with USEVNODES. When both are set to TRUE, the host nodes are not queried.
RMCFG[pbs] IGNHNODES=TRUE
   
JOBCANCELURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab cancels jobs via the resource manager. (See URL Notes below.)
RMCFG[base] JOBCANCELURL=exec:///opt/moab/job.cancel.lsf.pl
Moab executes /opt/moab/job.cancel.lsf.pl to cancel specific jobs.
   
JOBEXTENDDURATION
[[[DD:]HH:]MM:]SS[,[[[DD:]HH:]MM:]SS][!][<] (or <MIN TIME>[,<MAX TIME>][!])
---
Specifies the minimum and maximum amount of time that can be added to a job's walltime if it is possible for the job to be extended. (See MINWCLIMIT.) As the job runs longer than its current specified minimum wallclock limit (-l minwclimit, for example), Moab attempts to extend the job's limit by the minimum JOBEXTENDDURATION. This continues until either the extension can no longer occur (it is blocked by a reservation or job), the maximum JOBEXTENDDURATION is reached, or the user's specified wallclock limit (-l wallclock) is reached. When a job is extended, it is marked as PREEMPTIBLE, unless the '!' is appended to the end of the configuration string. If the '<' is at the end of the string, however, the job is extended the maximum amount possible.
RMCFG[base] JOBEXTENDDURATION=30,1:00:00
Moab extends a job's walltime by 30 seconds each time the job is about to run out of walltime until it is bound by one hour, a reservation/job, or the job's original "maximum" wallclock limit.
   
JOBMODIFYURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab modifies jobs via the resource manager. (See URL Notes and interface details.)
RMCFG[base] JOBMODIFYURL=exec://$TOOLSDIR/job.modify.dyn.pl
Moab executes /opt/moab/job.modify.dyn.pl to modify specific jobs.
JOBRSVRECREATE
Boolean
TRUE
Specifies whether Moab will re-create a job reservation each time job information is updated by a resource manager. (See Considerations for Large Clusters for more information.)
RMCFG[base] JOBRSVRECREATE=FALSE

Moab only creates a job reservation once when the job first starts.

   
JOBSTARTURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab starts jobs via the resource manager. (See URL Notes below.)
RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi

Moab triggers the jobstart.cgi script via http to start specific jobs.

   
JOBSUBMITURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab submits jobs to the resource manager. (See URL Notes below.)
RMCFG[base] JOBSUBMITURL=exec://$TOOLSDIR/job.submit.dyn.pl

Moab submits jobs directly to the database located on host dbserver.flc.com.

   
JOBSUSPENDURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab suspends jobs via the resource manager. (See URL Notes below.)
RMCFG[base] JOBSUSPENDURL=EXEC://$HOME/scripts/job.suspend
Moab executes the job.suspend script when jobs are suspended.
   
JOBVALIDATEURL
<protocol>://[<host>[:<port>]][<path>]
---
Specifies how Moab validates newly submitted jobs. (See URL Notes below.) If the script returns with a non-zero exit code, the job is rejected. (See User Proxying/Alternate Credentials.)
RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl

Moab executes the 'job.validate.pl' script when jobs are submitted to verify they are acceptable.

   
MAXDSOP
<INTEGER>
-1 (unlimited)
Specifies the maximum number of data staging operations that may be simultaneously active.
RMCFG[ds] MAXDSOP=16
   
MAX.JOB
<STRING>
---
Specifies the job template to use to check various maximum/excluded job attributes that are specified by the submitter.
RMCFG[base] MAX.JOB=maxjob

Moab will use the maxjob job template to identify and enforce maximum/excluded job attributes.

   
MAXJOBPERMINUTE
<INTEGER>
-1 (unlimited)
Specifies the maximum number of jobs allowed to start per minute via the resource manager.
RMCFG[base] MAXJOBPERMINUTE=5
The scheduler only allows five jobs per minute to launch via the resource manager base.
   
MAXJOBS
<INTEGER>
0 (limited only by the Moab MAXJOB setting)
Specifies the maximum number of active jobs that this interface is allowed to load from the resource manager.
Note Only works with Moab peer resource managers at this time.
RMCFG[cluster1] SERVER=moab://cluster1
MAXJOBS=200
The scheduler loads up to 200 active jobs from the remote Moab peer cluster1.
   
MINETIME
<INTEGER>
1
Specifies the minimum time in seconds between processing subsequent scheduling events.
RMCFG[base] MINETIME=5
The scheduler batch-processes scheduling events that occur less than five seconds apart.
   
MIN.JOB
<STRING>
---
Specifies the job template to use to check various minimum/required job attributes that are specified by the submitter.
RMCFG[base] MIN.JOB=minjob

Moab uses the minjob job template to identify and enforce minimum/required job attributes.

   
NMPORT
<INTEGER>
(any valid port number)
Allows specification of the resource manager's node manager port and is only required when this port has been set to a non-default value.
RMCFG[base] NMPORT=13001

The scheduler contacts the node manager located on each compute node at port 13001.

   
NODEFAILURERSVPROFILE
<STRING>
N/A
Specifies the rsv template to use when placing a reservation onto failed nodes. (See also NODEFAILURERESERVETIME.)
moab.cfg
RMCFG[base] NODEFAILURERSVPROFILE=long

RSVPROFILE[long]        DURATION=25:00
RSVPROFILE[long]        USERLIST=john

The scheduler will use the long rsv profile when creating reservations over failed nodes belonging to base.

   
NODESTATEPOLICY
one of OPTIMISTIC or PESSIMISTIC
PESSIMISTIC
Specifies how Moab should determine the state of a node when multiple resource managers are reporting state.
OPTIMISTIC specifies that if any resource manager reports a state of up, that state will be used.
PESSIMISTIC specifies that if any resource manager reports a state of down, that state will be used.
moab.cfg
RMCFG[ganglia] TYPE=NATIVE CLUSTERQUERYURL=ganglia://
RMCFG[ganglia] FLAGS=SLAVEPEER NODESTATEPOLICY=OPTIMISTIC
   
OMAP
<protocol>://[<host>[:<port>]][<path>]
---
Specifies an object map file that is used to map credentials and other objects when using this resource manager peer. (See Grid Credential Management for full details.)
moab.cfg
RMCFG[peer1] OMAP=file:///opt/moab/omap.dat
When communicating with the resource manager peer1, objects are mapped according to the rules defined in the /opt/moab/omap.dat file.
   
POLLINTERVAL
[[[DD:]HH:]MM:]SS
30
Specifies how often the scheduler will poll the resource manager for information.
RMCFG[base] POLLINTERVAL=1:00
Moab contacts resource manager base every minute for updates.
   
POLLTIMEISRIGID
<BOOLEAN>
FALSE
Determines whether the POLLINTERVAL parameter is interpreted as an interval or a set time for contacting.
RMCFG[base] POLLTIMEISRIGID=TRUE POLLINTERVAL=1:00:00
Moab polls the resource manager at startup and on the hour.
   
PORT
<INTEGER>
0
Specifies the port on which the scheduler should contact the associated resource manager. The value '0' specifies that the resource manager default port should be used.
RMCFG[base] TYPE=PBS HOST=cws PORT=20001
Moab attempts to contact the PBS server daemon on host cws, port 20001.
   
PTYSTRING
<STRING>
srun -n1 -N1 --pty

When a SLURM interactive job is submitted, it builds an salloc command that gets the requested resources and an srun command that creates a terminal session on one of the nodes. The srun command is called the PTYString. PTYString is configured in moab.cfg.

There are two special things you can do with PTYString:

  1. You can have PTYSTRING=$salloc which says to use the default salloc command (SallocDefaultCommand, look in the slurm.conf man page) defined in slurm.conf. Internally, Moab won't add a PTYString because SLURM will call the SallocDefaultCommand.
  2. As in the example below, you can add $SHELL. $SHELL will be expanded to either what you request on the command line (such as msub -S /bin/tcsh -l) or to the value of $SHELL in your current session.

PTYString works only with SLURM.

RMCFG[slurm] PTYSTRING="srun -n1 -N1 --pty --preserve-env $SHELL"
   
RESOURCECREATEURL
[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
RESOURCECREATEURL has a default only when the RM type is declared as MSM (RMCFG[] TYPE=Native:MSM). If this is the case, the default is: exec://$TOOLSDIR/msm/contrib/node.create.pl
Specifies a script or method that can be used by Moab to create resources dynamically, such as creating a virtual machine on a hypervisor.
RMCFG[base] RESOURCECREATEURL=exec:///opt/script/vm.provision.py

Moab invokes the vm.provision.py script, passing in data as command line arguments, to request a creation of new resources.
   
RESOURCETYPE
{COMPUTE|FS|LICENSE|NETWORK}
---
Specifies which type of resource this resource manager is configured to control. See Native Resource Managers for more information.
RMCFG[base] TYPE=NATIVE RESOURCETYPE=FS

Resource manager base will function as a NATIVE resource manager and control file systems.
   
RMSTARTURL
[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
---
Specifies how Moab starts the resource manager.
RMCFG[base] RMSTARTURL=exec:///tmp/nat.start.pl
Moab executes /tmp/nat.start.pl to start the resource manager base.
   
RMSTOPURL
[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
---
Specifies how Moab stops the resource manager.
RMCFG[base] RMSTOPURL=exec:///tmp/nat.stop.pl
Moab executes /tmp/nat.stop.pl to stop the resource manager base.
   
SBINDIR
<PATH>
N/A
For use with TORQUE; specifies the location of the TORQUE system binaries (supported in TORQUE 1.2.0p4 and higher).
RMCFG[base] TYPE=pbs  SBINDIR=/usr/local/torque/sbin
Moab tells TORQUE that its system binaries are located in /usr/local/torque/sbin.
   
SERVER
<URL>
N/A
Specifies the resource management service to use. If not specified, the scheduler locates the resource manager via built-in defaults or, if available, with an information service.
RMCFG[base] server=ll://supercluster.org:9705
Moab attempts to use the Loadleveler scheduling API at the specified location.
   
SET.JOB
<STRING>
N/A
Specifies the job template to use to force various job attributes regardless of whether or not they are specified by the submitter.
RMCFG[internal] SET.JOB=test
RMCFG[pbs] SET.JOB=test
JOBCFG[test] CLASS=class1
Moab uses the test job template to identify and enforce mandatory job attributes.
   
SLURMFLAGS
<STRING>
N/A
Specifies characteristics of the SLURM resource manager interface.

RMCFG[slurm] SLURMFLAGS=COMPRESSOUTPUT
Moab uses the specified flag to determine interface characteristics with SLURM. The COMPRESSOUTPUT flag instructs Moab to use the compact hostlist format for job submissions to SLURM. The flag NODEDELTAQUERY instructs Moab to request delta node updates when it queries SLURM for node configuration.
   
SOFTTERMSIG
<INTEGER> or SIG<X>
---
Specifies what signal to send the resource manager when a job reaches its soft wallclock limit. (See JOBMAXOVERRUN.)
RMCFG[base] SOFTTERMSIG=SIGUSR1

Moab routes the signal SIGUSR1 through the resource manager to the job when a job reaches its soft wallclock limit.

   
STAGETHRESHOLD
[[[DD:]HH:]MM:]SS
N/A
Specifies the maximum time a job waits to start locally before considering being migrated to a remote peer. In other words, if a job's start time on a remote cluster is less than the start time on the local cluster, but the difference between the two is less than STAGETHRESHOLD, then the job is scheduled locally. The aim is to avoid job/data staging overhead if the difference in start times is minimal.
Note If this attribute is used, backfill is disabled for the associated resource manager.
RMCFG[remote_cluster] STAGETHRESHOLD=00:05:00
Moab only migrates jobs to remote_cluster if the jobs can start five minutes sooner on the remote cluster than they could on the local cluster.
   
STARTCMD
<STRING>
N/A
Specifies the full path to the resource manager job start client. If the resource manager API fails, Moab executes the specified start command in a second attempt to start the job.
Note Moab calls the start command with the format '<CMD> <JOBID> -H <HOSTLIST>' unless the environment variable 'MOABNOHOSTLIST' is set in which case Moab will only pass the job ID.
RMCFG[base] STARTCMD=/usr/local/bin/qrun
Moab uses the specified start command if API failures occur when launching jobs.
   
SUBMITCMD
<STRING>
N/A
Specifies the full path to the resource manager job submission client.
RMCFG[base] SUBMITCMD=/usr/local/bin/qsub
Moab uses the specified submit command when migrating jobs.
   
SUBMITPOLICY
one of NODECENTRIC or PROCCENTRIC
PROCCENTRIC
If set to NODECENTRIC, each specified node requested by the job is interpreted as a true compute host, not as a task or processor.
RMCFG[base] SUBMITPOLICY=NODECENTRIC

Moab uses the specified submit policy when migrating jobs.

   
SUSPENDSIG
<INTEGER> (valid Unix signal between 1 and 64)
--- (resource manager specific default)
If set, Moab sends the specified signal to a job when a job suspend request is issued.
RMCFG[base] SUSPENDSIG=19

Moab uses the specified suspend signal when suspending jobs within the base resource manager.

Note SUSPENDSIG should not be used with TORQUE or other PBS-based resource managers.
   
SYNCJOBID
<BOOLEAN>
---
Specifies that Moab should migrate jobs to the local resource manager with the job's Moab-assigned job ID. Currently only SLURM and other Moabs in a grid accept the specification of a job ID. In a grid, the grid-head will only pass dependencies to the underlying Moab if SYNCJOBID is set.
RMCFG[base] TYPE=wiki:slurm SYNCJOBID=TRUE
Moab migrates jobs to the SLURM queue with a jobid matching the Moab-assigned job ID.
   
SYSTEMMODIFYURL
[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
---
Specifies how Moab modifies attributes of the system. This interface is used in data staging.
RMCFG[base] SYSTEMMODIFYURL=exec:///tmp/system.modify.pl
Moab executes /tmp/system.modify.pl when it modifies system attributes in conjunction with the resource manager base.
   
SYSTEMQUERYURL
[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
---
Specifies how Moab queries attributes of the system. This interface is used in data staging.
RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query
Moab reads /tmp/system.query when it queries the system in conjunction with base resource manager.
   
TARGETUSAGE
<INTEGER>[%]
90%
Amount of resource manager resources to explicitly use. In the case of a storage resource manager, indicates the target usage of data storage resources to dedicate to active data migration requests. If the specified value contains a percent sign (%), the target value is a percent of the configured value. Otherwise, the target value is considered to be an absolute value measured in megabytes (MB).
RMCFG[storage] TYPE=NATIVE RESOURCETYPE=storage
RMCFG[storage] TARGETUSAGE=80%
Moab schedules data migration requests to never exceed 80% usage of the storage resource manager's disk cache and network resources.
   
TIMEOUT
<INTEGER>
30
Time (in seconds) the scheduler waits for a response from the resource manager.
RMCFG[base] TIMEOUT=40
Moab waits 40 seconds to receive a response from the resource manager before timing out and giving up. Moab tries again on the next iteration.
   
TRANSLATIONSCRIPT
<STRING>
---
Script to be inserted into user command file if migration involves translation from one resource manager type to another. The script takes two arguments where the first is the source resource manager type and the second is the destination resource manager type. Types can be any of the following: PBS, SLURM, LSF, SGE, LOADLEVELER, or BPROC.
RMCFG[base] TRANSLATIONSCRIPT=/opt/moab/tools/tscript.sh
Moab inserts a line that will source the specified script into the start of each translated job command file.
   
TRIGGER
<TRIG_SPEC>
---
A trigger specification indicating behaviors to enforce in the event of certain events associated with the resource manager, including resource manager start, stop, and failure.
RMCFG[base] TRIGGER=<X>
   
TYPE
<RMTYPE>[:<RMSUBTYPE>] where <RMTYPE> is one of the following: TORQUE, LL, LSF, NATIVE, PBS, RMS, SGE, SSS, or WIKI and the optional <RMSUBTYPE> value is one of RMS.
PBS
Specifies type of resource manager to be contacted by the scheduler.
Note For TYPE WIKI, AUTHTYPE must be set to CHECKSUM. The <RMSUBTYPE> option is currently only used to support Compaq's RMS resource manager in conjunction with PBS. In this case, the value PBS:RMS should be specified.
RMCFG[clusterA] TYPE=PBS HOST=clusterA PORT=15003
RMCFG[clusterB] TYPE=PBS HOST=clusterB PORT=15005
Moab interfaces to two different PBS resource managers, one located on server clusterA at port 15003 and one located on server clusterB at port 15005.
   
UCALLOCDURATION
[[[DD:]HH:]MM:SS
--- (no allocation duration)
Specifies the minimum duration utility computing resources are allocated when the resource manager's UCTHRESHOLD is satisfied. This feature is used when allocating automatic utility computing resources.
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCALLOCDURATION=00:30:00
RMCFG[base] UCALLOCSIZE=2

The resource manager allocates two processors at a time for a period of at least 30 minutes if the cluster maintains a 20-minute backlog for more than three minutes.

   
UCALLOCSIZE
<INTEGER> (processors)
1
Specifies the number of additional nodes to allocate each time a dynamic utility computing threshold is reached. This feature is used primarily with utility computing resources.
Note Either UCMAXSIZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCALLOCSIZE=4

The resource manager allocates four additional nodes each time the utility computing threshold is reached.

   
UCMAXSIZE
<INTEGER> (processors)
1
Specifies the maximum number of nodes the local cluster can allocate in response to utility computing thresholds. This feature is used primarily with utility computing resources.
Note Either UCMAXSIZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
RMCFG[base] UCMAXSIZE=256

Moab may not allocate more than a total of 256 processors from the utility computing resource even if the utility computing threshold is in violation.

   
UCTHRESHOLD
[[[DD:]HH:]MM:SS
--- (no activation threshold)
Specifies the cluster backlog duration required before the resource manager automatically activates. This feature is used primarily to activate utility computing resources.
Note This parameter is required to enable performance or threshold based automatic utility computing usage.
RMCFG[base] UCTHRESHOLD=00:20:00

The resource manager should be activated if the cluster obtains a 20 minute backlog.

   
UCTHRESHOLDDURATION
[[[DD:]HH:]MM:SS
--- (no threshold duration)
Specifies how long the resource manager's UCTHRESHOLD must be satisfied before resource manager activation is allowed. This parameter prevents statistical spikes from causing unnecessary utility computing allocations. This feature is used primarily to activate utility computing resources.
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCTHRESHOLDDURATION=00:03:00

Utility computing resources should be allocated if the cluster maintains a 20 minute backlog for more than three minutes.

USEVNODES
<BOOLEAN>
FALSE
Specifies whether to schedule on PBS virtual nodes. When set to TRUE, Moab queries PBSPro for vnodes and puts jobs on vnodes rather than "hosts." In some systems, such as PBS + Altix, it may not be desirable to read in the host nodes; for such situations refer to the IGNHNODES parameter.
RMCFG[pbs] USEVNODES=TRUE
   
VARIABLES
<VAR>=<VAL>[,VAR>=<VAL>]
---
Opaque resource manager variables.
RMCFG[base] VARIABLES=SCHEDDHOST=head1
Moab associates the variable SCHEDDHOST with the value head1 on resource manager base.
   
VERSION
<STRING>
SLURM: 10200 (i.e., 1.2.0)
Resource manager-specific version string.
RMCFG[base] VERSION=10124
Moab assumes that resource manager base has a version number of 1.1.24.
   
VMOWNERRM
<STRING>
---
Used with provisioning resource managers that can create VMs. It specifies the resource manager that will own any VMs created by the resource manager.
RMCFG[torque]
RMCFG[prov] RESOURCETYPE=PROV VMOWNERRM=torque
   
WORKLOADQUERYURL
[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
---
Specifies how Moab queries the resource manager for workload information. (See Native RM, URL Notes, and interface details.)
RMCFG[dynamic_jobs] WORKLOADQUERYURL=exec://$TOOLSDIR/job.query.dyn.pl
Moab executes /opt/moab/tools/job.query.dyn.pl to obtain updated workload information from resource manager dynamic_jobs.

URL notes

URL parameters can load files by using the file, exec, and http protocols.

For the protocol file, Moab loads the data directly from the text file pointed to by path.

RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query

For the protocol exec, Moab executes the file pointed to by path and loads the output written to STDOUT. If the script requires arguments, you can use a question mark (?) between the script name and the arguments, and an ampersand (&) for each space.

RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl
RMCFG[native] CLUSTERQUERYURL=exec://opt/moab/tools/cluster.query.pl?-group=group1&-arch=x86

For the protocol http, Moab loads the data from the web based HTTP (hypertext transfer protocol) destination.

RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi

13.2.2 Resource Manager Configuration Details

As with all scheduler parameters, RMCFG follows the syntax described within the Parameters Overview.

13.2.2.1 Resource Manager Types

The RMCFG parameter allows the scheduler to interface to multiple types of resource managers using the TYPE or SERVER attributes. Specifying these attributes, any of the following listed resource managers may be supported. To further assist in configuration, Integration Guides are provided for PBS, SGE, and Loadleveler systems.

TYPE Resource Managers Details
Loadleveler version 2.x and 3.x N/A
Platform's Load Sharing Facility, version 5.1 and higher N/A
Moab Workload Manager Use the Moab peer-to-peer (grid) capabilities to enable grids and other configurations. (See Grid Configuration.)
Moab Native Interface Used for connecting directly to scripts, files, databases, and web services. (See Managing Resources Directly with the Native Interface.)
TORQUE (all versions), OpenPBS (all versions), PBSPro (all versions) N/A
RMS (for Quadrics based systems) For development use only; not production quality.
Sun Grid Engine version 5.3 and higher N/A
Scalable Systems Software Project version 2.0 and higher N/A
Wiki interface specification version 1.0 and higher Used for LRM, YRM, ClubMASK, BProc, SLURM, and others.

13.2.2.2 Resource Manager Name

Moab can support more than one resource manager simultaneously. Consequently, the RMCFG parameter takes an index value such as RMCFG[clusterA]. This index value essentially names the resource manager (as done by the deprecated parameter RMNAME). The resource manager name is used by the scheduler in diagnostic displays, logging, and in reporting resource consumption to the allocation manager. For most environments, the selection of the resource manager name can be arbitrary.

13.2.2.3 Resource Manager Location

The HOST, PORT, and SERVER attributes can be used to specify how the resource manager should be contacted. For many resource managers (such as OpenPBS, PBSPro, Loadleveler, SGE, and LSF) the interface correctly establishes contact using default values. These parameters need only to be specified for resource managers such as the WIKI interface (that do not include defaults) or with resources managers that can be configured to run at non-standard locations (such as PBS). In all other cases, the resource manager is automatically located.

13.2.2.4 Resource Manager Flags

The FLAGS attribute can be used to modify many aspects of a resources manager's behavior.

Flag Description
Jobs started on this resource manager start asynchronously. In this case, the scheduler does not wait for confirmation that the job correctly starts before proceeding. (See Large Cluster Tuning for more information.)
Jobs staged to this resource manager do not need to be explicitly started by the scheduler. The resource manager itself handles job launch.
Resource manager starts and stops together with Moab.
Note This requires that the resource manager support a resource manager start and stop API or the RMSTARTURL and RMSTOPURL attributes are set.
Nodes reported by this resource manager will transfer ownership to this resource manager if they are currently owned by another resource manager that does not have this flag set.
A client resource manager object is created for diagnostic/statistical purposes or to configure Moab's interaction with this resource manager. It represents an external entity that consumes server resources or services, allows a local administrator to track this usage, and configures specific policies related to that resource manager. A client resource manager object loads no data and provides no services.
Setting CLOCKSKEWCHECKING allows you to configure clock skew adjustments. Most of the time it is sufficient to use an NTP server to keep the clocks in your system synchronized.
The resource manager masks details about local workload and resources and presents only information relevant to the remote server.
The resource manager creates credentials within the cluster as needed to support workload. (See Identity Manager Overview.)
The resource manager is capable of launching and executing batch workload.
Add this flag if the working file system doesn't exist on the server to prevent Moab from validating files and directories at migration.
Always checkpoint full job information (useful with Native resource managers).
The resource manager interface is used to negotiate an adjustment in dynamic resource access.
The queue state reported by the resource manager should be ignored. May be used if queues must be disabled inside of a particular resource manager to allow an external scheduler to properly operate.

When this flag is applied to a native resource manager, any jobs that are reported via that resource manager's "workload query URL" have their reported state ignored. For example, if an RM has the IgnWorkloadState flag and it reports that a set of jobs have a state of "Running," this state is ignored and the jobs will either have a default state set or will inherit the state from another RM reporting on that same set of jobs.

This flag only changes the behavior of RMs of type "NATIVE".

When using Moab-type resource managers, Moab will export local reservations when requested.
When set, this flag causes additional job information to be migrated to the resource manager; additional job information includes things such as node features applied via CLASSCFG[name] DEFAULT.FEATURES, the account to which the job was submitted, and job walltime limit.
If the resource manager does not report CPU usage to Moab because CPU usage is at 0%, Moab assumes full CPU usage. When set, Moab recognizes the resource manager report as 0% usage. This is only valid for PBS.
RMs with this flag only update resources/jobs discovered by other non-slave RMs.
Note Non-Moab slaves cannot report resources not seen by non-slave RMs. Moab slaves can report resources not seen by non-slave RMs, but cannot take independent scheduling action.
RMs with this flag only update resources discovered by other non-slave RMs.
The resources and workload reported by the resource manager are not reported to non-administrator users.
Enables job changes made on a grid slave to be pushed to the grid head or master. Without this flag, jobs being reported to the grid head do not show any changes made on the remote Moab server (via mjobctl and so forth).
N/A
Enables Admin proxy job submission, which means administrators may submit jobs in behalf of other users.
Resources of this resource manager may be scheduled by multiple independent sources and may not be assumed to be owned by any single source.
Information from this resource manager may not be used to identify new jobs or nodes. Instead, this information may only be used to update jobs and nodes discovered and loaded from other non-slave resource managers.
This resource manager only provides partial object information and this information does not change over time. Consequently, this resource manager may only be called once per object to modify job and node information.

Example

# resource manager 'torque' should use asynchronous job start 
# and report resources in 'grid' mode
RMCFG[torque] FLAGS=asyncstart,grid

13.2.3 Scheduler/Resource Manager Interactions

In the simplest configuration, Moab interacts with the resource manager using the following four primary functions:

GETJOBINFO

Collect detailed state and requirement information about idle, running, and recently completed jobs.

GETNODEINFO

Collect detailed state information about idle, busy, and defined nodes.

STARTJOB

Immediately start a specific job on a particular set of nodes.

CANCELJOB

Immediately cancel a specific job regardless of job state.

Using these four simple commands, Moab enables nearly its entire suite of scheduling functions. More detailed information about resource manager specific requirements and semantics for each of these commands can be found in the specific resource manager (LL, PBS, or WIKI) overviews.

In addition to these base commands, other commands are required to support advanced features such as dynamic job support, suspend/resume, gang scheduling, and scheduler initiated checkpoint restart.

Information on creating a new scheduler resource manager interface can be found in the Adding New Resource Manager Interfaces section.