Moab Adaptive Computing Suite Administrator's Guide 5.4
Home

Resource Manager Configuration

Defining and Configuring Resource Manager Interfaces

Moab resource manager interfaces are defined using the RMCFG parameter. This parameter allows specification of key aspects of the interface. In most cases, only the TYPE attribute needs to be specified and Moab determines the needed defaults required to activate and use the selected interface. In the following example, an interface to a Loadleveler resource manager is defined.

RMCFG[orion] TYPE=LL
...

Note that the resource manager is given a label of orion. This label can be any arbitrary site-selected string and is for local usage only. For sites with multiple active resource managers, the labels can be used to distinguish between them for resource manager specific queries and commands.

Resource Manager Attributes

The following table lists the possible resource manager attributes that can be configured.

ADMINEXEC
Format:"jobsubmit"
Default:NONE
Description: Normally, when the JOBSUBMITURL is executed, Moab will drop to the UID and GID of the user submitting the job. Specifying an ADMINEXEC of jobsubmit causes Moab to use its own UID and GID instead (usually root). This is useful for some native resource managers where the JOBSUBMITURL is not a user command (such as qsub) but a script that interfaces directly with the resource manager.
Example:
RMCFG[base] ADMINEXEC=jobsubmit
Moab will not use the user's UID and GID for executing the JOBSUBMITURL.
   
AUTHALIST
Format:comma delimited list of local account names
Default:ALL (all accounts may use resources)
Description:Specifies which local accounts are allowed to use reported resource manager resources. In the case of multi-resource manager usage, only the authorized account list on the master resource manager is considered. In the case of peer resource managers, jobs are only migrated if allowed by the AUTHALIST parameter and by all policies on the destination cluster.
Example:
RMCFG[base] AUTHALIST=er342,ex332
Only jobs from accounts er342 and ex332 are considered for execution on resources reported by the resource manager base.
   
AUTHCLIST
Format:comma delimited list of local class names
Default:ALL (all classes may utilize resources)
Description:Specifies which local classes are allowed to use reported resource manager resources. In the case of multi-resource manager usage, only the authorized class list on the master resource manager is considered. In the case of peer resource managers, jobs are only migrated if allowed by the AUTHCLIST parameter and by all policies on the destination cluster.
Example:
RMCFG[base] AUTHCLIST=fast,special
Only jobs from classes fast and special are considered for execution on resources reported by the resource manager base.
   
AUTHGLIST
Format:comma delimited list of local group names
Default:ALL (all groups may use resources)
Description:Specifies which local groups will be allowed to use reported resource manager resources. In the case of multi-resource manager usage, only the authorized group list on the master resource manager is considered. In the case of peer resource managers, jobs are only migrated if allowed by the AUTHGLIST parameter and by all policies on the destination cluster.
Example:
RMCFG[base] AUTHGLIST=staff,development
Only jobs from groups staff and development are considered for execution on resources reported by the resource manager base.
   
AUTHQLIST
Format:comma delimited list of local QOS names
Default:ALL (all QoS's may use resources)
Description:Specifies which local QoS's are allowed to use reported resource manager resources. In the case of multi-resource manager usage, only the authorized QoS list on the master resource manager is considered. In the case of peer resource managers, jobs are only migrated if allowed by the AUTHQLIST parameter and by all policies on the destination cluster.
Example:
RMCFG[base] AUTHQLIST=prio2,prio3
Only jobs from QOS's prio2 and prio3 will be considered for execution on resources reported by the resource manager base.
   
AUTHTYPE
Format:one of CHECKSUM, OTHER, PKI, SECUREPORT, or NONE.
Default:CHECKSUM
Description:Specifies the security protocol to be used in scheduler-resource manager communication.

Note Only valid with WIKI based interfaces
Example:
RMCFG[base] AUTHTYPE=CHECKSUM

Moab requires a secret key based checksum associated with each resource manager message.
   
AUTHULIST
Format:comma delimited list of local user names
Default:ALL (all users may use resources)
Description:Specifies which local users are allowed to use reported resource manager resources. In the case of multi-resource manager usage, only the authorized user list on the master resource manager is considered. In the case of peer resource managers, jobs are only migrated if allowed by the AUTHULIST parameter and by all policies on the destination cluster.
Example:
RMCFG[base] AUTHULIST=steve,john
Only jobs from users steve and john are considered for execution on resources reported by the resource manager base.
   
BANDWIDTH
Format:<FLOAT>[{M|G|T}]
Default:-1 (unlimited)
Description:Specifies the maximum deliverable bandwidth between the Moab server and the resource manager for staging jobs and data. Bandwidth is specified in units per second and defaults to a unit of MB/s. If a unit modifier is specified, the value is interpreted accordingly (M - megabytes/sec, G - gigabytes/sec, T - terabytes/sec).
Example:
RMCFG[base] BANDWIDTH=340G
Moab will reserve up to 340 GB of network bandwidth when scheduling job and data staging operations to and from this resource manager.
   
CHECKPOINTSIG
Format:one of suspend, <INTEGER>, or SIG<X>
Default:---
Description:Specifies what signal to send the resource manager when a job is checkpointed. (See Checkpoint Overview.)
Example:
RMCFG[base] CHECKPOINTSIG=SIGKILL

Moab routes the signal SIGKILL through the resource manager to the job when a job is checkpointed.
   
CHECKPOINTTIMEOUT
Format:[[[DD:]HH:]MM:]SS
Default:0 (no timeout)
Description:Specifies how long Moab waits for a job to checkpoint before canceling it. If set to 0, Moab does not cancel the job if it fails to checkpoint. (See Checkpoint Overview.)
Example:
RMCFG[base] CHECKPOINTTIMEOUT=5:00

Moab cancels any job that has not exited 5 minutes after receiving a checkpoint request.
   
CLIENT
Format:<PEER>
Default:use name of resource manager for peer client lookup
Description:If specified, the resource manager will use the peer value to authenticate remote connections. (See configuring peers). If not specified, the resource manager will search for a CLIENTCFG entry of RM:<RMNAME> in the moab-private.cfg file.
Example:
RMCFG[clusterBI] CLIENT=clusterB

Moab will look up and use information for peer clusterB when authenticating the clusterBI resource manager.
   
CLUSTERQUERYURL
Format:[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:---
Description:Specifies how Moab queries the resource manager. (See Native RM, URL Notes, and interface details.)
Example:
RMCFG[base] CLUSTERQUERYURL=file:///tmp/cluster.config
Moab reads /tmp/cluster.config when it queries base resource manager.
   
CONFIGFILE
Format:<STRING>
Default:N/A
Description:Specifies the resource manager specific configuration file that must be used to enable correct API communication.

Note Only valid with LL- and SLURM-based interfaces
Example:
RMCFG[base] TYPE=LL CONFIGFILE=/home/loadl/loadl_config

The scheduler uses the specified file when establishing the resource manager/scheduler interface connection.

   
DATARM
Format:<RM NAME>
Default:N/A
Description:If specified, the resource manager uses the given storage resource manager to handle staging data in and out.
Example:
RMCFG[clusterB] DATARM=clusterB_storage

When data staging is required by jobs starting/completing on clusterB, Moab uses the storage interface defined by clusterB_storage to stage and monitor the data.
   
DEFAULTCLASS
Format:<STRING>
Default:N/A
Description:Specifies the class to use if jobs submitted via this resource manager interface do not have an associated class.
Example:
RMCFG[internal] DEFAULTCLASS=batch
Moab assigns the class batch to all jobs from the base resource manager that do not have a class assigned.

Note If you are using PBS as the resource manager, a job will never come from PBS without a class, and the default will never apply
   
DEFAULT.JOB
Format:<STRING>
Default:N/A
Description: Specifies the job template to use to set various job attributes that are not specified by the submitter.
Example:
RMCFG[base] DEFAULT.JOB=defjob
Moab uses the defjob job template to identify and apply job attribute defaults.
   
DEFAULTHIGHSPEEDADAPTER
Format:<STRING>
Default:sn0
Description:Specifies the default high speed switch adapter to use when starting LoadLeveler jobs (supported in version 4.2.2 and higher of Moab and 3.2 of LoadLeveler).
Example:
RMCFG[base]     DEFAULTHIGHSPEEDADAPTER=sn1
The scheduler will start jobs requesting a high speed adapter on sn1.
   
DESCRIPTION
Format:<STRING>
Default:N/A
Description:Specifies the human-readable description for the resource manager interface. If white space is used, the description should be quoted.
Example:
RMCFG[ganglia] TYPE=NATIVE:ganglia DESCRIPTION='resource monitor providing extended resource utilization stats'
Moab annotates the ganglia resource manager accordingly.
   
ENV
Format:Semi-colon delimited (;) list of <KEY>=<VALUE> pairs
Default:MOABHOMEDIR=<MOABHOMEDIR>
Description: Specifies a list of environment variables that will be passed to URL's of type 'exec://' for that resource manager.
Example:
RMCFG[base] ENV=HOST=node001;RETRYTIME=50
RMCFG[base] CLUSTERQUERYURL=exec:///opt/moab/tools/cluster.query.pl
RMCFG[base] WORKLOADQUERYURL=exec:///opt/moab/tools/workload.query.pl

The environment variables HOST and RETRYTIME (with values 'node001' and '50' respectively) are passed to the /opt/moab/tools/cluster.query.pl and /opt/moab/tools/workload.query.pl when they are executed.

   
EPORT
Format:<INTEGER>
Default:N/A
Description:Specifies the event port to use to receive resource manager based scheduling events.
Example:
RMCFG[base] EPORT=15017
The scheduler will look for scheduling events from the resource manager host at port 15017.
   
FAILTIME
Format:<[[[DD:]HH:]MM:]SS
Default:N/A
Description:Specifies how long a resource manager must be down before any failure triggers associated with the resource manager fire.
Example:
RMCFG[base] FAILTIME=3:00
If the base resource manager is down for three minutes, any resource manager failure triggers fire.
   
FLAGS
Format:comma delimited list of zero or more of the following: asyncstart, autostart, autosync, client, fullcp, executionServer, grid, hostingCenter, ignqueuestate, loadbalance, private, report, shared, slavepeer or static
Default:N/A
Description:Specifies various attributes of the resource manager. See Flag Details for more information.
Example:
RMCFG[base] FLAGS=static,slavepeer
Moab uses this resource manager to perform a single update of node and job objects reported elsewhere.
   
FLOWINTERVAL
Format:[[[DD:]HH:]MM:]SS
Default:01:00:00 (one hour)
Description:Specifies the duration of the flow control sliding window.
Example:
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FLOWLIMIT
Format:<INTEGER>
Default:1
Description:Specifies the limit of flow metric consumption allowed within the sliding window.
Example:
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FLOWMETRIC
Format:one of jobs, procs, nodes, procseconds, or peseconds
Default:jobs
Description:Specifies the metric of consumption of the flow control sliding window.
Example:
RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30
The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
   
FNLIST
Format:comma delimited list of zero or more of the following: clusterquery, jobcancel, jobrequeue, jobresume, jobstart, jobsuspend, queuequery, resourcequery or workloadquery
Default:N/A
Description:By default, a resource manager utilizes all functions supported to query and control batch objects. If this parameter is specified, only the listed functions are used.
Example:
RMCFG[base] FNLIST=queuequery
Moab only uses this resource manager interface to load queue configuration information.
   
HOST
Format:<STRING>
Default:localhost
Description:The host name of the machine on which the resource manager server is running.
Example:
RMCFG[base] host=server1
   
IGNHNODES
Format:<BOOLEAN>
Default:FALSE
Description:Specifies whether to read in the PBSPro host nodes. This parameter is used in conjunction with USEVNODES. When both are set to TRUE, the host nodes are not queried.
Example:
RMCFG[pbs] IGNHNODES=TRUE
   
JOBCANCELURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab cancels jobs via the resource manager. (See URL Notes below.)
Example:
RMCFG[base] JOBCANCELURL=exec:///opt/moab/job.cancel.lsf.pl
Moab executes /opt/moab/job.cancel.lsf.pl to cancel specific jobs.
   
JOBEXTENDDURATION
Format:[[[DD:]HH:]MM:]SS[,[[[DD:]HH:]MM:]SS][!][<] (or <MIN TIME>[,<MAX TIME>][!])
Default:---
Description: Specifies the minimum and maximum amount of time that can be added to a job's walltime if it is possible for the job to be extended. (See MINWCLIMIT.) As the job runs longer than its current specified minimum wallclock limit (-l minwclimit, for example), Moab attempts to extend the job's limit by the minimum JOBEXTENDDURATION. This continues until either the extension can no longer occur (it is blocked by a reservation or job), the maximum JOBEXTENDDURATION is reached, or the user's specified wallclock limit (-l wallclock) is reached. When a job is extended, it is marked as PREEMPTIBLE, unless the '!' is appended to the end of the configuration string. If the '<' is at the end of the string, however, the job is extended the maximum amount possible.
Example:
RMCFG[base] JOBEXTENDDURATION=30,1:00:00
Moab extends a job's walltime by 30 seconds each time the job is about to run out of walltime until it is bound by one hour, a reservation/job, or the job's original "maximum" wallclock limit.
   
JOBMODIFYURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab modifies jobs via the resource manager. (See Dynamic Jobs, URL Notes, and interface details.)
Example:
RMCFG[base] JOBMODIFYURL=exec://$TOOLSDIR/job.modify.dyn.pl
Moab executes /opt/moab/job.modify.dyn.pl to modify specific jobs.
   
JOBPREEMPTURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab preempts jobs via the resource manager. (See URL Notes below.)
Example:
RMCFG[base] JOBPREEMPTURL=exec:///opt/moab/job.preempt.pl

Moab executes /opt/moab/job.preempt.pl to preempt specific jobs.
   
JOBRSVRECREATE
Format:Boolean
Default:TRUE
Description:Specifies whether Moab will re-create a job reservation each time job information is updated by a resource manager. (See Considerations for Large Clusters for more information.)
Example:
RMCFG[base] JOBRSVRECREATE=FALSE

Moab only creates a job reservation once when the job first starts.
   
JOBSTARTURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab starts jobs via the resource manager. (See URL Notes below.)
Example:
RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi

Moab triggers the jobstart.cgi script via http to start specific jobs.
   
JOBSUBMITURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab submits jobs to the resource manager. (See URL Notes below.)
Example:
RMCFG[base] JOBSUBMITURL=sql://dbserver.flc.com:3554/admin

Moab submits jobs directly to the database located on host dbserver.flc.com.
   
JOBVALIDATEURL
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies how Moab validates newly submitted jobs. (See URL Notes below.) If the script returns with a non-zero exit code, the job is rejected. (See User Proxying/Alternate Credentials.)
Example:
RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl

Moab executes the 'job.validate.pl' script when jobs are submitted to verify they are acceptable.
   
MAXDSOP
Format:<INTEGER>
Default:-1 (unlimited)
Description:Specifies the maximum number of data staging operations that may be simultaneously active.
Example:
RMCFG[ds] MAXDSOP=16
   
MAX.JOB
Format:<STRING>
Default:---
Description: Specifies the job template to use to check various maximum/excluded job attributes that are specified by the submitter.
Example:
RMCFG[base] MAX.JOB=maxjob

Moab will use the maxjob job template to identify and enforce maximum/excluded job attributes.

   
MAXJOBPERMINUTE
Format:<INTEGER>
Default:-1 (unlimited)
Description:Specifies the maximum number of jobs allowed to start per minute via the resource manager.
Example:
RMCFG[base] MAXJOBPERMINUTE=5
The scheduler only allows five jobs per minute to launch via the resource manager base.
   
MAXJOBS
Format:<INTEGER>
Default:0 (limited only by the Moab MAXJOB setting)
Description:Specifies the maximum number of active jobs that this interface is allowed to load from the resource manager.

Note Only works with Moab peer resource managers at this time
Example:
RMCFG[cluster1] SERVER=moab://cluster1
MAXJOBS=200
The scheduler loads up to 200 active jobs from the remote Moab peer cluster1.
   
MINETIME
Format:<INTEGER>
Default:1
Description:Specifies the minimum time in seconds between processing subsequent scheduling events.
Example:
RMCFG[base] MINETIME=5
The scheduler batch-processes scheduling events that occur less than five seconds apart.
   
MIN.JOB
Format:<STRING>
Default:---
Description: Specifies the job template to use to check various minimum/required job attributes that are specified by the submitter.
Example:
RMCFG[base] MIN.JOB=minjob

Moab uses the minjob job template to identify and enforce minimum/required job attributes.

   
NMPORT
Format:<INTEGER>
Default:(any valid port number)
Description:Specifies a non-default resource manager node manager through which extended node attribute information may be obtained.
Example:
RMCFG[base] NMPORT=13001

The scheduler contacts the node manager located on each compute node at port 13001.
   
NODEFAILURERSVPROFILE
Format:<STRING>
Default:N/A
Description:Specifies the rsv template to use when placing a reservation onto failed nodes. (See also NODEFAILURERESERVETIME.)
Example:
moab.cfg
RMCFG[base] NODEFAILURERSVPROFILE=long

RSVPROFILE[long]	DURATION=25:00
RSVPROFILE[long]	USERLIST=john

The scheduler will use the long rsv profile when creating reservations over failed nodes belonging to base.

   
NODESTATEPOLICY
Format:one of OPTIMISTIC or PESSIMISTIC
Default:PESSIMISTIC
Description:Specifies how Moab should determine the state of a node when multiple resource managers are reporting state.
OPTIMISTIC specifies that if any resource manager reports a state of up, that state will be used.
PESSIMISTIC specifies that if any resource manager reports a state of down, that state will be used.
Example:
moab.cfg
RMCFG[ganglia] TYPE=NATIVE CLUSTERQUERYURL=ganglia://
RMCFG[ganglia] FLAGS=SLAVEPEER NODESTATEPOLICY=OPTIMISTIC
   
OMAP
Format:<protocol>://[<host>[:<port>]][<path>]
Default:---
Description:Specifies an object map file that is used to map credentials and other objects when using this resource manager peer. (See Grid Credential Management for full details.)
Example:
moab.cfg
RMCFG[peer1] OMAP=file:///opt/moab/omap.dat
When communicating with the resource manager peer1, objects are mapped according to the rules defined in the /opt/moab/omap.dat file.
   
PBSALTIX
Format:<BOOLEAN>
Default:FALSE
Description: Specifies whether Moab should schedule this resource manager as a PBSPro + Altix system.
Example:
moab.cfg
RMCFG[pbs] PBSALTIX=TRUE
   
POLLINTERVAL
Format:[[[DD:]HH:]MM:]SS
Default:30
Description:Specifies how often the scheduler will poll the resource manager for information.
Example:
RMCFG[base] POLLINTERVAL=1:00
Moab contacts resource manager base every minute for updates.
   
POLLTIMEISRIGID
Format:<BOOLEAN>
Default:FALSE
Description:Determines whether the POLLINTERVAL parameter is interpreted as an interval or a set time for contacting.
Example:
RMCFG[base] POLLTIMEISRIGID=TRUE POLLINTERVAL=1:00:00
Moab polls the resource manager at startup and on the hour.
   
PORT
Format:<INTEGER>
Default:0
Description:Specifies the port on which the scheduler should contact the associated resource manager. The value '0' specifies that the resource manager default port should be used.
Example:
RMCFG[base] TYPE=PBS HOST=cws PORT=20001
Moab attempts to contact the PBS server daemon on host cws, port 20001.
   
RESOURCETYPE
Format:{COMPUTE|FS|LICENSE|NETWORK}
Default:---
Description:Specifies which type of resource this resource manager is configured to control. See Native Resource Managers for more information.
Example:
RMCFG[base] TYPE=NATIVE RESOURCETYPE=FS

Resource manager base will function as a NATIVE resource manager and control file systems.
   
RMSTARTURL
Format:[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:---
Description:Specifies how Moab starts the resource manager.
Example:
RMCFG[base] RMSTARTURL=exec:///tmp/nat.start.pl
Moab executes /tmp/nat.start.pl to start the resource manager base.
   
RMSTOPURL
Format:[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:---
Description:Specifies how Moab stops the resource manager.
Example:
RMCFG[base] RMSTOPURL=exec:///tmp/nat.stop.pl
Moab executes /tmp/nat.stop.pl to stop the resource manager base.
   
SBINDIR
Format:<PATH>
Default:N/A
Description:For use with TORQUE; specifies the location of the TORQUE system binaries (supported in TORQUE 1.2.0p4 and higher).
Example:
RMCFG[base] TYPE=pbs  SBINDIR=/usr/local/torque/sbin
Moab tells TORQUE that its system binaries are located in /usr/local/torque/sbin.
   
SERVER
Format:<URL>
Default:N/A
Description:Specifies the resource management service to use. If not specified, the scheduler locates the resource manager via built-in defaults or, if available, with an information service.
Example:
RMCFG[base] server=ll://supercluster.org:9705
Moab attempts to use the Loadleveler scheduling API at the specified location.
   
SET.JOB
Format:<STRING>
Default:N/A
Description: Specifies the job template to use to force various job attributes regardless of whether or not they are specified by the submitter.
Example:
JOBCFG[setjob] QOS=fast

RMCFG[base] SET.JOB=setjob
Moab uses the setjob job template to identify and enforce mandatory job attributes.
   
SLURMFLAGS
Format:<STRING>
Default:N/A
Description:Specifies characteristics of the SLURM resource manager interface.
Example:

RMCFG[slurm] SLURMFLAGS=COMPRESSOUTPUT
Moab uses the specified flag to determine interface characteristics with SLURM. The COMPRESSOUTPUT flag instructs Moab to use the compact hostlist format for job submissions to SLURM. The flag NODEDELTAQUERY instructs Moab to request delta node updates when it queries SLURM for node configuration.
   
SOFTTERMSIG
Format:<INTEGER> or SIG<X>
Default:---
Description:Specifies what signal to send the resource manager when a job reaches its soft wallclock limit. (See JOBMAXOVERRUN.)
Example:
RMCFG[base] SOFTTERMSIG=SIGUSR1

Moab routes the signal SIGUSR1 through the resource manager to the job when a job reaches its soft wallclock limit.
   
STAGETHRESHOLD
Format:[[[DD:]HH:]MM:]SS
Default:N/A
Description: Specifies the maximum time a job waits to start locally before considering being migrated to a remote peer. In other words, if a job's start time on a remote cluster is less than the start time on the local cluster, but the difference between the two is less than STAGETHRESHOLD, then the job is scheduled locally. The aim is to avoid job/data staging overhead if the difference in start times is minimal.
Note: If this attribute is used, backfill is disabled for the associated resource manager.
Example:
RMCFG[remote_cluster] STAGETHRESHOLD=00:05:00
Moab only migrates jobs to remote_cluster if the jobs can start five minutes sooner on the remote cluster than they could on the local cluster.
   
STARTCMD
Format:<STRING>
Default:N/A
Description:Specifies the full path to the resource manager job start client. If the resource manager API fails, Moab executes the specified start command in a second attempt to start the job.

Note Moab calls the start command with the format '<CMD> <JOBID> -H <HOSTLIST>' unless the environment variable 'MOABNOHOSTLIST' is set in which case Moab will only pass the job ID
Example:
RMCFG[base] STARTCMD=/usr/local/bin/qrun
Moab uses the specified start command if API failures occur when launching jobs.
   
SUBMITCMD
Format:<STRING>
Default:N/A
Description:Specifies the full path to the resource manager job submission client.
Example:
RMCFG[base] SUBMITCMD=/usr/local/bin/qsub
Moab uses the specified submit command when migrating jobs.
   
SUBMITPOLICY
Format:one of NODECENTRIC or PROCCENTRIC
Default:PROCCENTRIC
Description:If set to NODECENTRIC, each specified node requested by the job is interpreted as a true compute host, not as a task or processor.
Example:
RMCFG[base] SUBMITPOLICY=NODECENTRIC

Moab uses the specified submit policy when migrating jobs.
   
SUSPENDSIG
Format:<INTEGER> (valid UNIX signal between 1 and 64)
Default:--- (resource manager specific default)
Description:If set, Moab sends the specified signal to a job when a job suspend request is issued.
Example:
RMCFG[base] SUSPENDSIG=19

Moab uses the specified suspend signal when suspending jobs within the base resource manager.

Note SUSPENDSIG should not be used with TORQUE or other PBS-based resource managers.
   
SYNCJOBID
Format:<BOOLEAN>
Default:---
Description:Specifies that Moab should migrate jobs to the local resource manager queue with a job ID matching the job's Moab-assigned job ID (only available with SLURM.)
Example:
RMCFG[base] TYPE=wiki:slurm SYNCJOBID=TRUE
Moab migrates jobs to the SLURM queue with a jobid matching the Moab-assigned job ID.
   
SYSTEMMODIFYURL
Format:[exec://<path> | http://<address> | <path>]
 
If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:---
Description:Specifies how Moab modifies attributes of the system. This interface is used in data staging.
Example:
RMCFG[base] SYSTEMMODIFYURL=exec:///tmp/system.modify.pl
Moab executes /tmp/system.modify.pl when it modifies system attributes in conjunction with the resource manager base.
   
SYSTEMQUERYURL
Format:[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:---
Description:Specifies how Moab queries attributes of the system. This interface is used in data staging.
Example:
RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query
Moab reads /tmp/system.query when it queries the system in conjunction with base resource manager.
   
TARGETUSAGE
Format:<INTEGER>[%]
Default:90%
Description:Amount of resource manager resources to explicitly use. In the case of a storage resource manager, indicates the target usage of data storage resources to dedicate to active data migration requests. If the specified value contains a percent sign (%), the target value is a percent of the configured value. Otherwise, the target value is considered to be an absolute value measured in megabytes (MB).
Example:
RMCFG[storage] TYPE=NATIVE RESOURCETYPE=storage
RMCFG[storage] TARGETUSAGE=80%
Moab schedules data migration requests to never exceed 80% usage of the storage resource manager's disk cache and network resources.
   
TIMEOUT
Format:<INTEGER>
Default:30
Description:Time (in seconds) the scheduler waits for a response from the resource manager.
Example:
RMCFG[base] TIMEOUT=40
Moab waits 40 seconds to receive a response from the resource manager before timing out and giving up. Moab tries again on the next iteration.
   
TRANSLATIONSCRIPT
Format:<STRING>
Default:---
Description:Script to be inserted into user command file if migration involves translation from one resource manager type to another. The script takes two arguments where the first is the source resource manager type and the second is the destination resource manager type. Types can be any of the following: PBS, SLURM, LSF, SGE, LOADLEVELER, or BPROC.
Example:
RMCFG[base] TRANSLATIONSCRIPT=/opt/moab/tools/tscript.sh
Moab inserts a line that will source the specified script into the start of each translated job command file.
   
TRIGGER
Format:<TRIG_SPEC>
Default:---
Description:A trigger specification indicating behaviors to enforce in the event of certain events associated with the resource manager, including resource manager start, stop, and failure.
Example:
RMCFG[base] TRIGGER=<X>
   
TYPE
Format:<RMTYPE>[:<RMSUBTYPE>] where <RMTYPE> is one of the following: TORQUE, LL, LSF, NATIVE, PBS, RMS, SGE, SSS, or WIKI and the optional <RMSUBTYPE> value is one of RMS.
Default:PBS
Description:Specifies type of resource manager to be contacted by the scheduler.

Note For TYPE WIKI, AUTHTYPE must be set to CHECKSUM. The <RMSUBTYPE> option is currently only used to support Compaq's RMS resource manager in conjunction with PBS. In this case, the value PBS:RMS should be specified.
Example:
RMCFG[clusterA] TYPE=PBS HOST=clusterA PORT=15003
RMCFG[clusterB] TYPE=PBS HOST=clusterB PORT=15005
Moab interfaces to two different PBS resource managers, one located on server clusterA at port 15003 and one located on server clusterB at port 15005.
   
UCALLOCDURATION
Format:[[[DD:]HH:]MM:SS
Default:--- (no allocation duration)
Description:Specifies the minimum duration utility computing resources are allocated when the resource manager's UCTHRESHOLD is satisfied. This feature is used when allocating automatic utility computing resources.
Example:
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCALLOCDURATION=00:30:00
RMCFG[base] UCALLOCSIZE=2

The resource manager allocates two processors at a time for a period of at least 30 minutes if the cluster maintains a 20-minute backlog for more than three minutes.

   
UCALLOCSIZE
Format:<INTEGER> (processors)
Default:1
Description:Specifies the number of additional nodes to allocate each time a dynamic utility computing threshold is reached. This feature is used primarily with utility computing resources.

Note Either UCMAXSIMZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
Example:
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCALLOCSIZE=4

The resource manager allocates four additional nodes each time the utility computing threshold is reached.

   
UCMAXSIZE
Format:<INTEGER> (processors)
Default:1
Description:Specifies the maximum number of nodes the local cluster can allocate in response to utility computing thresholds. This feature is used primarily with utility computing resources.

Note Either UCMAXSIMZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
Example:
RMCFG[base] UCMAXSIZE=256

Moab may not allocate more than a total of 256 processors from the utility computing resource even if the utility computing threshold is in violation.

   
UCTHRESHOLD
Format:[[[DD:]HH:]MM:SS
Default:--- (no activation threshold)
Description:Specifies the cluster backlog duration required before the resource manager automatically activates. This feature is used primarily to activate utility computing resources.

Note This parameter is required to enable performance or threshold based automatic utility computing usage.
Example:
RMCFG[base] UCTHRESHOLD=00:20:00

The resource manager should be activated if the cluster obtains a 20 minute backlog.

   
UCTHRESHOLDDURATION
Format:[[[DD:]HH:]MM:SS
Default:--- (no threshold duration)
Description:Specifies how long the resource manager's UCTHRESHOLD must be satisfied before resource manager activation is allowed. This parameter prevents statistical spikes from causing unnecessary utility computing allocations. This feature is used primarily to activate utility computing resources.
Example:
RMCFG[base] UCTHRESHOLD=00:20:00
RMCFG[base] UCTHRESHOLDDURATION=00:03:00

Utility computing resources should be allocated if the cluster maintains a 20 minute backlog for more than three minutes.

   
USEVNODES
Format:<BOOLEAN>
Default:FALSE
Description:Specifies whether to schedule on PBS virtual nodes. When set to TRUE, Moab queries PBSPro for vnodes and puts jobs on vnodes rather than "hosts." In some systems, such as PBS + Altix, it may not be desirable to read in the host nodes; for such situations refer to the IGNHNODES parameter.
Example:
RMCFG[pbs] USEVNODES=TRUE
   
VARIABLE
Format:<VAR>=<VAL>[,VAR>=<VAL>]
Default:---
Description:Opaque resource manager variables.
Example:
RMCFG[base] VARIABLE=SCHEDDHOST=head1
Moab associates the variable SCHEDDHOST with the value head1 on resource manager base.
   
VERSION
Format:<STRING>
Default:SLURM: 10200 (i.e., 1.2.0)
Description:Resource manager-specific version string.
Example:
RMCFG[base] VERSION=10124
Moab assumes that resource manager base has a version number of 1.1.24.
   
WORKLOADQUERYURL
Format:[file://<path> | http://<address> | <path>]
 
If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:---
Description:Specifies how Moab queries the resource manager for workload information. (See Native RM, Dynamic Jobs, URL Notes, and interface details.)
Example:
RMCFG[dynamic_jobs] WORKLOADQUERYURL=exec://$TOOLSDIR/job.query.dyn.pl
Moab executes /opt/moab/tools/job.query.dyn.pl to obtain updated workload information from resource manager dynamic_jobs.
   

URL notes

URL parameters can load files by using the file, exec, http, and sql protocols.

For the protocol file, Moab loads the data directly from the text file pointed to by path.

RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query

For the protocol exec, Moab executes the file pointed to by path and loads the output written to STDOUT. If the script requires arguments, you can use a question mark (?) between the script name and the arguments, and an ampersand (&) for each space.

RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl
RMCFG[native] CLUSTERQUERYURL=exec://opt/moab/tools/cluster.query.pl?-group=group1&-arch=x86

For the protocol http, Moab loads the data from the web based HTTP (hypertext transfer protocol) destination.

RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi

For the protocol sql, Moab loads the data from the specified database.

RMCFG[base] JOBSUBMITURL=sql://dbserver.flc.com:3554/admin

Resource Manager Configuration Details

As with all scheduler parameters, RMCFG follows the syntax described within the Parameters Overview.

Resource Manager Types

The RMCFG parameter allows the scheduler to interface to multiple types of resource managers using the TYPE or SERVER attributes. Specifying these attributes, any of the following listed resource managers may be supported. To further assist in configuration, Integration Guides are provided for PBS, SGE, and Loadleveler systems.

TYPE Resource Managers Details
LL Loadleveler version 2.x and 3.x N/A
LSF Platform's Load Sharing Facility, version 5.1 and higher N/A
Moab Moab Workload Manager Use the Moab peer-to-peer (grid) capabilities to enable grids and other configurations. (See Grid Configuration.)
Native Moab Native Interface Used for connecting directly to scripts, files, databases, and web services. (See Managing Resources Directly with the Native Interface.)
PBS TORQUE (all versions), OpenPBS (all versions), PBSPro (all versions) N/A
RMS RMS (for Quadrics based systems) For development use only; not production quality.
SGE Sun Grid Engine version 5.3 and higher N/A
SSS Scalable Systems Software Project version 2.0 and higher N/A
WIKI Wiki interface specification version 1.0 and higher Used for LRM, YRM, ClubMASK, BProc, SLURM, and others.

Resource Manager Name

Moab can support more than one resource manager simultaneously. Consequently, the RMCFG parameter takes an index value such as RMCFG[clusterA]. This index value essentially names the resource manager (as done by the deprecated parameter RMNAME). The resource manager name is used by the scheduler in diagnostic displays, logging, and in reporting resource consumption to the allocation manager. For most environments, the selection of the resource manager name can be arbitrary.

Resource Manager Location

The HOST, PORT, and SERVER attributes can be used to specify how the resource manager should be contacted. For many resource managers (such as OpenPBS, PBSPro, Loadleveler, SGE, and LSF) the interface correctly establishes contact using default values. These parameters need only to be specified for resource managers such as the WIKI interface (that do not include defaults) or with resources managers that can be configured to run at non-standard locations (such as PBS). In all other cases, the resource manager is automatically located.

Resource Manager Flags

The FLAGS attribute can be used to modify many aspects of a resources manager's behavior.

Flag Description
ASYNCSTART Jobs started on this resource manager start asynchronously. In this case, the scheduler does not wait for confirmation that the job correctly starts before proceeding. (See Large Cluster Tuning for more information.)
AUTOSTART Jobs staged to this resource manager do not need to be explicitly started by the scheduler. The resource manager itself handles job launch.
AUTOSYNC Resource manager starts and stops together with Moab.

Note This requires that the resource manager support a resource manager start and stop API or the RMSTARTURL and RMSTOPURL attributes are set
BECOMEMASTER Nodes reported by this resource manager will transfer ownership to this resource manager if they are currently owned by another resource manager that does not have this flag set.
CLIENT A client resource manager object loads no data and provides no services. It is created for diagnostic and statistical purposes only. A client resource manager is created to represent an external entity that is consuming server resources or services and allows a local administrator to track this usage.
DYNAMICCRED The resource manager creates credentials within the cluster as needed to support workload. (See Identity Manager Overview.)
EXECUTIONSERVER The resource manager is capable of launching and executing batch workload.
FULLCP Always checkpoint full job information (useful with Native resource managers).
GRID The resource manager masks details about local workload and resources and presents only information relevant to the remote server.
HOSTINGCENTER The resource manager interface is used to negotiate an adjustment in dynamic resource access.
IGNQUEUESTATE The queue state reported by the resource manager should be ignored. May be used if queues must be disabled inside of a particular resource manager to allow an external scheduler to properly operate.
LOADBALANCE N/A
LOCALRSVEXPORT N/A
MIGRATEALLJOBATTRIBUTES When set, this flag causes additional job information to be migrated to the resource manager; additional job information includes things such as node features applied via CLASSCFG[name] DEFAULT.FEATURES, the account to which the job was submitted, and job walltime limit.
NOAUTORES If the resource manager does not report CPU usage to Moab because CPU usage is at 0%, Moab assumes full CPU usage. When set, Moab recognizes the resource manager report as 0% usage.
PRIVATE The resources and workload reported by the resource manager are not reported to non-administrator users.
PUSHSLAVEJOBUPDATES Enables job changes made on a grid slave to be pushed to the grid head or master. Without this flag, jobs being reported to the grid head do not show any changes made on the remote Moab server (via mjobctl and so forth).
REPORT N/A
ROOTSUBMIT Moab submits jobs to the resource manager using root's credentials and environment.

Note Affects Native resource managers only.
SHARED Resources of this resource manager may be scheduled by multiple independent sources and may not be assumed to be owned by any single source.
SLAVEPEER Information from this resource manager may not be used to identify new jobs or nodes. Instead, this information may only be used to update jobs and nodes discovered and loaded from other non-slave resource managers.
STATIC This resource manager only provides partial object information and this information does not change over time. Consequently, this resource manager may only be called once per object to modify job and node information.

Example 

# resource manager 'torque' should use asynchronous job start 
# and report resources in 'grid' mode
RMCFG[torque] FLAGS=asyncstart,grid

Other Attributes

The maximum amount of time Moab waits on a resource manager call can be controlled by the TIMEOUT parameter that defaults to 30 seconds. Only rarely will this parameter need to be changed. The AUTHTYPE attribute allows specification of how security over the scheduler/resource manager interface is to be handled. Currently, only the WIKI interface is affected by this parameter.

Another RMCFG attribute is CONFIGFILE, which specifies the location of the resource manager's primary configuration file and is used when detailed resource manager information not available via the scheduling interface is required. It is currently only used with the Loadleveler interface and may be specified when using Moab grid-scheduling capabilities.

Finally, the NMPORT attribute allows specification of the resource manager's node manager port and is only required when this port has been set to a non-default value. It is currently only used within PBS to allow MOM specific information to be gathered and utilized by Moab.

Scheduler/Resource Manager Interactions

In the simplest configuration, Moab interacts with the resource manager using the following four primary functions:

GETJOBINFO

Collect detailed state and requirement information about idle, running, and recently completed jobs.

GETNODEINFO

Collect detailed state information about idle, busy, and defined nodes.

STARTJOB

Immediately start a specific job on a particular set of nodes.

CANCELJOB

Immediately cancel a specific job regardless of job state.

Using these four simple commands, Moab enables nearly its entire suite of scheduling functions. More detailed information about resource manager specific requirements and semantics for each of these commands can be found in the specific resource manager (LL, PBS, or WIKI) overviews.

In addition to these base commands, other commands are required to support advanced features such as dynamic job support, suspend/resume, gang scheduling, and scheduler initiated checkpoint restart.

Information on creating a new scheduler resource manager interface can be found in the Adding New Resource Manager Interfaces section.

Home