Appendix A: Moab Parameters

Specifies the minimum job wallclock time for virtual scaling (optimistic-like backfilling.) Any job with a wallclock time less than this setting will not be virtually scaled. The value specified relates to a job's original walltime and not its virtually-scaled walltime.

Specifies the factor by which eligible jobs' wallclock time is virtually scaled (optimistic-like backfilling).

If you do not want scaling, set BFVIRTUALWALLTIMESCALINGFACTOR to "0" (default). Setting to "1" is not recommended as it impacts performance. When set to "1", Moab will exercise the code paths of scaling but no actual scaling will occur.

Time between automatic Moab checkpoints.

If RMPOLLINTERVAL does not specify both a minimum and maximum poll time, Moab will ignore CHECKPOINTINTERVAL and checkpoint every iteration.

List of zero or more space delimited <ATTR>=<VALUE> pairs where <ATTR> is one of the following:

General Credential Flags, DEFAULT.ATTR, DEFAULT.DISK, DEFAULT.FEATURES, DEFAULT.GRES, DEFAULT.MEM, DEFAULT.NODE, DEFAULT.NODESET, DEFAULT.PROC, ENABLEPROFILING, EXCL.FEATURES, EXCLUDEUSERLIST, HOSTLIST, JOBEPILOG, JOBPROLOG, MAXPROCPERNODE, MAX.NODE, MAX.PROC, MAX.WCLIMIT, MIN.NODE, MIN.PROC, MIN.TPN, MIN.WCLIMIT, PARTITION, PRIORITY, PRIORITYF, QDEF, QLIST, REQ.FEATURES, REQUIREDACCOUNTLIST, REQUIREDUSERLIST, RESFAILPOLICY, SYSPRIO, WCOVERRUN, usage limit, or fairshare usage limit specification.

Disallows interactive jobs submitted with msub -I.

Note: It is possible for users to submit interactive jobs directly to a resource manager, which can bypass the DISABLEINTERACTIVEJOBS parameter. However, some resource managers (such as TORQUE) will check with Moab before allowing an interactive job.

ACCOUNTCENTRIC, HIDEBLOCKED, HIDEDRAINED, NODECENTRIC, or USEBLOCKING

Specifies flags that control how Moab client commands display varied information.

ACCOUNTCENTRIC will display account information in showq, rather than group information.

HIDEBLOCKED will prevent showq from listing information about blocked jobs which are not owned by the user if the user is not an admin.

HIDEDRAINED prevents mdiag -n from displaying nodes and mvmctl -q from displaying VMs in the DRAINED state. An override option of mdiag -n -w nodestate=drained lists only those nodes with a DRAINED state. Similarly, an override option of mvmctl -q -w state=drained lists only those VMs with a DRAINED state.

NODECENTRIC will display node allocation information instead of processor allocation information in showq.

USEBLOCKING disables threading for commands that support it; those commands include showq, mdiag -n, and mdiag -j.

By default, when a job is purged or removed by the TORQUE resource manager for a walltime violation, the job takes on a state of Completed and a completion code of 0. If TRUE, the job state is set to Removed and has a completion code of 98. ENABLEFAILUREFORPURGEDJOB is for the TORQUE resource manager only.

For ENABLEFAILUREFORPURGEDJOB to return Removed job states, you must reset the TORQUE server attribute keep_completed to 0 in qmgr. See "Queue Attributes" in the TORQUE Administrator Guide for more information.

Configures Moab so that it will accept msub submissions, start jobs, process triggers, etc., in a manner which minimizes their processing time. The downside is that Moab will return minimal information about these jobs at submit time.

If ENABLEHIGHTHROUGHPUT is TRUE, you must set NODEALLOCATIONPOLICY to FIRSTAVAILABLE.

• Replace all comments with spaces (excludes Resource Manager directives)
• Strip empty lines
• Replace \r with \n
• Lock job to a PBS resource manager if $PBS is found in the script

Include the FILTERCMDFILE parameter in the moab.cfg file that resides on the clients.

FILTERCMDFILE must be FALSE for REJECTDOSSCRIPTS to work correctly.

Specifies the unit of tracking fairshare usage. The following options are:

• DEDICATEDPS (dedicated processor seconds delivered) tracks dedicated processor seconds.
• DEDICATEDPS% (dedicated processor seconds available) to specify that percentage based fairshare should be used. See Fairshare Overview and Fairshare Consumption Metrics for more information.
• DEDICATEDPES (dedicated processor-equivalent seconds delivered) tracks dedicated processor-equivalent seconds.
• UTILIZEDPS (utilized processor seconds delivered) tracks the number of utilized processor seconds.

List of zero or more space delimited <ATTR >=<VALUE> pairs where <ATTR> is one of the following:

TYPE, PRIVATE, INVERTTASKCOUNT, FEATUREGRES, or STARTDELAY

Specifies associations of generic resources into resource groups.

When PRIVATE is set to TRUE, Moab puts the requested generic resource on a separate job request.

By default a private request is a request with 1 task with X number of generic resources per task. When INVERTTASKCOUNTandPRIVATE are set to TRUE, Moab makes the generic resource's private request a request with X number of tasks with 1 generic resource per task.

See 12.6 Managing Consumable Generic Resources for more information.

List of zero or more space delimited <ATTR>=<VALUE> pairs where <ATTR> is one of the following:

General Credential Flags, PRIORITY, ENABLEPROFILING, QLIST, QDEF, PLIST, FLAGS, usage limits, or a fairshare usage limit specification.

Causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires (essentially, waiting for the PREEMPTEE jobs to finish).

It may take some time for the PREEMPTEE jobs to clear out. During that time, the PREEMPTOR job might want to look elsewhere to run, which would be disruptive as it might preempt another set of jobs. If you wish it prevent this, it is recommended that you set GUARANTEEDPREEMPTION to TRUE.

For related information, see About preemption, Reservation Policies, DEFERSTARTCOUNT, DEFERTIME, RESERVATIONRETRYTIME, NODEFAILURERESERVETIME, and JOBRETRYTIME.

This parameter enables the identity manager interface allowing credential, policy, and usage information to be shared with an external information service.

Only one identity manager can be configured at a time.

Specifies the minimum amount of time the scheduler should wait after receiving a job event until it should process that event. This parameter allows sites with bursty job submissions to process job events in groups decreasing total job scheduling cycles and allowing the scheduler to make more intelligent choices by aggregating job submissions and choosing between the jobs. See Considerations for Large Clusters.

The JOBCFG parameter supports the following attributes:

NONE, ACCOUNT, ACTION, AUTOSIZE, CLASS, CPULIMIT, DESCRIPTION, DGRES, FAILUREPOLICY, GROUP, IFLAGS, JOBSCRIPT MEM (for MEM=<value>), MEMORY (for MEMORY=$LEARN), NODEACCESSPOLICY, NODEMOD, PARTITION, PREF, QOS, RESTARTABLE, RM, RMSERVICEJOB, SELECT, SOFTWARE, SRM, TEMPLIMIT, TFLAGS, USER, VMUSAGE, WALLTIME, WORK

It also supports the following Wiki attributes:

ARGS, DMEM, DDISK, DWAP, ERROR, EXEC, EXITCODE, GATTR, GEVENT, IWD, JNAME, NAME, PARTITIONMASK, PRIORITYF, RDISK, RSWAP, RAGRES, RCGRES, TASKPERNODE, TRIGGER, VARIABLE, NULL

Note: The index to the JOBCFG parameter can either be an admin-chosen job template name or the exact name of job reported by one or more workload queries. See Wiki Attributes and Job Template Extensions.

Extends the job walltime when Moab starts the job up to the lesser of the maximum or the next reservation (rounded down to the nearest minute).

JOBEXTENDSTARTWALLTIME TRUE and JOBEXTENDDURATION cannot be configured together. If they are in the same moab.cfg or are both active, then the JOBEXTENDDURATION will not be honored.

JOBEXTENDSTARTWALLTIME TRUE

Submit job with a minimum wallclock limit and a walltime; for example:

echo sleep 500 | msub -A ee -l
nodes=5,minwclimit=5:00,walltime=30:00,partition=g02

Specifies the number of times a job is requeued and restarted by Moab if the job fails (if the job itself returns a non-zero exit code). Some types of jobs may succeed if automatically retried several times in short succession. This parameter was created with these types of jobs in mind. Note that the job in question must also be restartable (the job needs to have the "RESTARTABLE" flag set on it) and the RM managing the job must support requeuing and starting completed jobs.

If a job fails too many times, and reaches the number of retries given by JobFailRetryCount, then a UserHold is placed on the job and a message is attached to it signifying that the job has a "restart count violation."

Soft and hard limit of the amount of time Moab will allow a job to exceed its wallclock limit before it first sends a mail to the primary admin (soft limit) and then terminates the job (hard limit). See WCVIOLATIONACTION or Usage-based Limits.

If you run Moab with the TORQUE resource manager, you must set the $ignwalltime parameter to true in the /var/spool/torque/mom_priv/config file, otherwise the pbs_mom will kill any job that exceeds its walltime. See the TORQUE documentation for more information.

Specifies how Moab should track the dynamic aspects of a job's priority. ACCRUE indicates that the job will accrue queuetime based priority from the time it is submitted unless it violates any of the policies not specified in JOBPRIOEXCEPTIONS. RESET indicates that it will accrue priority from the time it is submitted unless it violates any of the JOBPRIOEXCEPTIONS. However, with RESET, if the job does violate JOBPRIOEXCEPTIONS then its queuetime based priority will be reset to 0.

JOBPRIOACCRUALPOLICY is a global parameter, but can be configured to work only in QOSCFG:

QOSCFG[arrays]  JOBPRIOACCRUALPOLICY=ACCRUE

The following old JOBPRIOACCRUALPOLICY values have been deprecated and should be adjusted to the following values:

• QUEUEPOLICY= ACCRUE and JOBPRIOEXCEPTIONSSOFTPOLICY,HARDPOLICY
• QUEUEPOLICYRESET= RESET and JOBPRIOEXCEPTIONSSOFTPOLICY,HARDPOLICY
• ALWAYS= ACCRUE and JOBPRIOEXCEPTIONSALL
• FULLPOLICY= ACCRUE and JOBPRIOEXCEPTIONSNONE
• FULLPOLICYRESET= RESET and JOBPRIOEXCEPTIONSNONE

Specifies exceptions for calculating a job's dynamic priority (QUEUETIME, XFACTOR, TARGETQUEUETIME). Normally, when a job violates a policy, is placed on hold, or has an unsatisfied dependency, it will not accrue priority. Exceptions can be configured to allow a job to accrue priority in spite of any of these violations. With DEPENDS a job will increase in priority even if there exists an unsatisfied dependency. With SOFTPOLICY, HARDPOLICY, or IDLEPOLICY a job can accrue priority despite violating a specific limit. With DEFER, USERHOLD, BATCHHOLD, or SYSTEMHOLD a job can accrue priority despite being on hold.

JOBPRIOEXCEPTIONS is a global parameter, but can be configured to work only in QOSCFG:

QOSCFG[arrays]  JOBPRIOEXCEPTIONS=IDLEPOLICY

Moab will remove the specified environment variables from the job's environment before migrating the job to its destination resource manager. This is useful when jobs submit themselves from one cluster to another with the full environment.

This parameter is currently only supported with TORQUE resource managers.

If set to NONE, no mail is sent. If set to DEFAULT, Moab sends mail via the system's default mail program (usually /usr/bin/sendmail). If set to the local path of a mail program, Moab uses the specified mail program to send mail.

By default, Moab mail notification is disabled. To enable, you must set MAILPROGRAM to DEFAULT or specify some other locally available mail program. If the default mail domain is set, emails will be routed to this domain unless a per-user domain is specified using the EMAILADDRESS attribute of the USERCFG parameter. If neither of these values is set, Moab uses "@localhost" as the mail domain. See Notify Admins.

For jobs, the email address used on the msub -M option overrides all other user email addresses. Additionally, administrators are notified in the case of job violations.

Specifies the maximum quantity of jobs for which Moab should allocate memory used for tracking jobs. If Moab is tracking the maximum quantity of jobs specified by this parameter, it rejects subsequent jobs submitted by any user since it has no memory left with which to track newly submitted jobs.

If a user submitted a job with the msub command, this rejection behavior requires the user to resubmit the job at a later time after other jobs have completed, which frees memory in which Moab can place later-submitted jobs. If a user submitted a job with the TORQUE qsub command, TORQUE will automatically resubmit the job to Moab until Moab accepts it.

The mdiag -S command indicates if any job overflows have occurred.

If this parameter's value is changed, it does not go into effect until Moab restarts. Moab reads the parameter only on initial startup and uses its value to allocate the memory it uses to track jobs.

Specifies the maximum number of compute nodes supported.

Specifies the maximum number of reservations on a node.

For large SMP systems (>512 processors/node), Adaptive Computing advises adjusting the value to approximately twice the average sum of admin, standing, and job reservations present.

A second number, led by a comma, can also be specified to set a maximum number of reservations for nodes that are part of the SHARED partition.

The maximum possible value of MAXRSVPERNODE is 8192 for a global node and 4096 for any other node.
Moab must be restarted for any changes to this parameter to take effect. The command mdiag -S indicates whether any node reservation overflows have occurred. See Considerations for Large Clusters.

Do not lower the MAXRSVPERNODE value while there are active jobs in the queue. This can lead to queue instability and certain jobs could become stuck or disconnected from the system.

Causes Moab to encrypt the events delivered via zeroMQ using the Advanced Encryption Standard (AES) algorithm. Must be a Base64-encoded, 128-bit (16-byte) key. Messages will be encrypted using AES in CBC mode where inputs are padded with PKCS5 padding. The initialization vector is calculated by using an MD5 hash of the key specified in MESSAGEQUEUESECRETKEY.

MESSAGEQUEUESECRETKEY can only be specified in the moab-private.cfg file.

Specifies the length of the interval (in seconds) between job queries when using msub -K. Jobs submitted with the -K option query the scheduler every MSUBQUERYINTERVAL seconds until the job is completed.

MSUBQUERYINTERVAL can exist as an environment variable. Any value in moab.cfg overrides the environment variable.

Note: If MSUBQUERYINTERVAL is set to 0, the -K option will be disabled. Jobs will still submit correctly, but the client will not continue to check on the job.

One of the following:

SHARED, SHAREDONLY, SINGLEJOB, SINGLETASK, SINGLEUSER, or UNIQUEUSER

NODEACCESSPOLICY SINGLEUSER

Moab will allow resources on a node to be used by more than one job provided that the jobs are all owned by the same user.

One of the following:

FIRSTAVAILABLE, LASTAVAILABLE, MINRESOURCE, CPULOAD, LOCAL, CONTIGUOUS, MAXBALANCE, PRIORITY, or PLUGIN.

Specifies how Moab should allocate available resources to jobs. See Node Allocation Overview for more information.

If ENABLEHIGHTHROUGHPUT is TRUE, you must set NODEALLOCATIONPOLICY to FIRSTAVAILABLE.

One of the following:

CANCEL, HOLD, IGNORE, MIGRATE, NOTIFY, or REQUEUE

<POLICY>[:<RESOURCETYPE>] ...

where <POLICY> is one of COMBINED, DEDICATED, or UTILIZED

and <RESOURCETYPE> is one of PROC, MEM, SWAP, or DISK

where <LABEL> is any string and <NODECAT> is one of the defined node categories.

When dynamic nodes are created in Moab, they are usually created with a request ID.

NODEIDLEPURGETIME is the amount of time for all nodes with the same request ID to be idle before Moab begins firing the node end trigger for each iteration.

If one or more of the nodes with the same request ID becomes non-idle, Moab stops firing the node end trigger for all of the nodes with the same request ID until the NODEIDLEPURGETIME is once again met.

A value of 0 disables this feature.

Causes Moab to attempt to span a job evenly across nodesets unless doing so delays the job beyond the requested NODESETDELAY.

Must use with NODESETPLUS set to SPANEVENLY; if you do not want to use SPANEVENLY, use NODESETISOPTIONAL instead of NODESETDELAY.

Specifies how Moab distributes jobs among nodesets. See Node Set Overview.

Neither SPANEVENLY nor DELAY values of the NODESETPLUS parameter will work with multi-req jobs or preemption.

Length of time Moab will assume untracked generic resources will remain unavailable for scheduling if a system reservation is not explicitly created for the node.

If NODEUNTRACKEDRESDELAYTIME is enabled and there is an untracked resource preventing a job from running, then the job remains in the idle queue instead of being deferred.

Disables all Moab OS credential lookups, including UID, GID, user to group mappings, and any other OS specific information.

Setting OSCREDLOOKUP by itself does not allow job submission; additional configuration is required. When submitting jobs from user accounts that do not exist on the head node (where Moab HPC Suite and TORQUE are running), you must also set the PROXYJOBSUBMISSION flag in addition to specifying configuration settings in the resource manager configuration file. See the example that follows for information on required resource manager settings.

By default Moab's scheduling routine schedules each job on each partition using the following algorithm:

prioritize

foreach (job)

     find the partition on which that job should run

     schedule job

In this model, a job's priority is the same on each partition as it uses a single global priority. Because a job's priority is the same on every partition, Moab prioritizes the queue once and then schedules the prioritized queue across all partitions.

When PERPARTITIONSCHEDULING TRUE is set, the following algorithm is used:

foreach (partition)

     prioritize

     foreach (job)

          schedule job

In this case, each partition may have a unique priority configuration and Moab will re-prioritize the jobs for each partition on the system. Each job is prioritized and scheduled on each partition. See PARCFG for more information. Also, note that Moab will order the partitions as they are discovered in the moab.cfg file. Partitions should be explicitly ordered via PARCFG in the moab.cfg file.

Note: If this policy is set to REQUEUE, preemptible jobs should be marked as RESTARTABLE. If this policy is set to SUSPEND, preemptible jobs should be marked as SUSPENDABLE. Note: Moab uses preemption escalation to preempt resources if the specified preemption facility is not applicable. This means if the policy is set to SUSPEND and the job is not SUSPENDABLE, Moab may attempt to requeue or even cancel the job.

Determines which jobs to preempt based on size or priority. The higher the value, the more emphasis is placed on the priority of the job, causing the lower priority jobs to be preempted first. The lower the value, the more emphasis is placed on the size of the job, causing the smaller jobs to be preempted first. If set to 0, job priority will be ignored, job size will take precedence and the smallest jobs will be preempted.

The special setting of -1 places the emphasis solely on resource utilization. This means that jobs will be preempted in a manner that keeps the resource utilization at the highest level, regardless of job priority or size.

Specifies the number of statistical profiles to maintain.

PROFILECOUNT must be set high enough that at least one day of statistics is maintained. The statistics time window can be determined by measuring PROFILEDURATION * PROFILECOUNT. If PROFILEDURATION is one hour then PROFILECOUNT must be at least 24 so 24 hours worth of statistics are maintained. If PROFILEDURATION is 30:00 then PROFILECOUNT must be set to at least 48. If PROFILECOUNT is not high enough for at least one day of statistics, Moab adjusts it automatically.

Specifies whether or not you want to send cache objects (nodes, jobs, services, etc.) to Moab Web Services.

List of zero or more space delimited <ATTR>=<VALUE> pairs where <ATTR> is one of the following:

General Credential Flags, PRIORITY, ENABLEPROFILING, FSTARGET, JOBPRIOACCRUALPOLICY, JOBPRIOEXCEPTIONS, MEMBERULIST, QTWEIGHT, QTTARGET, XFWEIGHT, XFTARGET, PREEMPTMINTIME, PREEMPTMAXTIME, PREEMPTQTTHRESHOLD, PREEMPTXFTHRESHOLD, PREEMPTEES, RSVQTTHRESHOLD, RSVXFTHRESHOLD, ACLBLTHRESHOLD, ACLQTTHRESHOLD, ACLXFTHRESHOLD, PLIST, QFLAGS, or a usage limit.