Appendices > SCHEDCFG flags

Conventions

SCHEDCFG flags

Flag Description
AGGREGATENODEFEATURES

AGGREGATENODEFEATURES causes Moab to aggregate features reported by the different RMs. For example, if you have two RMs reporting different features for the same node, Moab will add both features together (instead of one being overwritten by the other).

In order to set features manually, you can use mnodectl -m features (for details, see mnodectl).

ALLOWINFINITEJOBS ALLOWINFINITEJOBS allows infinite wallclock times to be accepted. Previously, jobs with infinite job times were allowed by default.
ALLOWMULTICOMPUTE ALLOWMULTICOMPUTE tells Moab how to resolve conflicting information from different resource managers. If ALLOWMULTICOMPUTE is specified, Moab will use the STATE and OS information from the resource manager that reports the node as online.
DISABLEPERJOBNODESETS Disables a job's ability to override the system specified node set. See 13.3 Resource Manager Extensions for more information.
DISABLEPARTIALNODERESERVATIONS Blocks partial node reservations.
ENABLESLURMMEMPERCPU

By default Moab calls sbatch or srun with a --mem= request in a SLURM environment. When you set ENABLESLURMMEMPERCPU, Moab instead calls --mem-per-cpu=. This is to allow sites with policies that require the other parameter to use --mem-per-cpu.

ENFORCERESERVEDNODES Without this flag Moab tries to optimize the reservation for a job before it starts, meaning a job may start on nodes that weren't part of its reservation. With this flag Moab tries to start jobs only on the nodes that were reserved.
ENFORCESAMENODESET The same nodeset is not enforced across job requirements by default, rather each requirement is scheduled separately and the nodesets are determined on a per-req basis. To have Moab enforce the same nodeset across all job requirements set this flag.
EXTENDEDGROUPSUPPORT Allows Moab to consider a user's secondary Linux groups when dealing with reservation ACLs.
FASTGROUPLOOKUP Moab will use the system call getgrouplist to gather group information. This can significantly improve performance on some LDAP systems.
FASTRSVSTARTUP

Speeds up start time if there are existing reservations.

FASTRSVSTARTUP is incompatible with partial node reservations.

On very large systems, if there is a reservation in the checkpoint file on all the nodes, it would take a really long time for Moab to start up. For every node in the reservation, Moab checks every other node. With this flag, Moab just uses the nodelist that was checkpointed to create the reservation. It speeds up the startup process because it doesn't have to check every node. Where Moab would take 8 - 10 minutes to start up with an 18,000 node reservation without the flag, Moab can start up in 2-3 minutes with the flag.

With the flag you will see one difference in checknode. A reservation that uses all the procs on a node initially shows that all the procs are blocked. Without the flag, and as jobs fill on the node, the blocked resources will be configured - dedicated (ex. 5/6). With the flag, the blocked resources will always be what the reservation is blocking and won't change when jobs fill on the node.

Without flag:
Reservations:
brian.1x1 User -00:12:52 -> INFINITY ( INFINITY)
Blocked Resources@-00:00:02 Procs: 5/6 (83.33%) Mem: 0/5000 (0.00%)
Blocked Resources@00:04:58 Procs: 6/6 (100.00%) Mem: 0/5000 (0.00%)
m.2x1 Job:Running -00:00:02 -> 00:04:58 (00:05:00)
Jobs: m.2

 

With flag:
Reservations:
brian.1x1 User -00:00:15 -> INFINITY ( INFINITY)
Blocked Resources@-00:00:02 Procs: 6/6 (100.00%) Mem: 0/5000 (0.00%)
Blocked Resources@00:04:58 Procs: 6/6 (100.00%) Mem: 0/5000 (0.00%)
m.1x1 Job:Running -00:00:02 -> 00:04:58 (00:05:00)
Jobs: m.1

When you set the FASTRVSSTARTUP flag, Moab will also set the DISABLEPARTIALNODERESERVATIONS flag.

FILELOCKHA This is a High Availability feature. FILELOCKHA prevents scheduling conflicts between multiple Moab servers.
FREECOMPLETEDJOBSUBMITSTRING Moab frees the job submit string for completed jobs, decreasing the amount of memory needed during operation. This is useful in environments with large job scripts that can create a large memory footprint.
IGNOREPIDFILELOCK Moab will not fail if it cannot get a lock on the .moab.pid file. This is useful when Moab is running on a shared filesystem where file locking can be unpredictable.
JOBSUSERSVWALLTIME Allows jobs submitted without a walltime request or default walltime received from a class or queue but with an ADVRES:reservation to inherit their walltime limit from the reservation instead of the Moab default. The job walltime limit is then the remaining time of the reservation to which the job was submitted.
NOCLASSUPDATE While running against TORQUE, Moab will not update classes when it refreshes each iteration. Moab loads the classes at startup, but does not refresh them until the next time it is restarted.
NORMALIZETASKDEFINITIONS

Instructs Moab to normalize all tasks that it receives via an mshow -a command. Moab normalizes the task definition to one processor and then changes the tasks requested to the number of processors requested. For example, when the following is received by Moab:

mshow -a -w mintasks=1@procs:4+mem:4096

It is changed to this:

mshow -a -w mintasks=4@procs:1+,mem:1024,tpn=4
NOVMDESTROYDEPENDENCIES The destroy job a in cloud workflow does not have any dependencies, allowing it to run whenever you cancel the service. For more information about destroy jobs, see DESTROYTEMPLATE and Creating a Cloud Workflow.
OPTIMIZEDBACKFILL On large systems that utilize system-wide reservations, backfill can take a considerable amount of time. This flag speeds up backfill scheduling by using an alternative BETA backfill algorithm. This flag will be the default in future versions of Moab.
OUTOFBANDVMRSV

Causes Moab to put reservations on untracked VMs (VMs that are created outside of Moab) so that the untracked VMs are visible when you use checkjob, mshow, checknode, etc. commands. These reservations are updated each new iteration, so if the VM is migrated or modified outside of Moab, the reservation will accordingly adjust automatically.

The reservations are not checkpointed. The system re-creates them when you restart Moab (new reservation IDs, etc.). If a VM tracking job appears for the VM, then the reservation will be removed.

This scheduler flag is highly recommended for Cloud users. For more information, see Workload-Driven Cloud Services.

PRIORITYPOLICYBLOCKING

By default, a job that violates a policy is placed into the blocked queue. Jobs with a lower priority, but that do not violate the policy, will run. This can lead to situations in which small jobs starve out larger, higher priority jobs.

When you set the PRIORITYPOLICYBLOCKING flag, Moab allows the job that violates the policy to continue consuming the policy slots while it remains blocked. With the policy slots consumed, the smaller, lower priority jobs will not run. The higher priority job will continue to consume the policy slots until it has consumed enough to actually run.

Note that because the blocked job consumes policy slots, this will inevitably lead to lower system utilization.

SHOWREQUESTEDPROCS Shows requested processors regardless of NodeAccessPolicy in showq. When SINGLEJOB NODEACCESSPOLICY is used and the job requests one processor, showq displays the job with one processor.
SHOWUSERJOBSONLY Causes Moab, when a non-admin user runs showq, to return only that user's jobs. If an administrator runs showq when this flag is set,Moab returns the jobs of all users; no restrictions are placed on administrators.
STRICTSPOOLDIRPERMISSIONS Enforces at least a 511 permission on the Moab spool directory.
UNMIGRATEONDEFER Forces Moab to unmigrate a job if it enters a deferred state.