You are here: 2 Scheduler Basics > Configuring the Scheduler

2.6 Configuring the Scheduler

Adjusting Server Behavior

Scheduler configuration is maintained using the flat text configuration file moab.cfg. All configuration file entries consist of simple <PARAMETER> <VALUE> pairs that are whitespace delimited. Parameter names are not case sensitive but <VALUE> settings are. Some parameters are array values and should be specified as <PARAMETER>[<INDEX>] (Example: QOSCFG[hiprio] PRIORITY=1000); the <VALUE> settings may be integers, floats, strings, or arrays of these. Some parameters can be specified as arrays wherein index values can be numeric or alphanumeric strings. If no array index is specified for an array parameter, an index of zero (0) is assumed. The example below includes both array based and non-array based parameters:

SCHEDCFG[cluster2] SERVER=head.c2.org MODE=NORMAL
LOGLEVEL 6
LOGDIR   /var/tmp/moablog

See the parameters documentation for information on specific parameters.

The moab.cfg file is read when Moab is started up or recycled. Also, the mschedctl -m command can be used to reconfigure the scheduler at any time, updating some or all of the configurable parameters dynamically. This command can be used to modify parameters either permanently or temporarily. For example, the command mschedctl -m LOGLEVEL 3will temporarily adjust the scheduler log level. When the scheduler restarts, the log level restores to the value stored in the Moab configuration files. To adjust a parameter permanently, the option --flags=persistent should be set.

At any time, the current server parameter settings may be viewed using the mschedctl -l command.

2.6.1 Adjusting Server Behavior

Most aspects of Moab behavior are configurable. This includes both scheduling policy behavior and daemon behavior. In terms of configuring server behavior, the following realms are most commonly modified.

2.6.1.A Logging

Moab provides extensive and highly configurable logging facilities controlled by parameters.

Parameter	Description
LOGDIR	Indicates directory for log files.
LOGFACILITY	Indicates scheduling facilities to track.
LOGFILE	Indicates path name of log file.
LOGFILEMAXSIZE	Indicates maximum size of log file before rolling.
LOGFILEROLLDEPTH	Indicates maximum number of log files to maintain.
LOGLEVEL	Indicates verbosity of logging.

2.6.1.B Checkpointing

Moab checkpoints its internal state. The checkpoint file records statistics and attributes for jobs, nodes, reservations, users, groups, classes, and almost every other scheduling object.

Parameter	Description
CHECKPOINTEXPIRATIONTIME	Indicates how long unmodified data should be kept after the associated object has disappeared; that is, job priority for a job no longer detected.
CHECKPOINTFILE	Indicates path name of checkpoint file.
CHECKPOINTINTERVAL	Indicates interval between subsequent checkpoints.

2.6.1.C Client Interface

The Client interface is configured using the SCHEDCFG parameter. Most commonly, the attributes SERVER and PORT must be set to point client commands to the appropriate Moab server. Other parameters such as CLIENTTIMEOUT may also be set.

2.6.1.D Scheduler Mode

The scheduler mode of operation is controlled by setting the MODE attribute of the SCHEDCFG parameter. The following modes are allowed:

Mode	Description
INTERACTIVE	Moab interactively confirms each scheduling action before taking any steps. (See interactive mode overview for more information.)
MONITOR	Moab observes cluster and workload performance, collects statistics, interacts with allocation management services, and evaluates failures, but it does not actively alter the cluster, including job migration, workload scheduling, and resource provisioning. (See monitor mode overview for more information.)
NORMAL	Moab actively schedules workload according to mission objectives and policies; it creates reservations; starts, cancels, preempts, and modifies jobs; and takes other scheduling actions.
SIMULATION	Moab obtains workload and resource information from specified simulation trace files and schedules the defined virtual environment.
SINGLESTEP	Moab behaves as in NORMAL mode but will only schedule a single iteration and then exit.
SLAVE	Moab behaves as in NORMAL mode but will only start a job when explicitly requested by a trusted grid peer service or administrator.
TEST	Moab behaves as in NORMAL mode, will make reservations, and scheduling decisions, but will then only log scheduling actions it would have taken if running in NORMAL mode. In most cases, "TEST" mode is identical to MONITOR mode. (See test mode overview for more information.)

2.6.1.E Configuring a job ID offset

Moab assigns job IDs as integers in numeric order as jobs are submitted, starting with 1. In some situations, you might want to offset the integer at which Moab starts to assign job IDs in your system.

This example describes how you would offset the job IDs in a compound system consisting of Site A, Site B, and Site C, each of which runs its own instance of Moab. Users belonging to any of the sites can submit jobs to their own site and to the other two. To simplify aggregation of usage records from the three sites, offset the job IDs for Site B to a starting value higher than the expected total lifetime value for the system; in this example, to 20000000. Likewise, set Site C to 20,000,000 more, or 40000000. To do so, set the MINJOBID attribute of SCHEDCFG in each system's moab.cfg to the offset value. To ensure that Moab will never use the same job ID for two different sites, also set MAXJOBID. If the Moab job naming process ever reaches the MAXJOBID, it will start over again with the MINJOBID.

SCHEDCFG[moab] SERVER=moab_siteA:4244 MAXJOBID=19999999

SCHEDCFG[moab] SERVER=moab_siteB:4344 MINJOBID=20000000 MAXJOBID=39999999

SCHEDCFG[moab] SERVER=moab_siteC:4444 MINJOBID=40000000 MAXJOBID=59999999

When users submit jobs to Moab using msub, Moab selects the job ID in numeric order, starting with 1 in Site A, 20000000 in Site B, and 40000000 in Site C.

If the compound system in this example uses Torque as its resource manager and users submit jobs directly to Torque using qsub, Torque assigns the job ID instead of Moab. In this case, you should also offset the Torque job IDs by setting the next_job_number server parameter of Site B and Site C to 20000000 and 40000000, respectively.

$user qmgr "set server next_job_number=20000000"

$user qmgr "set server next_job_number=40000000"

Torque job ID limits will allow you to use the 20,000,000 offset scheme for up to 4 sites.