Moab Workload Manager

Moab-Loadleveler Integration Guide

Overview

Moab can be used as an external scheduler for Loadleveler. In this configuration, Loadleveler manages the job queue and the compute resources while Moab queries the Loadleveler negotiator via the Loadleveler data API to obtain up to date job and node information. Using this information, Moab directs Loadleveler to manage jobs in accordance with specified Moab policies, priorities, and reservations.

LoadLeveler Configuration

Moab drives LL via the Loadleveler scheduling API. To enable this API and thus the external scheduler, the following steps must be taken:

- set 'SCHEDULER_API=yes' in the 'LoadL_config' file typically located in the user 'loadl' home directory.

- set the 'NEGOTIATOR_REMOVE_COMPLETED' parameter (also located in the 'LoadL_config' file) to a value of at least 5 minutes, ie 'NEGOTIATOR_REMOVE_COMPLETED=300'. (This allows Moab to obtain job info from LL required to maintain accurate job statistics)

- AGGREGATE_ADAPTERS should be set to NO in the LoadL_config file.

- recycle negotiator using the command 'llctl recycle' on the central manager node.

Moab Configuration

To enable Moab to communicate with the Loadleveler negotiator daemon, the RMCFG parameter must be set with a TYPE of LL. By default, this should already be done for you automatically by the configure script.

Because only a subset of LoadLeveler command file keywords can be interpreted by Moab, the parameter INSTANTSTAGE should be used when jobs are submitted through msub.

Issues

The Loadleveler scheduling API is not event driven so Moab has no way of knowing when a new job is submitted. Under these conditions, it will not evaluate a newly submitted job until its next scheduling iteration, typically within 15 to 30 seconds. This lag can be removed by utilizing Loadleveler's 'SUBMITFILTER'. The Moab command mschedctl -r 2 can be added as the last statement in this filter causing Moab to 'wake-up' and attempt to schedule new jobs immediately. The mschedctl command is a administrative command and so may need an suid wrapper in order to allow use by non-privileged users. (see example).

Note: Do NOT use the above submit filter when jobs will be submitted using msub or via Moab Access Portal.

Note: You can return to Loadleveler default scheduling at any time by setting 'SCHEDULER_API=no' in the LoadL_config file and re-issuing the 'llctl recycle' command.

Moab supports interactive job hostlists but these hostlists must currently be specified using the network interface Loadleveler utilizes. For example, an SP node may have two names, node001e and node001sw representing its ethernet and switch interfaces respectively. Loadleveler is configured to communicate with the nodes on one of these interfaces. (This can be determined by issuing 'llstatus' and observing the name used to specify each node.) Interactive job hostlists must be specified using the same interface that Loadleveler is configured to use. Efforts are underway to extend Moab interface tracking to remedy this.

Note: The LoadLeveler API is not thread safe, therefore, do not build Moab with __MTHREAD enabled.

Note: Some releases of Loadleveler will requeue all active jobs when reconfigured to use the external scheduler interface. In such cases, it may be best to drain the queue before enabling Moab.

Note: If using Loadleveler with Moab Access Portal or with a Moab Peer Based Grid, the parameter INSTANTSTAGE must be set.