Moab-TORQUE/PBS Integration Guide

Appendices > Appendix G: Integrating Other Resources with Moab > Compute Resource Managers > Moab-TORQUE/PBS Integration Guide

Conventions

Moab-TORQUE Integration Guide

Overview
Integration Steps
Current Limitations
Troubleshooting

Install TORQUE

Install TORQUE

Keep track of the PBS target directory, $PBSTARGDIR

Install Moab

Untar the Moab distribution file.
Change the directory to the moab-<version> directory.
Run ./configure.
Specify the PBS target directory ($PBSTARGDIR from step 2.1) when queried by ./configure.

Moab interfaces to PBS by utilizing a few PBS libraries and include files. If you have a non-standard PBS installation, you may need to modify Makefile and change PBSIP and PBSLP values and references as necessary for your local site configuration.

The ./configure script automatically sets up Moab so that the user running configure will become the default Primary Moab Administrator ($MOABADMIN). This can be changed by modifying the ADMINCFG[1] USERS=<USERNAME> line in the Moab configuration file (moab.cfg). The primary administrator is the first user listed in the USERS attribute and is the ID under which the Moab daemon runs.

Some Tru64 and IRIX systems have a local libnet library that conflicts with PBS's libnet library. To resolve this, try setting PBSLIB to '${PBSLIBDIR}/libnet.a -lpbs' in the Moab Makefile.

Moab is 64-bit compatible. If PBS/TORQUE is running in 64-bit mode, Moab likewise needs to be built in this manner to use the PBS scheduling API (i.e., for IRIX compilers, add -64 to OSCCFLAGS and OSLDFLAGS variables in the Makefile).

General Configuration for All Versions of TORQUE

Make $MOABADMIN a PBS admin.
- By default, Moab only communicates with the pbs_server daemons and the $MOABADMIN should be authorized to talk to this daemon (See suggestions for more information.).
(OPTIONAL) Set default PBS queue, nodecount, and walltime attributes (See suggestions for more information.).
(OPTIONAL - TORQUE Only) Configure TORQUE to report completed job information by setting the qmgrkeep_completed parameter:>
```
> qmgr -c 'set server keep_completed = 300'
```

PBS nodes can be configured as time shared or space shared according to local needs. In almost all cases, space shared nodes provide the desired behavior.

PBS/TORQUE supports the concept of virtual nodes. Using this feature, Moab can individually schedule processors on SMP nodes. The online TORQUE documentation describes how to set up the $PBS_HOME/server_priv/nodes file to enable this capability. (For example, <NODENAME> np=<VIRTUAL NODE COUNT>)

Version-Specific Configuration for TORQUE

Do not start the pbs_sched daemon. This is the default scheduler for TORQUE; Moab provides this service.

Moab uses PBS's scheduling port to obtain real-time event information from PBS regarding job and node transitions. Leaving the default qmgr setting of set server scheduling=True allows Moab to receive and process this real-time information.

Configure Moab

By default, Moab automatically interfaces with TORQUE/PBS when it is installed. Consequently, in most cases, the following steps are not required:

Specify PBS as the primary resource manager by setting RMCFG[base] TYPE=PBS in the Moab configuration file (moab.cfg).

If a non-standard PBS installation/configuration is being used, additional Moab parameters may be required to enable the Moab/PBS interface as in the line RMCFG[base] HOST=$PBSSERVERHOST PORT=$PBSSERVERPORT. See the Resource Manager Overview for more information.

Moab's user interface port is set using the SCHEDCFG parameter and is used for user-scheduler communication. This port must be different from the PBS scheduler port used for resource manager-scheduler communication.

TORQUE/Moab Considerations

The default meaning of a node for TORQUE and Moab are not the same. By default, a node is a host in TORQUE. The node may have one or more execution slots (procs) allocated to it in the $TORQUE_HOME/server_priv/nodes file. However, the number of nodes recognized by TORQUE is equivalent to the number of node entries in the $TORQUE_HOME/server_priv/nodes file. A node specification from qsub such as -1 nodes=2:ppn=2 will direct TORQUE to allocate to execution slots on two separate nodes.

Moab is more liberal in its interpretations of a node. To Moab, the qsub request above would be interpreted to mean allocate four tasks with at least two tasks on a node. Where TORQUE would require two nodes for the request, Moab will place all four tasks on the name node (host) if four execution slots are available.

If a cluster has four nodes with eight processors each, TORQUE still sees only four nodes. Moab sees 32 nodes. However, if a user made a qsub request with -1 nodes=10, TORQUE would reject the request because there are only four nodes available. To enable TORQUE to accommodate Moab's more liberal node interpretation, the server parameter available_resources.nodect can be set as a server parameter in TORQUE. The value of available_resources.nodect should equal at least the number of execution slots in the cluster.

For our example, cluster available_resources.nodect should be 32. With this parameter set, the user can now make a request such as -1 nodes=8:ppn=2. In this example, the user is still limited to a maximum node request of 32.

With available_resources.nodect set in TORQUE, Moab can be directed to honor the default TORQUE behavior by setting JOBNODEMATCHPOLICY to EXACTNODE.

PBS Features Not Supported by Moab

Moab supports basic scheduling of all PBS node specifications.

Moab Features Not Supported by PBS

PBS does not support the concept of a job QoS or other extended scheduling features by default. This can be handled using the techniques described in the PBS Resource Manager Extensions section. See the Resource Manager Extensions Overview for more information.

Troubleshooting

On TRU64 systems, the PBS libpbs library does not properly export a number of symbols required by Moab. This can be worked around by modifying the Moab Makefile to link the PBS rm.o object file directly into Moab.