Moab interfaces to PBS by utilizing a few PBS libraries and include files. If you have a non-standard PBS installation, you may need to modify Makefile and change PBSIP and PBSLP values and references as necessary for your local site configuration.
The configure script automatically sets up Moab so that the user running configure will become the default Primary Moab Administrator ($MOABADMIN). This can be changed by modifying the 'ADMINCFG[1] USERS= <USERNAME>' line in the Moab configuration file (moab.cfg). The primary administrator is the first user listed in the USERS attribute and is the ID under which the Moab daemon runs.
Some Tru64 and IRIX systems have a local libnet library that conflicts with PBS's libnet library. To resolve this, try setting PBSLIB to '${PBSLIBDIR}/libnet.a -lpbs' in the Moab Makefile.
Moab is 64-bit compatible. If PBS/TORQUE is running in 64-bit mode, Moab likewise needs to be built in this manner to use the PBS scheduling API (i.e., for IRIX compilers, add '-64' to OSCCFLAGS and OSLDFLAGS variables in the Makefile).
moab.cfg
> qmgr -c 'set server keep_completed = 300'
PBS nodes can be configured as time shared or space shared according to local needs. In almost all cases, space shared nodes provide the desired behavior.
PBS/TORQUE supports the concept of virtual nodes. Using this feature, Moab can individually schedule processors on SMP nodes. The online TORQUE documentation describes how to set up the '$PBS_HOME/server_priv/nodes' file to enable this capability. (For example, <NODENAME> np=<VIRTUAL NODE COUNT>)
Do not start the pbs_sched daemon. This is the default scheduler for PBS/TORQUE; Moab provides this service.
Moab uses PBS's scheduling port to obtain real-time event information from PBS regarding job and node transitions. Leaving the default qmgr setting of 'set server scheduling=True' allows Moab to receive and process this real-time information.
PBSPro 7.x, 8.x, and higher require that the pbs_sched daemon execute for proper operation, but PBS must be configured to take no indpenedent action that conflicts with Moab. With these PBSPro releases, sites should allow pbs_sched to run after putting the following PBS configuration in place:
qmgr configuration
> qmgr -c 'set server scheduling = false' > qmgr -c 'set server scheduler_iteration = 100000000' > qmgr -c 'unset server node_fail_requeue'
sched_priv/sched_config
preemptive_sched: false ALL
By default, Moab automatically interfaces with TORQUE/PBS when it is installed. Consequently, in most cases, the following steps are not required:
Moab's user interface port is set using the SCHEDCFG parameter and is used for user-scheduler communication. This port must be different from the PBS scheduler port used for resource manager-scheduler communication.
Moab supports basic scheduling of all PBS node specifications.
Moab is, by default, liberal in its interpretation of <NODECOUNT>:PPN=<X>. In its standard configuration, Moab interprets this as 'give the job <NODECOUNT>*<X> tasks with AT LEAST <X> tasks per node'. Set the JOBNODEMATCHPOLICY parameter to EXACTNODE to have Moab support PBS's default allocation behavior of <NODECOUNT> nodes with exactly <X> tasks per node.
Moab Features Not Supported by PBS
PBS does not support the concept of a job QoS or other extended scheduling features by default. This can be handled using the techniques described in the PBS Resource Manager Extensions section. See the Resource Manager Extensions Overview for more information.
Some Versions of PBS Do Not Maintain Job Completion Information
An external scheduler cannot determine if the job completed successfully or if internal PBS problems occurred preventing the job from being properly updated. This problem will not in any way affect proper scheduling of jobs but may potentially affect scheduler statistics. If your site is prone to frequent PBS hangs, you may want to set the Moab JOBPURGETIME parameter to allow Moab to hold job information in memory for a period of time until PBS recovers. (Note: It is not recommended that PURGETIME be set to over 2:00).
On TRU64 systems, the PBS 'libpbs' library does not properly export a number of symbols required by Moab. This can be worked around by modifying the Moab Makefile to link the PBS 'rm.o' object file directly into Moab.