Integrating an xCAT Physical Provisioning Resource Manager with Moab
- xCAT Configuration Requirements
- MSM Installation
- Integrating MSM and xCAT
- MSM Configuration
- Configuration Validation
- Troubleshooting
- Deploying Images with TORQUE
- Installing Moab on the Management Node
- Moab Configuration File Example
- Verifying the Installation
- xCAT Plug-in Configuration Parameters
Introduction
Moab can dynamically provision compute machines to requested operating systems and power off compute machines when not in use. Moab can intelligently control xCAT and use its advanced system configuration mechanisms to adapt systems to current workload requirements. Moab communicates with xCAT using the Moab Service Manager (MSM). MSM is a translation utility that resides between Moab and xCAT and acts as aggregator and interpreter. The Moab Workload Manager will query MSM, which in turn queries xCAT, about system resources, configurations, images, and metrics. After learning about these resources from MSM, Moab then makes intelligent decisions about the best way to maximize system utilization.
In this model Moab gathers system information from two resource managers. The first is TORQUE, which handles the workload on the system; the second is MSM, which relays information gathered by xCAT. By leveraging these software packages, Moab intelligently adapts clusters to deliver on-site goals.
This document assumes that xCAT has been installed and configured. It describes the process of getting MSM and xCAT communicating, and it offers troubleshooting guidance for basic integration. This document offers a description for how to get Moab communicating with MSM and the final steps in verifying a complete software stack.
xCAT Configuration Requirements
Observe the following xCAT configuration requirements before installing MSM:
- Configure xCAT normally for your site.
- Test the following commands to verify proper function:
- rpower
- nodeset
- makedhcp
- makedns
- nodestat
- rvitals
- If MSM will run on a different machine than the one on which xCAT runs, install the xCAT client packages on that machine, and test the previously listed commands on that machine as well.
- Configure and test all stateful/stateless images you intend to use.
- Configure xCAT to use either PostgreSQL or MySQL. Note that the default of SQLite may not function properly when MSM drives xCAT.
- PostgreSQL: See xCATSetupPostgreSQL.pdf for more information.
- MySQL: See xCAT2.SetupMySQL.pdf for more information.
 |
You must have a valid Moab license file (moab.lic) with provisioning and green enabled. For information on acquiring an evaluation license, please contact info@adaptivecomputing.com. |
MSM Installation
- Determine the installation directory (usually /opt/moab/tools/msm)
- Untar the MSM tarball into the specified directory (making it the MSM home directory, or $MSMHOMEDIR)
- Verify the required Perl modules and version are available
perl -e 'use Storable 2.18' perl -MXML::Simple -e 'exit' perl -MProc::Daemon -e 'exit' perl -MDBD::SQLite -e 'exit'
Integrating MSM and xCAT
Copy the x_msm table schema to the xCAT schema directory:
> cp $MSMHOMEDIR/contrib/xcat/MSM.pm $XCATROOT/lib/perl/xCAT_schema
Restart xcatd and check the x_msm table is correctly created:
> service xcatd restart
> tabdump x_msm
Prepare xCAT images and ensure they provision correctly (see xCAT documentation)
Populate the x_msm table with your image definitions:
> tabedit x_msm #flavorname,arch,profile,os,nodeset,features,vmoslist,hvtype,hvgroupname,vmgroupname,comments,disable "compute","x86_64","compute","centos5.3","netboot","torque",,,,,, "science","x86","compute","scientific_linux","netboot","torque",,,,,,
- flavorname - A user specified name for the image and settings; also an xCAT group name, nodes are added to this group when provisioned
- arch - Architecture as used by xCAT
- profile - Profile as used by xCAT
- os - Operating system as used by xCAT
- nodeset - One of netboot|install|statelite
- features - Names of xCAT groups that identify special hardware features ('torque' and 'paravirt' are special cases)
- vmoslist - Note: Not used. List of flavorname's this image may host as VMs (hypervisor images only)
- hvtype - Note: Not used. One of esx|xen|kvm (hypervisor images only)
- hvgroupname - Note: Not used. Name of xCAT group nodes will be added to when provisioned to this image
- vmgroupname - Note: Not used. Name of xCAT group VMs will be added to when hosted on a hypervisor of this image
- comments - User specified comments
- disable - Flag to temporarily disable use of this image
Ensure all xCAT group names in the x_msm table exist in the xCAT nodegroup table
> tabedit nodegroup
Edit as necessary to simulate the following example:
#groupname,grouptype,members,wherevals,comments,disable "compute",,,,, "esxi4",,,,, "esxhv",,,,, "esxvmmgt",,,,,
After making any necessary edits, run the following command:
> nodels compute,esxi4,esxhv,esxvmmgt # should complete without error, ok if doesn't return anything
MSM Configuration
Edit $MSMHOMEDIR/msm.cfg and configure the xCAT plug-in. Below is a generic example for use with TORQUE without virtualization. See the section on configuration parameters for a complete list of parameters and descriptions.
# MSM configuration options RMCFG[msm] PORT=24603 RMCFG[msm] POLLINTERVAL=45 RMCFG[msm] LOGFILE=/opt/moab/log/msm.log RMCFG[msm] LOGLEVEL=8 RMCFG[msm] DEFAULTNODEAPP=xcat # xCAT plugin specific options APPCFG[xcat] DESCRIPTION="xCAT plugin" APPCFG[xcat] MODULE=Moab::MSM::App::xCAT APPCFG[xcat] LOGLEVEL=3 APPCFG[xcat] POLLINTERVAL=45 APPCFG[xcat] TIMEOUT=3600 APPCFG[xcat] _USEOPIDS=0 APPCFG[xcat] _NODERANGE=moab,esxcompute APPCFG[xcat] _USESTATES=boot,netboot,install APPCFG[xcat] _LIMITCLUSTERQUERY=1 APPCFG[xcat] _RPOWERTIMEOUT=120 APPCFG[xcat] _DONODESTAT=1 APPCFG[xcat] _REPORTNETADDR=1 APPCFG[xcat] _CQXCATSESSIONS=4
Configuration Validation
Set up environment to manually call MSM commands:
# substitute appropriate value(s) for path(s) export MSMHOMEDIR=/opt/moab/tools/msm export MSMLIBDIR=/opt/moab/tools/msm export PATH=$PATH:/$MSMLIBDIR/contrib:$MSMLIBDIR/bin
Verify that MSM starts without errors:
> msmd
Verify that the expected nodes are listed, without errors, using the value of _NODERANGE from msm.cfg.
> nodels <_NODERANGE>
Verify that the expected nodes, are listed in the cluster query output from MSM:
> cluster.query.pl
Provision all nodes through MSM for the first time (pick and image name from x_msm):
> for i in `nodels <_NODERANGE>; do node.modify.pl $i --set os=<image_name>;done
Verify the nodes correctly provision and that the correct OS is reported (which may take some time after the provisioning requests are made):
> cluster.query.pl
Troubleshooting
- msmctl -a does not report the xCAT plugin - Check the log file (path specified in msm.cfg) for error messages. A common cause is missing Perl modules (Storable, DBD::SQLite, xCAT::Client).
- cluster.query.pl does not report any nodes - Check that the xCAT command 'nodels <noderange>', where <noderange> is the value configured for _NODERANGE in msm.cfg, outputs the nodes expected.
- cluster.query.pl does not report OS - MSM must provision a node to recognize what the current operating system is. It is not sufficient to look up the values in the nodetype table because MSM has no way of recognizing whether nodeset and rpower were run with the current values in the nodetype table.
- cluster.query.pl does not report OSLIST, or does not report the expected OSLIST for a node - Check that the node belongs to the appropriate groups, particularly any listed in the features field of the x_msm table for the missing image name.
Deploying Images with TORQUE
When using MSM + xCAT to deploy images with TORQUE, there are some special configuration considerations. Most of these also apply to other workload resource managers.
Note that while the MSM xCAT plugin contains support for manipulating TORQUE directly, this is not an ideal solution. If you are using a version of xCAT that supports prescripts, it is more appropriate to write prescripts that manipulate TORQUE based on the state of the xCAT tables. This approach is also applicable to other workload resource managers, while the xCAT plugin only deals with TORQUE.
Several use cases and configuration choices are discussed in what follows.
Each image should be configured to report its image name through TORQUE. In the TORQUE pbs_mom mom_config file the "opsys" value should mirror the name of the image. See Node Manager (MOM) Configuration in the TORQUE Administrator's Guide for more information.
Installing Moab on the Management Node
Moab is the intelligence engine that coordinates the capabilities of xCAT and TORQUE to dynamically provision compute nodes to the requested operating system. Moab also schedules workload on the system and powers off idle nodes. Download and install Moab.
Moab Configuration File Example
Moab stores its configuration in the moab.cfg file: /opt/moab/moab.cfg. A sample configuration file, set up and optimized for adaptive computing follows:
# Example moab.cfg SCHEDCFG[Moab] SERVER=gpc-sched:42559 ADMINCFG[1] USERS=root,egan LOGLEVEL 7 # How often (in seconds) to refresh information from Torque and MSM RMPOLLINTERVAL 60 RESERVATIONDEPTH 10 DEFERTIME 0 ################################################################### # Location of msm directory # # www.adaptivecomputing.com/moabdocs/a.fparameters.php#toolsdir # ################################################################### TOOLSDIR /opt/moab/tools ############################################################################### # TORQUE and MSM configuration # # http://www.adaptivecomputing.com/resources/docs/mwm/a.fparameters.php#rmcfg # ############################################################################### RMCFG[torque] TYPE=PBS RMCFG[msm] TYPE=NATIVE:msm FLAGS=autosync,NOCREATERESOURCE RESOURCETYPE=PROV RMCFG[msm] TIMEOUT=60 RMCFG[msm] PROVDURATION=10:00 AGGREGATENODEACTIONS TRUE ############################################################################### # ON DEMAND PROVISIONING SETUP # # www.adaptivecomputing.com/moabdocs/3.5credoverview.php#qos # # www.adaptivecomputing.com/moabdocs/5.2nodeallocation.php#PRIORITY # # www.adaptivecomputing.com/moabodcs/a.fparameters.php#jobprioaccrualpolicy # ############################################################################### QOSCFG[od] QFLAGS=PROVISION USERCFG[DEFAULT] QLIST=od NODEALLOCATIONPOLICY PRIORITY NODECFG[DEFAULT] PRIORITYF=1000*OS+1000*POWER NODEAVAILABILITYPOLICY DEDICATED CLASSCFG[DEFAULT] DEFAULT.OS=scinetcompute ############################################################### # GREEN POLICIES # # www.adaptivecomputing.com/moabdocs/18.0greencomputing.php # ############################################################### NODECFG[DEFAULT] POWERPOLICY=ONDEMAND PARCFG[ALL] NODEPOWEROFFDURATION=20:00 NODEIDLEPOWERTHRESHOLD 600 # END Example moab.cfg
Verifying the Installation
When Moab starts it immediately communicates with its configured resource managers. In this case Moab communicates with TORQUE to get compute node and job queue information. It then communicates with MSM to determine the state of the nodes according to xCAT. It aggregates this information and processes the jobs discovered from TORQUE.
When a job is submitted, Moab determines whether nodes need to be provisioned to a particular operating system to satisfy the requirements of the job. If any nodes need to be provisioned Moab performs this action by creating a provisioning system job (a job that is internal to Moab). This system job communicates with xCAT to provision the nodes and remain active while the nodes are provisioning. Once the system job has provisioned the nodes it informs the user’s job that the nodes are ready at which time the user’s job starts running on the newly provisioned nodes.
When a node has been idle for a specified amount of time (see NODEIDLEPOWERTHRESHOLD), Moab creates a power-off system job. This job communicates with xCAT to power off the nodes and remain active in the job queue until the nodes have powered off. Then the system job informs Moab that the nodes are powered off but are still available to run jobs. The power off system job then exits.
To verify correct communication between Moab and MSM run the mdiag -R –v msm command.
$ mdiag -R -v msm diagnosing resource managers RM[msm] State: Active Type: NATIVE:MSM ResourceType: PROV Timeout: 30000.00 ms Cluster Query URL: $HOME/tools/msm/contrib/cluster.query.xcat.pl Workload Query URL: exec://$TOOLSDIR/msm/contrib/workload.query.pl Job Start URL: exec://$TOOLSDIR/msm/contrib/job.start.pl Job Cancel URL: exec://$TOOLSDIR/msm/contrib/job.modify.pl Job Migrate URL: exec://$TOOLSDIR/msm/contrib/job.migrate.pl Job Submit URL: exec://$TOOLSDIR/msm/contrib/job.submit.pl Node Modify URL: exec://$TOOLSDIR/msm/contrib/node.modify.pl Node Power URL: exec://$TOOLSDIR/msm/contrib/node.power.pl RM Start URL: exec://$TOOLSDIR/msm/bin/msmd RM Stop URL: exec://$TOOLSDIR/msm/bin/msmctl?-k System Modify URL: exec://$TOOLSDIR/msm/contrib/node.modify.pl Environment: MSMHOMEDIR=/home/wightman/test/scinet/tools//msm;MSMLIBDIR=/home/wightman/test/scinet/tools//msm Objects Reported: Nodes=10 (0 procs) Jobs=0 Flags: autosync Partition: SHARED Event Management: (event interface disabled) RM Performance: AvgTime=0.10s MaxTime=0.25s (38 samples) RM Languages: NATIVE RM Sub-Languages: -
To verify nodes are configured to provision use the checknode -v
$ checknode n01 node n01 State: Idle (in current state for 00:00:00) Configured Resources: PROCS: 4 MEM: 1024G SWAP: 4096M DISK: 1024G Utilized Resources: --- Dedicated Resources: --- Generic Metrics: watts=25.00,temp=40.00 Power Policy: Green (global policy) Selected Power State: Off Power State: Off Power: Off MTBF(longterm): INFINITY MTBF(24h): INFINITY Opsys: compute Arch: --- OS Option: compute OS Option: computea OS Option: gpfscompute OS Option: gpfscomputea Speed: 1.00 CPULoad: 0.000 Flags: rmdetected RM[msm]: TYPE=NATIVE:MSM ATTRO=POWER EffNodeAccessPolicy: SINGLEJOB Total Time: 00:02:30 Up: 00:02:19 (92.67%) Active: 00:00:11 (7.33%)
To verify nodes are configured for Green power management, run the mdiag –G command. Each node will show its power state.
$ mdiag -G NOTE: power management enabled for all nodes Partition ALL: power management enabled Partition NodeList: Partition local: power management enabled Partition NodeList: node n01 is in state Idle, power state On (green powerpolicy enabled) node n02 is in state Idle, power state On (green powerpolicy enabled) node n03 is in state Idle, power state On (green powerpolicy enabled) node n04 is in state Idle, power state On (green powerpolicy enabled) node n05 is in state Idle, power state On (green powerpolicy enabled) node n06 is in state Idle, power state On (green powerpolicy enabled) node n07 is in state Idle, power state On (green powerpolicy enabled) node n08 is in state Idle, power state On (green powerpolicy enabled) node n09 is in state Idle, power state On (green powerpolicy enabled) node n10 is in state Idle, power state On (green powerpolicy enabled) Partition SHARED: power management enabled
To submit a job that dynamically provisions compute nodes, run the msub –l os=<image> command.
$ msub -l os=computea job.sh
yuby.3
$ showq
active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME
provision-4 root Running 8 00:01:00 Fri Jun 19 09:12:56
1 active job 8 of 40 processors in use by local jobs (20.00%)
2 of 10 nodes active (20.00%)
eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
yuby.3 wightman Idle 8 00:10:00 Fri Jun 19 09:12:55
1 eligible job
blocked jobs-----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 blocked jobs
Total jobs: 2Notice that Moab created a provisioning system job named provision-4 to provision the nodes. When provision-4 detects that the nodes are correctly provisioned to the requested OS, the submitted job yuby.3 runs:
$ showq
active jobs------------------------
JOBID USERNAME STATE PROCS REMAINING STARTTIME
yuby.3 wightman Running 8 00:08:49 Fri Jun 19 09:13:29
1 active job 8 of 40 processors in use by local jobs (20.00%)
2 of 10 nodes active (20.00%)
eligible jobs----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 eligible jobs
blocked jobs-----------------------
JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME
0 blocked jobs
Total job: 1The checkjob command shows information about the provisioning job as well as the submitted job. If any errors occur, run the checkjob –v <jobid> command to diagnose failures.
xCAT Plug-in Configuration Parameters
- Description
- Module
- LogLevel
- PollInterval
- TimeOut
- _NodeRange
- _CQxCATSessions
- _DORVitals
- _PowerString
- _DoNodeStat
- _DoxCATStats
- _LockDir
- _HVxCATPasswdKey
- _FeatureGroups
- _DefaultVMCProc
- _DefaultVMDisk
- _DefaultVMCMemory
- _KVMStoragePath
- _ESXStore
- _ESXCFGPath
- _VMInterfaces
- _XenHostInterfaces
- _KVMHostInterfaces
- _VMSovereign
- _UseStates
- _ImagesTabName
- _VerifyRPower
- _RPowerTimeOut
- _QueueRPower
- _RPowerQueueAge
- _RPowerQueueSize
- _MaskOSWhenOff
- _ModifyTORQUE
- _ReportNETADDR
- _UseOpIDs
- _VMIPRange
- _xCATHost
Plugin parameters that begin with an underscore character are specific to the xCAT plug-in; others are common to all plug-ins and may either be set in the RMCFG[msm] for all plug-ins, or per plug-in in the APPCFG[<plugin_name>].