13.10 Intelligent Platform Management Interface |
|
13.10 Intelligent Platform Management Interface |
|
The Intelligent Platform Management Interface (IPMI) specification defines a set of common interfaces system administrators can use to monitor system health and manage the system. The IPMI interface can monitor temperature and other sensor information, query platform status and power-on/power-off compute nodes. As IPMI operates independently of the node's OS interaction with the node can happen even when powered down. Moab can use IPMI to monitor temperature information, check power status, power-up, power-down, and reboot compute nodes.
IPMI must be enabled on each node in the compute cluster. This is usually done either through the node's BIOS or by using a boot CD containing IPMI utilities provided by the manufacturer. With regard to configuring IPMI on the nodes, be sure to enable IPMI-over-LAN and set a common login and password on all the nodes. Additionally, you must set a unique IP address for each node's BMC. Take note of these addresses as you will need them when reviewing the Creating the IPMI BMC-Node Map File section.
IPMItool is an open-source tool used to retrieve sensor information from the IPMI Baseboard Management Controller (BMC) or to send remote chassis power control commands. The IPMItool developer provides Fedora Core binary packages as well as a source tarball on the IPMItool download page. Download and install IPMItool on the Moab head node and make sure the ipmitool binary is in the current shell PATH.
Proper IPMI setup and IPMItool configuration can be confirmed by issuing the following command on the Moab head node.
> ipmitool -I lan -U username -P password -H BMC IP chassis status
The output of this command should be similar to the following.
System Power : off Power Overload : false Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : previous Last Power Event : Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false
Since the BMC can be controlled via LAN, it is possible for the BMC to have its own unique IP address. Since this IP address is separate from the IP address of the node, a simple mapping file is required for Moab to know each node's BMC address. The file is a flat text file and should be stored in the Moab home directory. If a mapping file is needed, specify the name in the config.ipmi.pl configuration file in the etc/ directory. The following is an example of the mapping file:
#<NodeID> <BMC IP> node01 10.10.10.101 node02 10.10.10.102 node03 10.10.10.103 node04 10.10.10.104 node05 10.10.10.105 # NodeID = the name of the nodes returned with "mdiag -n" # BMC IP = the IP address of the IPMI BMC network interface
Note that only the nodes specified in this file are queried for IPMI information. Also note that the mapping file is disabled by default and the nodes that are returned from Moab with mdiag -n are the ones that are queried for IPMI sensor data.
The tools/ subdirectory in the install directory already contains the Perl scripts needed to interface with IPMI. The following is a list of the Perl scripts that should be in the tools/ directory; confirm these are present and executable.
ipmi.mon.pl # The daemon front-end called by Moab ipmi.power.pl # The power control script called by Moab __mon.ipmi.pl # The IPMI monitor daemon that updates and caches IPMI data from nodes
Next, a few configuration settings need to be adjusted in the config.ipmi.pl file found in the etc subdirectory. The IPMI-over-LAN username and password need to be set to the values that were set in the Node IPMI Configuration section. Also, the IPMI query daemon's polling interval can be modified by adjusting $pollInterval. This specifies how often the IPMI-enabled nodes are queried to retrieve sensor data.
To allow Moab to use the IPMI tools, a native resource manager is configured. To do this, the following lines must be added to moab.cfg:
... # IPMI - Node monitor script RMCFG[ipminative] TYPE=NATIVE CLUSTERQUERYURL=exec://$TOOLSDIR/ipmi.mon.pl ...
Next, the following lines can be added to allow Moab to issue IPMI power commands.
... # IPMI - Power on/off/reboot script RMCFG[ipminative] NODEPOWERURL=exec://$TOOLSDIR/ipmi.power.pl ...
Moab can be configured to perform actions based on sensor data. For example, Moab can shut down a compute node if its CPU temperature exceeds 100 degrees Celsius, or it can power down idle compute nodes if workload is low. Generic event thresholds are used to tell Moab to perform certain duties given certain conditions. The following example is of a way for Moab to recognize it should power off a compute node if its CPU0 temperature exceeds 100 degrees Celsius.
... # IPMI - Power off compute node if its CPU0 temperature exceeds 100 degrees Celsius. GEVENTCFG[CPU0_TEMP>100] action=off ...
Once the preceding steps have been taken, Moab should be started as normal. The IPMI monitoring daemon should start automatically, which can be confirmed with the following:
moab@headnode:~/$ ps aux | grep __mon moab 11444 0.0 0.3 6204 3172 pts/3 S 10:54 0:00 /usr/bin/perl -w /opt/moab/tools/_mon.ipmi.pl --start
After a few minutes, IPMI data should be retrieved and cached. This can be confirmed with the following command:
moab@headnode:~/$ cat spool/ipmicache.gm node01 GMETRIC[CPU0_TEMP]=49 node01 GMETRIC[CPU1_TEMP]=32 node01 GMETRIC[SYS_TEMP]=31 node01 POWER=ON
Finally, issue the following to ensure Moab is grabbing the IPMI data. Temperature data should be present in the Generic Metrics row.
moab@headnode:~/$ checknode node01 node node01 State: Idle (in current state for 00:03:12) Configured Resources: PROCS: 1 MEM: 2000M SWAP: 3952M DISK: 1M Utilized Resources: --- Dedicated Resources: --- Generic Metrics: CPU0_TEMP=42.00,CPU1_TEMP=30.00,SYS_TEMP=29.00 ...
Copyright © 2012 Adaptive Computing Enterprises, Inc.®