(Click to open topic with navigation)

You are here: Appendices > Appendix L Cray-Specific Power Management and Energy-Consumption-by-Job Accounting > Enabling Moab/Cray Power Management

L.2 Enable Moab/Cray Power Management

Enabling Moab power management for Cray XC systems requires enabling Moab power management using the Moab-only method. It is the same regardless of the type of system Moab will manage. The only difference between systems, such as Cray XC and non-Cray, lies in the commands executed by the Moab-only power management scripts.

L.2.1 Install Moab Power Management Scripts

Adaptive Computing provides working Moab power management “reference” scripts that execute ipmitool commands.

Any HPC site that uses different, usually vendor-specific, commands requires the administrator to modify the reference scripts so they use site-specific commands.

The four reference scripts are located in the $MOABHOMEDIR/tools/ipmi directory, where $MOABHOMEDIR represents the Moab installation directory. The default $MOABHOMEDIR is /opt/moab, which may not match the default location on the Cray XC system (e.g. /var/spool/moab).

L.2.2 Configure Moab Cray-Specific Power Management

Adaptive Computing and Cray have worked together to provide working Cray-specific power management “reference” scripts for Cray XC systems. These scripts are located in the $MOABHOMEDIR/tools/ipmi/cray sub-directory and have the same names as the four ipmitool-based reference scripts.

If your Moab HPC Suite download does not contain the Cray-specific power management scripts, you can obtain them from Adaptive Computing’s Technical Support and install them in the default installation directory as illustrated by the commands below. This directory is where a Moab installation will place them.

cd $MOABHOMEDIR/tools/ipmi
mkdir cray
chmod 755 cray
chgrp root cray  # should match group of tools/ipmi directory
chown moab cray  # should match owner of tools/ipmi directory
cd cray
cp <CraySpecificFileNames>
chmod 755 *.py
chgrp root *.py  # should match group of tools/ipmi/*.py files
chown moab *.py  # should match owner of tools/ipmi/*.py files

L.2.2.A Set up the Moab Cray-Specific Power Management Scripts

  1. Copy the Moab Cray-specific power management reference scripts to a directory where you can customize them and not have the customized version overwritten on a Moab upgrade.
  2. Edit moab.cfg for green computing. See "To enable green computing with just Moab" in Enabling Green Computing.

  3. Set the CLUSTERQUERYURL and NODEPOWERURL parameters to point to the copied, Cray-specific, cluster query (ipmi.mon.py) and node power control (ipmi.power.py) scripts.

    Example: 

    NODECFG[DEFAULT] POWERPOLICY=OnDemand
RMCFG[ipmi] TYPE=NATIVE RESOURCETYPE=PROV
RMCFG[ipmi] CLUSTERQUERYURL=exec://$MOABHOMEDIR/etc/cray/ipmi.mon.py
RMCFG[ipmi] NODEPOWERURL=exec://$MOABHOMEDIR/etc/cray/ipmi.power.py

    This example assumes the $MOABHOMEDIR/etc/cray directory is the location to which the scripts were copied.

  4. Determine which Cray-system architecture to use for Moab and Torque. Moab and Torque can operate in one of two architectures with a Cray system.

    • Internal - Moab and the Torque pbs_server execute on a Cray service node which functions as the Moab head node. This the first (original) architecture.
    • External - Moab and the Torque pbs_server daemon execute on a non-Cray external server which is the Moab head node. This architecture is recommended by Adaptive Computing.

    Image L-1: Internal Architecture

    Image L-2: External Architecture

    Click to enlarge

    Click to enlarge

    The Moab/Torque architecture of your Cray site determines what values you will assign to Python variables in the Cray-specific, power management scripts you must customize.

  5. Edit the config.py file you copied and set the value of the variable assignments as necessary for your architecture.

    Text Line Instructions
    self.homeDir Path to the Moab installation directory.
    self.etcDir Path to the Moab configuration directory.
    self.capmcBinDir Path to the Cray "capmc" command binary image file.
    self.capmcRemote Set to "Y" if Moab and Torque pbs_server are operating in the external architecture; otherwise, set to "N" if they are operating in the internal architecture.
    self.capmcServer Fully-qualified domain name (FQDN) of the server with the "capmc" command binary image file.
    self.pollInterval Set the power state (cluster query) polling interval in seconds to a reasonable value; less than Moab's RMPOLLINTERVAL in the moab.cfg file, at least 60 seconds, etc.
    self.pidFile Leave as is; unless the directory path should be different.
    self.ipmifile Leave as is; unless the directory path should be different.
    self.daemon Path to the power state query daemon script (__mon.ipmi.py).

    os.environ["OS_CACERT"] = …

    os.environ["OS_KEY"] = …

    os.environ["OS_CERT"] = …

    os.environ["OS_SERVICE_URL"] = …

    Set the appropriate values for the four OS_… variables, especially the OS_SERVICE_URL value. These variables have to do with authentication (key file, certificate, etc) for the Cray "capmc" command.
    self.omapFormat If the Cray is part of a Moab "peer-to-peer" grid that uses the Moab "OMAP" feature to rename compute nodes, this variable may require modification to map Cray node id numbers into a remapped node id known to Moab. If so, follow the instructions in the script comments accompanying this variable assignment and modify as appropriate.

L.2.3 Testing Moab Power Management Scripts on Cray XC Systems

After configuring the moab.cfg and config.py files, restart Moab. You should now be able to power on, powered-down Cray XC compute nodes and shut down, powered-on Cray XC compute nodes using the mnodectl -m power=xxx command.

Example: 

mnodectl -m power=off <NodeExpr>
mnodectl -m power=on  <NodeExpr>

“<NodeExpr>” is a compute node host name, a regular expression that matches multiple host names, or a comma-separated list of host names.

If the Moab power management has been correctly configured for Cray-specific operation, executing the commands above for a Cray compute node should shut down the node and then power it up. Ensure you wait sufficient time after powering off the node before powering it up.

Cray XC systems have a minimum time limit between executing commands that power on and power off a compute node to ensure a command does not attempt to interrupt a compute node already executing a previous command. For example, commanding a node to power off and then shortly thereafter commanding it to power on while it is in the process of shutting down is forbidden by the Cray system and if so commanded, may have unknown behavior or consequences. See the Cray XC documentation or contact Cray Technical Support for authoritative information.

L.2.4 Enabling Moab Green Policy on Cray XC Systems

There is no difference between enabling green policy on Cray XC systems versus non-Cray systems. Follow the green policy configuration instructions to enable automated power management via "Green Computing" for a Cray XC system using the Moab-only method.

Ensure the moab.cfg file contains appropriate values for the following parameters to ensure a command does not attempt to interrupt a compute node already executing a previous command. See the warning earlier in this topic for more information regarding the minimum time limit between executing commands that power on and power off a compute node.

© 2016 Adaptive Computing