Appendices > Appendix G: Integrating Other Resources with Moab > Compute Resource Managers > Installation Notes for Moab and TORQUE on the Cray XT

Conventions

Installation Notes for Moab and TORQUE for Cray

Overview

Moab and TORQUE can be used to manage the batch system for Cray. This document describes how to configure Moab and TORQUE to bring Moab's unmatched scheduling capabilities to the Cray.

New to TORQUE 4.1, TORQUE now handles all communication with ALPS, specifically the pbs_mom. Previously, communication with ALPS was handled by a combination of Moab, scripts and TORQUE. In the new model, Moab treats TORQUE as a regular TORQUE cluster without any special configuration. TORQUE now uses an extra MOM called the alps_reporter MOM to communicate with ALPS regarding configured and available resources. From the information reported by the alps_reporter mom, TORQUE creates a virtual node for each Cray compute node. Previously, TORQUE only reported the login nodes.

Note: For clarity this document assumes that your SDB node is mounting a persistent /var filesystem from the bootnode. If you have chosen not to use persistent /var filesystems please be aware that the instructions below would have to be modified for your situation.

Upgrade Notes

When upgrading to TORQUE 4.1.0 and using the new Cray model as described in this document, there should be no running jobs. Jobs may be queued but not running.

Installing TORQUE on a Cray

These instructions are written for a partitioned system, with separate SDB and boot nodes. A combined SDB/boot node configuration is not supported.

For non-partitioned systems, change sdb-p1 to sdb in these instructions.

Before beginning, note the SDB and login nodes' IDs as you will need them throughout the install process.

 crayadm@smw> ssh root@boot-p1
 boot# grep sdb /etc/hosts
 10.128.0.32   nid00031       c0-0c0s0n3     sdb001  sdb002
 10.131.255.253 sdb sdb-p1 syslog syslog-p1 ufs ufs-p1
 
 boot# grep login /etc/hosts
 10.128.0.3    nid00002       c0-0c0s1n0     login   login-p1       login1  castor-p1 

In this example, and throughout this page, the login node has NID 2 and the SDB has NID 31.

  1. Copy Moab/TORQUE software to SMW and boot node

    workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/torque.4.5.0.tar.gz crayadm@smw:/home/crayadm/<yourusername>
    workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/moab-7.5.0-SUSE11-linux-x86_64-torque.tar.gz crayadm@smw:/home/crayadm/<yourusername>
     
    crayadm@smw> cd /home/crayadm//<yourusername>
    crayadm@smw> scp -p torque-4.5.0.tar.gz root@boot-p1:/rr/current/software
    crayadm@smw> scp -p moab-7.5.0-SUSE11-linux-x86_64-torque.tar.gz root@boot-p1:/rr/current/software
  2. Install TORQUE. SSH to the boot node and unpack the TORQUE tarball within xtopview.

    crayadm@smw> ssh root@boot-p1
    boot# xtopview -m "Installing TORQUE"
    default/:/# cd /software/
    default/:/# tar -zxvf torque-4.5.0.tar.gz
    
    (or, if installing on an esMS)
    esms# cd /path/to/software/
    esms# tar -zxvf torque-4.5.0.tar.gz
    
  3. Configure, build, and install TORQUE within xtopview.

    default/:/# cd torque-4.5.0
    default/:/# ./configure --prefix=/opt/torque/4.5.0 --with-server-home=/var/spool/torque --with-default-server=sdb-p1 --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/opt/modulefiles --with-job-create CFLAGS="-DCRAY_MOAB_PASSTHRU"
    
    (or, if installing on an esMS)
    esms# cd /path/to/software/torque-4.5.0
    esms# ./configure --prefix=/opt/torque/4.5.0 --with-server-home=/var/spool/torque --with-default-server=this-esms --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/cm/local/modulefiles CFLAGS="-DCRAY_MOAB_PASSTHRU"

    The server name in the example is sdb-p1. Change this to sdb on a non-partitioned system.

  4. Make and install TORQUE.

     default/:/# make
     default/:/# make packages
     default/:/# make install
     default/:/# ln -sf /opt/torque/4.5.0 /opt/torque/default # The previous default symlink might need to be deleted first if it exists
     default/:/# exit
  5. Copy the TORQUE server directory to the Moab server host.

     boot# cd /rr/current/var/spool
     boot# cp -pr torque /snv/31/var/spool
     boot# cp -pr torque /snv/2/var/spool

    In this example, the SDB node has NID 31, and the login node NID 2.

  6. Set up TORQUE on the SDB node.

    boot# ssh sdb-p1
    sdb# export PATH=/opt/torque/default/sbin:/opt/torque/default/bin:$PATH
    sdb# cd /software/torque-4.5.0
    sdb# ./torque.setup root
    root
    pbs_server port is: 15001
    trqauthd daemonized - port 15005
    trqauthd successfully started
    initializing TORQUE (admin: root@boot)
     
    You have selected to start pbs_server in create mode.
    If the server database exists it will be overwritten.
    do you wish to continue y/(n)?
     
    # Type y
     
    sdb# qmgr
    Qmgr: set server keep_completed = 60 # Number of seconds to keep completed jobs in qstat
    unset queue batch resources_default.nodes
    set server acl_host_enable = true
    set server acl_hosts += nid00002
    set server acl_hosts += castor-p1 # Where castor-p1 is the hostname of the login node
    set server acl_hosts += sdb-p1
    set server submit_hosts += login
    set server submit_hosts += login-p1 # Only needed on partitioned systems
    set server submit_hosts += castor-p1 # Where castor-p1 is the hostname of the login node
    set server submit_hosts += nid00002 # Where nid00002 is the NID of the login node
    set server tcp_timeout = 30
    set server query_other_jobs = True
    set server disable_server_id_check = True
    set queue batch resources_default.mppnppn=16  # On Cascade systems with aprun -j1 set as default
    set server cray_enabled = True
    set server resources_default.partition = castor # Where "castor" is the same as the RMCFG[clustername] in moab.cfg
    exit
  7. Get the number of nodes available on the system.

    sdb# echo Node count is $(($(apstat -v | grep XT | awk "{print \$3}")))

    In the nodes / nodect commands, put in your system's number of nodes.

    sdb# qmgr
    set server resources_available.nodes = 20
    set server resources_available.nodect = 20
    set queue batch resources_available.nodes = 20
    set queue batch resources_available.nodect = 20
    exit
  8. Create TORQUE nodes file.

    sdb# vi /var/spool/torque/server_priv/nodes
     
       castor-p1 alps_login np=1000 # Where "castor-p1" is the hostname of your login node
       sdb-p1 alps_reporter
     
     sdb# exit

    The np attribute is the number of processes that can be running at once. This number should be set appropriately high, depending on the number of nodes on the system.

  9. Install the torque_server init.d script on the SDB node.

    There is a known issue with some of the init scripts included with TORQUE. If the included init script doesn't work for you, use the process below to create a workaround torque_server init.d script for your system.

    boot# xtopview -n 31 -m "torque_server init.d"
    node/31:/ # touch /etc/init.d/torque_server
    node/31:/ # xtspec -n 31 /etc/init.d/torque_server
    node/31:/ # chmod a+x /etc/init.d/torque_server
    node/31:/ # vi /etc/init.d/torque_server
    
    #!/bin/sh
    #
    # pbs_server    This script will start and stop the PBS Server
    #
    ### BEGIN INIT INFO
    # Provides:       pbs_server
    # Required-Start: $local_fs network
    # Should-Start:
    # Required-Stop:
    # Should-Stop:
    # Default-Start:  2 3 5
    # Default-Stop:
    # Description: Torque is a versatile batch system for SMPs and clusters
    ### END INIT INFO
     
    PBS_DAEMON=/opt/torque/default/sbin/pbs_server
    PBS_HOME=/var/spool/torque
    PIDFILE=$PBS_HOME/server_priv/server.lock
    export PBS_DAEMON PBS_HOME PIDFILE
     
    # Source the library functions
    . /etc/rc.status
    rc_reset
     
    [ -f /etc/sysconfig/pbs_server ] && . /etc/sysconfig/pbs_server
    [ -x $PBS_DAEMON ] || exit
     
     
    # How were we called
    case "$1" in
          start)
                  echo -n "Starting TORQUE Server: "
                  ulimit -c unlimited
                  if [ -r $PBS_HOME/server_priv/serverdb ]
                  then
                          startproc $PBS_DAEMON $SERVER_ARGS
                  else
                          startproc $PBS_DAEMON -t create $DAEMON_ARGS
                  fi
                  rc_status -v
                  ;;
          stop)
                  echo -n "Shutting down TORQUE Server: "
                  killproc -p $PIDFILE $PBS_DAEMON
                  rc_status -v
                  ;;
          status)
                  echo -n "Checking TORQUE Server: "
                  checkproc -p $PIDFILE pbs_server
                  rc_status -v
                  ;;
          restart)
                  $0 stop
                  $0 start
                  rc_status
                  ;;
          try-restart)
                  $0 status >/dev/null && $0 restart
                  rc_status
                  ;;
          reload|force-reload)
                  echo -n "Reloading TORQUE Server: "
                  killproc -p $PIDFILE pbs_server -HUP
                  rc_status -v
                  ;;
          *)
                  echo "Usage: torque_server {start|stop|status|try-restart|restart|force-reload|reload}"
                  exit 1
    esac
    rc_exit
  10. Install the torque_mom init.d script on the SDB (or other node that runs the alps_reporter service for TORQUE) and login nodes.

    There is a known issue with some of the init scripts included with TORQUE. If the included init script doesn't work for you, use the process below to create a workaround torque_mom init.d script for your system.

    Example 25-1: SDB node

    boot# xtopview -n 31 -m "torque_mom init.d"
    node/31:/ #  touch /etc/init.d/torque_mom
    node/31:/ #  xtspec -n 31 /etc/init.d/torque_mom
    node/31:/ #  chmod +x /etc/init.d/torque_mom
    node/31:/ #  vi /etc/init.d/torque_mom
    
    #!/bin/sh
    #
    # pbs_mom       This script will start and stop the PBS Mom
    #
    ### BEGIN INIT INFO
    # Provides:       pbs_mom
    # Required-Start: $local_fs
    # Should-Start: pbs_server pbs_sched
    # Required-Stop:
    # Should-Stop:
    # Default-Start:  2 3 5
    # Default-Stop:
    # Description: Torque is a versatile batch system for SMPs and clusters
    ### END INIT INFO
     
    PBS_DAEMON=/opt/torque/default/sbin/pbs_mom
    PBS_HOME=/var/spool/torque
    PIDFILE=$PBS_HOME/mom_priv/mom.lock
    export PBS_DAEMON PBS_HOME PIDFILE
     
    ulimit -n 32768
    # Source the library functions
    . /etc/rc.status
    rc_reset
     
    [ -f /etc/sysconfig/pbs_mom ] && . /etc/sysconfig/pbs_mom
    [ -x $PBS_DAEMON ] || exit
     
    args=""
    if [ -z "$PREVLEVEL" ];then
     # being run manually, don't disturb jobs
     args="-p"
    fi
    
    # How were we called
    case "$1" in
          start)
                  echo -n "Starting TORQUE Mom: "
                  #ulimit -c unlimited
                  /sbin/startproc $PBS_DAEMON $args $DAEMON_ARGS
                  rc_status -v
                  ;;
          purge)
                  [ -f /var/lock/subsys/pbs_mom ] && $0 stop
                  echo -n "Starting TORQUE Mom with purge: "
                  startproc $PBS_DAEMON -r $DAEMON_ARGS
                  rc_status -v
                  ;;
          stop)
                  echo -n "Shutting down TORQUE Mom: "
                  /sbin/killproc -p $PIDFILE $PBS_DAEMON
                  rc_status -v
                  ;;
          status)
                  echo -n "Checking TORQUE Mom: "
                  checkproc -p $PIDFILE $PBS_DAEMON
                  rc_status -v
                  ;;
          restart)
                  $0 stop
                  sleep 1
                  $0 start -p
                  rc_status
                  ;;
          try-restart)
                  $0 status >/dev/null && $0 restart
                  rc_status
                  ;;
          reload|force-reload)
                  echo -n "Re-reading TORQUE Mom config file: "
                  killproc -p $PIDFILE -HUP pbs_mom
                  rc_status -v
                  ;;
          *)
                  echo "Usage: torque_mom {start|stop|status|try-restart|restart|force-reload|reload|purge}"
                  exit 1
    esac
    

    Example 25-2: Login nodes

    boot# xtopview -c login -m "torque_mom init.d"
    class/login:/ # touch /etc/init.d/torque_mom
    class/login:/ # xtspec -c login /etc/init.d/torque_mom
    class/login:/ # chmod +x /etc/init.d/torque_mom
    class/login:/ # vi /etc/init.d/torque_mom
    							
    # Use the same script as the SDB node above
  11. Create the MOM configuration file. This must be done on every login node and also the alps_reporter node (typically the SDB) specified in the TORQUE server_priv/nodes file.

    1. First, determine if the ALPS path needs to be configured in the MOM configuration file.

      login# which apbasil
      /usr/bin/apbasil
      # No configuration change needed
      
      login# which apbasil
      /opt/cray/alps/5.0.2-2.0500.7827.1.1.ari/bin/apbasil
      # MOM configuration change is needed. This path needs to be declared in the mom_priv/config file.
    2. Create and populate the MOM configuration file on the nodes.

      login and sdb# vi /var/spool/torque/mom_priv/config
       
      $usecp *:/ufs /ufs
      $usecp *:/home /home
      $usecp *:/home/users /home/users
      $usecp *:/scratch /scratch
      $usecp *:/lus /lus
      $usecp *:/extlus /extlus
      $login_node true # For login node
      $reporter_mom true # For SDB node
      $apbasil_protocol 1.2
      $prologalarm 120
      $apbasil_path /opt/cray/alps/default/bin/apbasil # Only if needed. Use the path discovered above.
    3. If needed, you can add CPR information to the MOM configuration on the login nodes .

      $checkpoint_run_exe /opt/cray/blcr/default/bin/cr_run
      $checkpoint_script /opt/cray/cprbatchutils/default/libexec/checkpoint.torque
      $restart_script /opt/cray/cprbatchutils/default/libexec/restart.torque
      $remote_checkpoint_dirs /lus/scratch/BLCR_checkpoint_dir
  12. Create the torque.cfg file (useful if having issues with LDAP users submitting jobs) on the SDB node.

    sdb# vi /var/spool/torque/torque.cfg
     
       QSUBSENDUID true
       VALIDATEPATH FALSE
    
  13. Install the trqauthd init.d script on the SDB and login nodes.

    boot# xtopview -n <SDB or login> -m "trqauthd"
    node/<SDB or login>:/ # cp /software/torque-4.5.0/contrib/init.d/suse.trqauthd /etc/init.d/trqauthd
    node/<SDB or login>:/ # chmod +x /etc/init.d/trqauthd
    node/<SDB or login>:/ # vi /etc/init.d/trqauthd
     
       PBS_DAEMON=/opt/torque/default/sbin/trqauthd
  14. Start the trqauth daemon on the SDB and login nodes.

    <SDB or login># /etc/init.d/trqauthd start

Enabling node features for Cray compute nodes

Node features can be set for Cray compute nodes. To add node features to a Cray compute node, use the cray_compute keyword on designated nodes in the nodes file.:

# node_id cray_compute feature_name  
2 cray_compute bigmem

Configuring TORQUE for ALPS 1.3

To configure TORQUE for ALPS 1.3, configure the apbasil_protocol parameter in mom_priv/config and set the nppcu server parameter. The nppcu parameter has three options that determine whether to use Hyper-Threading:

Table 25-1: nppcu values

Value Description
0 Allow ALPS to choose
1 Hyper-Threading disabled (default)
2 Hyper-Threading enabled

When nppcu is set to 0 or 2, pbs_nodes reports twice as many cores.

apbasil_protocol:

$apbasil_protocol 1.3
$loglevel 3

nppcu:

qmgr -c 'set server nppcu=1'

Installing Moab

  1. Unpack the Moab tarball within xtopview.

    boot# xtopview -m "Installing Moab"
    default/:/# cd /software/
    default/:/# tar -zxvf moab-7.5.0-linux-x86_64-torque-xt4.tar.gz
    default/:/# cd moab-7.5.0
    
  2. Configure, build, and install Moab within xtopview.

    default/:/# ./configure --prefix=/opt/moab/7.5.0 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/opt/modulefiles --with-xt4
     
    (or, if installing on an esMS)
    default/:/# ./configure --prefix=/opt/moab/7.5.0 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/cm/local/modulefiles --with-xt4
    
    default/:/# make install
    default/:/# ln -sf /opt/moab/7.5.0 /opt/moab/default # The previous default symlink may need to be deleted first if it exists
    default/:/# exit
  3. Configure the moab.cfg file.

    boot# cd /rr/current/var/spool/moab/etc
    boot# vi moab.cfg
     
       Change the value of SCHEDCFG[Moab] to SERVER=sdb-p1:42559 # Leave the port number as whatever default is present
       Change ADMINCFG[1] USERS=root to USERS=root,crayadm  # Where "crayadm" is the administrative user
       If applicable, change TOOLSDIR from /opt/moab/7.5.0/tools to /opt/moab/default/tools
       
       Change RMCFG[boot]    TYPE=PBS to
              RMCFG[castor]  TYPE=TORQUE # "castor" can be any logical name for the partition
       
       For Moab version 6.x, use TYPE=NATIVE:XT4
       
       Example:
           RMCFG[tuna]  TYPE=TORQUE SUBMITCMD=/opt/torque/default/bin/qsub FLAGS=asyncstart
       
       
       Add:
       RMPOLLINTERVAL        00:00:10
       DEFERTIME             00:05:00
       JOBNODEMATCHPOLICY    EXACTNODE
       NODECFG[DEFAULT]      OS=linux ARCH=XT
       NODEACCESSPOLICY      SINGLEJOB
       JOBMIGRATEPOLICY      IMMEDIATE
       NODEALLOCATIONPOLICY  PRIORITY
       NODECFG[DEFAULT]      PRIORITYF='PRIORITY'
       NODECFG[castor-p1]    Partition=login # Use a logical name such as "login" to keep the MOM nodes in a separate
                                             # partition from the compute nodes
                                             # "castor-p1" in this case is the hostname of the login node
       CLIENTCFG[DEFAULT]    DEFAULTSUBMITPARTITION=castor # Where "castor" is the name of the partition (see RMCFG[castor] above)
       JOBMAXTASKCOUNT        <total number of processors>
       
       
       # Comment out USEDATABASE   INTERNAL
       
       # If using a re-purposed compute node as the alps_reporter MOM, add:
       NODECFG[nid00060]    Partition=login # Where nid00060 is the nid of the RCN
       
       # If necessary to ignore nodes (such as 24 core nodes on a primarily 32 core system), add:
       IGNORENODES   57,58 # Where "57" and "58" are node hostnames
  4. Configure the config.xt4.pl file.

    boot# vi config.xt4.pl
    
    $basilProtocol = "1.2";
       
    # Uncomment the %loginReplaceTable line, and update the hostnames:
    %loginReplaceTable = (nid00002 => "login-p1"); # Where "login-p1" is the exact hostname of the login node, for example, "castor-p3"
       
    # Uncomment:
    $topologyOrdering = 1;
       
    # If on a system that has undergone ALPS standardization (See the apbasil notes above), replace the $torquePath line with:
    my $torquePath = "/opt/torque/default/bin:/usr/bin:/opt/cray/alps/default/bin";
  5. Copy Moab to the SDB node.

    boot# cd /rr/current/var/spool/
    boot# cp -pr moab /snv/31/var/spool/
    
    For Moab version 6.x only:
    boot# mv /snv/31/var/spool/moab/etc/moab.cfg /snv/31/var/spool/moab/
    
    For all versions:
    boot# mkdir -p /snv/2/var/spool/moab/etc /snv/2/var/spool/moab/log
    boot# cp moab/etc/moab.cfg /snv/2/var/spool/moab/etc/
    boot# cp moab/etc/config.xt4.pl /snv/2/var/spool/moab/etc/
  6. Install the moab init.d script

    There is a known issue with some of the init scripts included with Moab. If the included init script doesn't work for you, use the process below to create a workaround moab init.d script for your system.

    boot# xtopview -n 31 -m "Moab init.d"
    node/31:/ # touch /etc/init.d/moab
    node/31:/ # xtspec -n 31 /etc/init.d/moab
    node/31:/ # chmod a+x /etc/init.d/moab
    node/31:/ # vi /etc/init.d/moab
    #!/bin/bash
    #
    # Starts the Moab daemon
    #
    # chkconfig: 345 96 6
    # description: Moab Workload Manager
    # processname: moab
    #
    ### BEGIN INIT INFO
    # Provides: Moab
    # Required-Start: $local_fs $syslog $network $named
    # Required-Stop: $local_fs $syslog $network $named
    # Default-Start: 3 5
    # Default-Stop: 0 1 2 6
    # Short-Description: Moab daemon management
    # Description: Start Moab Workload Manager
    ### END INIT INFO
    #
    # 1. This file should be installed as /etc/init.d/moab
    #
    # 2. Start Moab with:
    #
    #      /etc/init.d/moab start
    #
    # Source function library.
    [ -f /etc/rc.status ] || exit 0
    . /etc/rc.status
     
     
    export MOABHOMEDIR=/var/spool/moab
    export MOABPARCLEANUP=Full
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/torque/default/lib
     
     
    prog=moab
    path=/opt/moab/default/sbin
    RETVAL=0
     
    function start()
    {
          echo -n "Starting $prog: "
          ulimit -s unlimited    # Increase stack size to unlimited
          ulimit -c unlimited  # Uncomment to preserve core files
     
          export MOABNOALLOCMASTER=1
          sleep 2
          startproc $path/moab
     
          RETVAL=$?
          echo
          [ $RETVAL -eq 0 ] && touch /var/lock/subsys/moab
          return $RETVAL
    }
     
    function stop()
    {
          echo -n "Shutting down $prog: "
          killproc moab
          RETVAL=$?
          echo
          [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/moab
          return $RETVAL
    }
     
    function restart()
    {
          stop
          sleep 2
          start
    }
     
    function condrestart()
    {
          if [ -f /var/lock/subsys/capi ] ; then
                  restart
          fi
    }
     
    function reload()
    {
          echo -n $"Reloading $prog: "
          schedctl -R
          RETVAL=$?
          echo
          return $RETVAL
    }
     
    case "$1" in
      start)
                  start
                  rc_status -v
                  ;;
      status)
                  status moab
                  RETVAL=$?
                  rc_status -v
                  ;;
      stop)
                  stop
                  rc_status -v
                  ;;
      restart)
                  restart
                  rc_status -v
                  ;;
      condrestart)
                  condrestart
                  ;;
      reload)
                  reload
                  rc_status -v
                  ;;
      *)
          echo "Usage: $prog {start|stop|restart|reload|status|condrestart}"
          exit 1
    esac
     
    exit $RETVAL
  7. For versions of Moab earlier than 7.1.3 and TORQUE 4.1.4, root must be allowed to submit jobs.

    boot# ssh sdb
    sdb# module load moab torque
    sdb# qmgr
     qmgr> set server acl_roots = root
     qmgr> exit
    sdb# /etc/init.d/torque_server restart
     
    sdb# vi /var/spool/moab/etc/moab.cfg # or /var/spool/moab/moab.cfg for Moab version 6
     
     Add
     ALLOWROOTJOBS TRUE
     
    sdb# /etc/init.d/moab restart
       or
    sdb# mschedctl -R
  8. For versions of Moab newer than 7.1.3, the root user can submit jobs as another user.

    sdb# qmgr
     qmgr> set server managers += root@*
     qmgr> exit
    
    sdb# ssh login
    
    login# qsub -I -l mppwidth=1 -P <otherusername>
  9. Copy your Moab license file to the SDB node in the /var/spool/moab/ directory.