(Click to open topic with navigation)
The instructions within this topic only apply to the installation of Moab and Torque on Cray service and login nodes within a Cray system.
For Cray systems, Adaptive Computing recommends that you install Moab and Torque Servers (head nodes) on commodity hardware (not on Cray compute/service/login nodes).
However, you must install the Torque pbs_mom daemon and Torque client commands on Cray login and "mom" service nodes since the pbs_mom must run on a Cray service node within the Cray system so it has access to the Cray ALPS subsystem.
See the Moab HPC Suite Installation and Configuration Guide for instructions on installing Moab and Torque on a non-Cray server.
In this topic:
Moab and Torque can be used to manage the batch system for Cray. This document describes how to configure Moab and Torque to bring Moab's unmatched scheduling capabilities to the Cray.
Beginning with Torque 4.1, Torque handles all communication with ALPS, specifically the pbs_mom. Previously, communication with ALPS was handled by a combination of Moab, scripts and Torque. In the new model, Moab treats Torque as a regular Torque cluster without any special configuration. Torque now uses an extra MOM called the alps_reporter MOM to communicate with ALPS regarding configured and available resources. From the information reported by the alps_reporter mom, Torque creates a virtual node for each Cray compute node. Previously, Torque only reported the login nodes.
For clarity purposes, this topic assumes that your SDB node is mounting a persistent /var filesystem from the bootnode. If you have chosen not to use persistent /var filesystems please be aware that the instructions that follow would have to be modified for your situation.
With the introduction of Cray's IMPS configuration management in Cray's CLE 6.0 (code-named Rhine), Cray has created recipe scripts and documentation for installing Moab and Torque on Cray service and login nodes. These recipe scripts and documentation are located in the <MoabInstallPath>/tools/cray/imps/CLE6 directory. Read the documentation contained in the Moab_Torque_IMPS_Install.pdf file and follow its instructions to create bootable images that boot with Moab and Torque running on Cray service, login, and "mom" nodes.
The installation instructions in the rest of this topic are for installing Torque and Moab on the service nodes of Cray ALPS-based systems running CLE versions prior to CLE 6.
Reference scripts are also available for the Cray CAPMC API (for KNL). These scripts are located in the <MoabInstallPath>/tools/cray/knl directory. Read the documentation contained in the README.INSTALL and the README.THEORY_OF_OPERATION files for more information.
When upgrading to Torque 4.1.0 and using the new Cray model as described in this document, there should be no running jobs. Jobs may be queued but not running.
Beginning with Torque 6.0.2, Cray-enabled Torque may be configured with cgroups. On the login node, each job will have all of the cpus and all of the memory controllers in it's cgroup.
G.6.4 Installing Torque on a Cray
These instructions are written for a partitioned system, with separate SDB and boot nodes. A combined SDB/boot node configuration is not supported.
For non-partitioned systems, change sdb-p1 to sdb in these instructions.
Before beginning, note the SDB and login nodes' IDs as you will need them throughout the install process.
crayadm@smw> ssh root@boot-p1 boot# grep sdb /etc/hosts 10.128.0.32 nid00031 c0-0c0s0n3 sdb001 sdb002 10.131.255.253 sdb sdb-p1 syslog syslog-p1 ufs ufs-p1 boot# grep login /etc/hosts 10.128.0.3 nid00002 c0-0c0s1n0 login login-p1 login1 castor-p1
In this example, and throughout this page, the login node has NID 2 and the SDB has NID 31.
Copy Moab/Torque software to SMW and boot node
workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/torque.9.0.3.tar.gz crayadm@smw:/home/crayadm/<yourusername> workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/moab-<version>-SUSE11-linux-x86_64-torque.tar.gz crayadm@smw:/home/crayadm/<yourusername> crayadm@smw> cd /home/crayadm//<yourusername> crayadm@smw> scp -p torque-6.0.3.tar.gz root@boot-p1:/rr/current/software crayadm@smw> scp -p moab-9.0.3-SUSE11-linux-x86_64-torque.tar.gz root@boot-p1:/rr/current/software
Install Torque. SSH to the boot node and unpack the Torque tarball within xtopview.
crayadm@smw> ssh root@boot-p1 boot# xtopview -m "Installing Torque" default/:/# cd /software/ default/:/# tar -zxvf torque-6.0.3.tar.gz (or, if installing on an esMS) esms# cd /path/to/software/ esms# tar -zxvf torque-6.0.3.tar.gz
Configure, build, and install Torque within xtopview.
default/:/# cd torque-6.0.3 default/:/# ./configure --prefix=/opt/torque/6.0.3 --with-server-home=/var/spool/torque --with-default-server=sdb-p1 --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/opt/modulefiles --with-job-create CFLAGS="-DCRAY_MOAB_PASSTHRU" (or, if installing on an esMS) esms# cd /path/to/software/torque-6.0.3 esms# ./configure --prefix=/opt/torque/6.0.3 --with-server-home=/var/spool/torque --with-default-server=this-esms --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/cm/local/modulefiles CFLAGS="-DCRAY_MOAB_PASSTHRU"
The server name in the example is sdb-p1. Change this to sdb on a non-partitioned system.
Make and install Torque.
default/:/# make default/:/# make packages default/:/# make install default/:/# ln -sf /opt/torque/6.0.3 /opt/torque/default # The previous default symlink might need to be deleted first if it exists default/:/# exit
Copy the Torque server directory to the Moab server host.
boot# cd /rr/current/var/spool boot# cp -pr torque /snv/31/var/spool boot# cp -pr torque /snv/2/var/spool
In this example, the SDB node has NID 31, and the login node NID 2.
Set up Torque on the SDB node.
boot# ssh sdb-p1 sdb# export PATH=/opt/torque/default/sbin:/opt/torque/default/bin:$PATH sdb# cd /software/torque-6.0.3 sdb# ./torque.setup root root pbs_server port is: 15001 trqauthd daemonized - port 15005 trqauthd successfully started initializing Torque (admin: root@boot) You have selected to start pbs_server in create mode. If the server database exists it will be overwritten. do you wish to continue y/(n)? # Type y sdb# qmgr Qmgr: set server keep_completed = 60 # Number of seconds to keep completed jobs in qstat unset queue batch resources_default.nodes set server acl_host_enable = true set server acl_hosts += nid00002 set server acl_hosts += castor-p1 # Where castor-p1 is the hostname of the login node set server acl_hosts += sdb-p1 set server submit_hosts += login set server submit_hosts += login-p1 # Only needed on partitioned systems set server submit_hosts += castor-p1 # Where castor-p1 is the hostname of the login node set server submit_hosts += nid00002 # Where nid00002 is the NID of the login node set server tcp_timeout = 30 set server query_other_jobs = True set server disable_server_id_check = True set queue batch resources_default.mppnppn=16 # On Cascade systems with aprun -j1 set as default set server cray_enabled = True set server resources_default.partition = castor # Where "castor" is the same as the RMCFG[clustername] in moab.cfg exit
Stop Torque.
sdb# qterm
Install the torque_server init.d script on the SDB node.
There is a known issue with some of the init scripts included with Torque. If the included init script doesn't work for you, use the process below to create a workaround torque_server init.d script for your system.
boot# xtopview -n 31 -m "torque_server init.d" node/31:/ # touch /etc/init.d/torque_server node/31:/ # xtspec -n 31 /etc/init.d/torque_server node/31:/ # chmod a+x /etc/init.d/torque_server node/31:/ # vi /etc/init.d/torque_server #!/bin/sh # # pbs_server This script will start and stop the PBS Server # ### BEGIN INIT INFO # Provides: pbs_server # Required-Start: $local_fs network # Should-Start: # Required-Stop: # Should-Stop: # Default-Start: 2 3 5 # Default-Stop: # Description: Torque is a versatile batch system for SMPs and clusters ### END INIT INFO PBS_DAEMON=/opt/torque/default/sbin/pbs_server PBS_HOME=/var/spool/torque PIDFILE=$PBS_HOME/server_priv/server.lock export PBS_DAEMON PBS_HOME PIDFILE # Source the library functions . /etc/rc.status rc_reset [ -f /etc/sysconfig/pbs_server ] && . /etc/sysconfig/pbs_server [ -x $PBS_DAEMON ] || exit # How were we called case "$1" in start) echo -n "Starting Torque Server: " ulimit -c unlimited if [ -r $PBS_HOME/server_priv/serverdb ] then startproc $PBS_DAEMON $SERVER_ARGS else startproc $PBS_DAEMON -t create $DAEMON_ARGS fi rc_status -v ;; stop) echo -n "Shutting down Torque Server: " killproc -p $PIDFILE $PBS_DAEMON rc_status -v ;; status) echo -n "Checking Torque Server: " checkproc -p $PIDFILE pbs_server rc_status -v ;; restart) $0 stop $0 start rc_status ;; try-restart) $0 status >/dev/null && $0 restart rc_status ;; reload|force-reload) echo -n "Reloading Torque Server: " killproc -p $PIDFILE pbs_server -HUP rc_status -v ;; *) echo "Usage: torque_server {start|stop|status|try-restart|restart|force-reload|reload}" exit 1 esac rc_exit node/31:/ # chkconfig torque_server on node/31:/ # service torque_server start
Get the number of nodes available on the system.
sdb# echo Node count is $(($(apstat -v | grep XT | awk "{print \$3}")))
In the nodes / nodect commands, put in your system's number of nodes.
sdb# qmgr set server resources_available.nodes = 20 set server resources_available.nodect = 20 set queue batch resources_available.nodes = 20 set queue batch resources_available.nodect = 20 exit
Create Torque nodes file.
sdb# vi /var/spool/torque/server_priv/nodes
castor-p1 alps_login np=1000 # Where "castor-p1" is the hostname of your login node
sdb-p1 alps_reporter
sdb# exit
The np attribute is the number of processes that can be running at once. This number should be set appropriately high, depending on the number of nodes on the system.
Install the torque_mom init.d script on the SDB (or other node that runs the alps_reporter service for Torque) and login nodes.
There is a known issue with some of the init scripts included with Torque. If the included init script doesn't work for you, use the process below to create a workaround torque_mom init.d script for your system.
Example G-1: SDB node
boot# xtopview -n 31 -m "torque_mom init.d" node/31:/ # touch /etc/init.d/torque_mom node/31:/ # xtspec -n 31 /etc/init.d/torque_mom node/31:/ # chmod +x /etc/init.d/torque_mom node/31:/ # vi /etc/init.d/torque_mom #!/bin/sh # # pbs_mom This script will start and stop the PBS Mom # ### BEGIN INIT INFO # Provides: pbs_mom # Required-Start: $local_fs # Should-Start: pbs_server pbs_sched # Required-Stop: # Should-Stop: # Default-Start: 2 3 5 # Default-Stop: # Description: Torque is a versatile batch system for SMPs and clusters ### END INIT INFO PBS_DAEMON=/opt/torque/default/sbin/pbs_mom PBS_HOME=/var/spool/torque PIDFILE=$PBS_HOME/mom_priv/mom.lock export PBS_DAEMON PBS_HOME PIDFILE ulimit -n 32768 # Source the library functions . /etc/rc.status rc_reset [ -f /etc/sysconfig/pbs_mom ] && . /etc/sysconfig/pbs_mom [ -x $PBS_DAEMON ] || exit args="" if [ -z "$PREVLEVEL" ];then # being run manually, don't disturb jobs args="-p" fi # How were we called case "$1" in start) echo -n "Starting Torque Mom: " #ulimit -c unlimited /sbin/startproc $PBS_DAEMON $args $DAEMON_ARGS rc_status -v ;; purge) [ -f /var/lock/subsys/pbs_mom ] && $0 stop echo -n "Starting Torque Mom with purge: " startproc $PBS_DAEMON -r $DAEMON_ARGS rc_status -v ;; stop) echo -n "Shutting down Torque Mom: " /sbin/killproc -p $PIDFILE $PBS_DAEMON rc_status -v ;; status) echo -n "Checking Torque Mom: " checkproc -p $PIDFILE $PBS_DAEMON rc_status -v ;; restart) $0 stop sleep 1 $0 start -p rc_status ;; try-restart) $0 status >/dev/null && $0 restart rc_status ;; reload|force-reload) echo -n "Re-reading Torque Mom config file: " killproc -p $PIDFILE -HUP pbs_mom rc_status -v ;; *) echo "Usage: torque_mom {start|stop|status|try-restart|restart|force-reload|reload|purge}" exit 1 esac node/31:/ # chkconfig torque_mom on node/31:/ # service torque_mom start
Example G-2: Login nodes
boot# xtopview -c login -m "torque_mom init.d"
class/login:/ # touch /etc/init.d/torque_mom
class/login:/ # xtspec -c login /etc/init.d/torque_mom
class/login:/ # chmod +x /etc/init.d/torque_mom
class/login:/ # vi /etc/init.d/torque_mom
# Use the same script as the SDB node above
class/login:/ # chkconfig torque_mom on
class/login:/ # service torque_mom start
Create the MOM configuration file. This must be done on every login node and also the alps_reporter node (typically the SDB) specified in the Torque server_priv/nodes file.
First, determine if the ALPS path needs to be configured in the MOM configuration file.
login# which apbasil /usr/bin/apbasil # No configuration change needed login# which apbasil /opt/cray/alps/5.0.2-2.0500.7827.1.1.ari/bin/apbasil # MOM configuration change is needed. This path needs to be declared in the mom_priv/config file.
Create and populate the MOM configuration file on the nodes.
login and sdb# vi /var/spool/torque/mom_priv/config $usecp *:/ufs /ufs $usecp *:/home /home $usecp *:/home/users /home/users $usecp *:/scratch /scratch $usecp *:/lus /lus $usecp *:/extlus /extlus $login_node true # For login node $reporter_mom true # For SDB node $apbasil_protocol 1.2 $prologalarm 120 $apbasil_path /opt/cray/alps/default/bin/apbasil # Only if needed. Use the path discovered above.
If needed, you can add CPR information to the MOM configuration on the login nodes .
$checkpoint_run_exe /opt/cray/blcr/default/bin/cr_run $checkpoint_script /opt/cray/cprbatchutils/default/libexec/checkpoint.torque $restart_script /opt/cray/cprbatchutils/default/libexec/restart.torque $remote_checkpoint_dirs /lus/scratch/BLCR_checkpoint_dir
Create the torque.cfg file (useful if having issues with LDAP users submitting jobs) on the SDB node.
sdb# vi /var/spool/torque/torque.cfg QSUBSENDUID true VALIDATEPATH FALSE
Install the trqauthd init.d script on the SDB and login nodes.
boot# xtopview -n <SDB or login> -m "trqauthd" node/<SDB or login>:/ # cp /software/torque-6.0.3/contrib/init.d/suse.trqauthd /etc/init.d/trqauthd node/<SDB or login>:/ # chmod +x /etc/init.d/trqauthd node/<SDB or login>:/ # vi /etc/init.d/trqauthd PBS_DAEMON=/opt/torque/default/sbin/trqauthd
Start the trqauth daemon on the SDB and login nodes.
<SDB or login># /etc/init.d/trqauthd start
G.6.5 Enabling Node Features for Cray Compute Nodes
Node features can be set for Cray compute nodes. To add node features to a Cray compute node, use the cray_compute keyword on designated nodes in the nodes file.:
# node_id cray_compute feature_name 2 cray_compute bigmem
G.6.6 Configuring Torque for ALPS 1.3
To configure Torque for ALPS 1.3, configure the apbasil_protocol parameter in mom_priv/config and set the nppcu server parameter. The nppcu parameter has three options that determine whether to use Hyper-Threading:
Table G-1: nppcu values
Value | Description |
---|---|
0 | Allow ALPS to choose |
1 | Hyper-Threading disabled (default) |
2 | Hyper-Threading enabled |
When nppcu is set to 0 or 2, pbs_nodes reports twice as many cores.
apbasil_protocol:
$apbasil_protocol 1.3 $loglevel 3
nppcu:
qmgr -c 'set server nppcu=1'
G.6.7 Installing Moab Workload Manager
Unpack the Moab tarball within xtopview.
boot# xtopview -m "Installing Moab" default/:/# cd /software/ default/:/# tar -zxvf moab-9.0.3-linux-x86_64-torque-xt4.tar.gz default/:/# cd moab-9.0.3
Configure, build, and install Moab within xtopview.
default/:/# ./configure --prefix=/opt/moab/9.0.3 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/opt/modulefiles --with-xt4 (or, if installing on an esMS) default/:/# ./configure --prefix=/opt/moab/9.0.3 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/cm/local/modulefiles --with-xt4 default/:/# make install default/:/# ln -sf /opt/moab/9.0.3 /opt/moab/default # The previous default symlink may need to be deleted first if it exists default/:/# exit
Configure the moab.cfg file.
boot# cd /rr/current/var/spool/moab/etc boot# vi moab.cfg Change the value of SCHEDCFG[Moab] to SERVER=sdb-p1:42559 # Leave the port number as whatever default is present Change ADMINCFG[1] USERS=root to USERS=root,crayadm # Where "crayadm" is the administrative user If applicable, change TOOLSDIR from /opt/moab/9.0.3/tools to /opt/moab/default/tools Change RMCFG[boot] TYPE=PBS to RMCFG[castor] TYPE=TORQUE # "castor" can be any logical name for the partition For Moab version 6.x, use TYPE=NATIVE:XT4 Example: RMCFG[tuna] TYPE=TORQUE SUBMITCMD=/opt/torque/default/bin/qsub FLAGS=asyncstart Add: RMPOLLINTERVAL 00:00:10 DEFERTIME 00:05:00 JOBNODEMATCHPOLICY EXACTNODE NODECFG[DEFAULT] OS=linux ARCH=XT NODEACCESSPOLICY SINGLEJOB JOBMIGRATEPOLICY IMMEDIATE NODEALLOCATIONPOLICY PRIORITY NODECFG[DEFAULT] PRIORITYF='PRIORITY' NODECFG[castor-p1] Partition=login # Use a logical name such as "login" to keep the MOM nodes in a separate # partition from the compute nodes # "castor-p1" in this case is the hostname of the login node CLIENTCFG[DEFAULT] DEFAULTSUBMITPARTITION=castor # Where "castor" is the name of the partition (see RMCFG[castor] above) JOBMAXTASKCOUNT <total number of processors> # Comment out USEDATABASE INTERNAL # If using a re-purposed compute node as the alps_reporter MOM, add: NODECFG[nid00060] Partition=login # Where nid00060 is the nid of the RCN # If necessary to ignore nodes (such as 24 core nodes on a primarily 32 core system), add: IGNORENODES 57,58 # Where "57" and "58" are node hostnames
Configure the config.xt4.pl file.
boot# vi config.xt4.pl $basilProtocol = "1.2"; # Uncomment the %loginReplaceTable line, and update the hostnames: %loginReplaceTable = (nid00002 => "login-p1"); # Where "login-p1" is the exact hostname of the login node, for example, "castor-p3" # Uncomment: $topologyOrdering = 1; # If on a system that has undergone ALPS standardization (See the apbasil notes above), replace the $torquePath line with: my $torquePath = "/opt/torque/default/bin:/usr/bin:/opt/cray/alps/default/bin";
Copy Moab to the SDB node.
boot# cd /rr/current/var/spool/ boot# cp -pr moab /snv/31/var/spool/ For Moab version 6.x only: boot# mv /snv/31/var/spool/moab/etc/moab.cfg /snv/31/var/spool/moab/ For all versions: boot# mkdir -p /snv/2/var/spool/moab/etc /snv/2/var/spool/moab/log boot# cp moab/etc/moab.cfg /snv/2/var/spool/moab/etc/ boot# cp moab/etc/config.xt4.pl /snv/2/var/spool/moab/etc/
Install the moab init.d script
There is a known issue with some of the init scripts included with Moab. If the included init script doesn't work for you, use the process below to create a workaround moab init.d script for your system.
boot# xtopview -n 31 -m "Moab init.d" node/31:/ # touch /etc/init.d/moab node/31:/ # xtspec -n 31 /etc/init.d/moab node/31:/ # chmod a+x /etc/init.d/moab node/31:/ # vi /etc/init.d/moab #!/bin/bash # # Starts the Moab daemon # # chkconfig: 345 96 6 # description: Moab Workload Manager # processname: moab # ### BEGIN INIT INFO # Provides: Moab # Required-Start: $local_fs $syslog $network $named # Required-Stop: $local_fs $syslog $network $named # Default-Start: 3 5 # Default-Stop: 0 1 2 6 # Short-Description: Moab daemon management # Description: Start Moab Workload Manager ### END INIT INFO # # 1. This file should be installed as /etc/init.d/moab # # 2. Start Moab with: # # /etc/init.d/moab start # # Source function library. [ -f /etc/rc.status ] || exit 0 . /etc/rc.status export MOABHOMEDIR=/var/spool/moab export MOABPARCLEANUP=Full export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/torque/default/lib prog=moab path=/opt/moab/default/sbin RETVAL=0 function start() { echo -n "Starting $prog: " ulimit -s unlimited # Increase stack size to unlimited ulimit -c unlimited # Uncomment to preserve core files export MOABNOALLOCMASTER=1 sleep 2 startproc $path/moab RETVAL=$? echo [ $RETVAL -eq 0 ] && touch /var/lock/subsys/moab return $RETVAL } function stop() { echo -n "Shutting down $prog: " killproc moab RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/moab return $RETVAL } function restart() { stop sleep 2 start } function condrestart() { if [ -f /var/lock/subsys/capi ] ; then restart fi } function reload() { echo -n $"Reloading $prog: " schedctl -R RETVAL=$? echo return $RETVAL } case "$1" in start) start rc_status -v ;; status) status moab RETVAL=$? rc_status -v ;; stop) stop rc_status -v ;; restart) restart rc_status -v ;; condrestart) condrestart ;; reload) reload rc_status -v ;; *) echo "Usage: $prog {start|stop|restart|reload|status|condrestart}" exit 1 esac exit $RETVAL
For versions of Moab earlier than 7.1.3 and Torque 4.1.4, root must be allowed to submit jobs.
boot# ssh sdb
sdb# module load moab torque
sdb# qmgr
qmgr> set server acl_roots = root
qmgr> exit
sdb# /etc/init.d/torque_server restart
sdb# vi /var/spool/moab/etc/moab.cfg # or /var/spool/moab/moab.cfg for Moab version 6
Add
ALLOWROOTJOBS TRUE
sdb# /etc/init.d/moab restart
or
sdb# mschedctl -R
For versions of Moab newer than 7.1.3, the root user can submit jobs as another user.
sdb# qmgr
qmgr> set server managers += root@*
qmgr> exit
sdb# ssh login
login# qsub -I -l mppwidth=1 -P <otherusername>