N.77 Installation Notes for Moab and Torque for Cray

The instructions within this topic only apply to the installation of Moab and Torque on Cray service and login nodes within a Cray system.

For Cray systems, Adaptive Computing recommends that you install Moab and Torque Servers (head nodes) on commodity hardware (not on Cray compute/service/login nodes).

However, you must install the Torque pbs_mom daemon and Torque client commands on Cray login and "mom" service nodes since the pbs_mom must run on a Cray service node within the Cray system so it has access to the Cray ALPS subsystem.

See the Moab Installation and Configuration Guide for instructions on installing Moab and Torque on a non-Cray server.

In this topic:

N.77.1 Overview
N.77.2 Available Scripts
N.77.3 Upgrade Notes
N.77.4 Installing Torque on a Cray
N.77.5 Enabling Node Features for Cray Compute Nodes
N.77.6 Configuring Torque for ALPS 1.3
N.77.7 Installing Moab Workload Manager

N.77.1 Overview

Moab and Torque can be used to manage the batch system for Cray. This document describes how to configure Moab and Torque to bring Moab's unmatched scheduling capabilities to the Cray.

Beginning with Torque 4.1, Torque handles all communication with ALPS, specifically the pbs_mom. Previously, communication with ALPS was handled by a combination of Moab, scripts and Torque. In the new model, Moab treats Torque as a regular Torque cluster without any special configuration. Torque now uses an extra MOM called the alps_reporter MOM to communicate with ALPS regarding configured and available resources. From the information reported by the alps_reporter mom, Torque creates a virtual node for each Cray compute node. Previously, Torque only reported the login nodes.

For clarity purposes, this topic assumes that your SDB node is mounting a persistent /var filesystem from the bootnode. If you have chosen not to use persistent /var filesystems please be aware that the instructions that follow would have to be modified for your situation.

N.77.2 Available Scripts

With the introduction of Cray's IMPS configuration management in Cray's CLE 6.0 (code-named Rhine), Cray has created recipe scripts and documentation for installing Moab and Torque on Cray service and login nodes. These recipe scripts and documentation are located in the <MoabInstallPath>/tools/cray/imps/CLE6 directory. Read the documentation contained in the Moab_Torque_IMPS_Install.pdf file and follow its instructions to create bootable images that boot with Moab and Torque running on Cray service, login, and "mom" nodes.

The installation instructions in the rest of this topic are for installing Torque and Moab on the service nodes of Cray ALPS-based systems running CLE versions prior to CLE 6.

Reference scripts are also available for the Cray CAPMC API (for KNL). These scripts are located in the <MoabInstallPath>/tools/cray/knl directory. Read the documentation contained in the README.INSTALL and the README.THEORY_OF_OPERATION files for more information.

N.77.3 Upgrade Notes

When upgrading to Torque 4.1.0 and using the new Cray model as described in this document, there should be no running jobs. Jobs may be queued but not running.

Beginning with Torque 6.0.2, Cray-enabled Torque may be configured with cgroups. On the login node, each job will have all of the cpus and all of the memory controllers in it's cgroup.

N.77.4 Installing Torque on a Cray

These instructions are written for a partitioned system, with separate SDB and boot nodes. A combined SDB/boot node configuration is not supported.

For non-partitioned systems, change sdb-p1 to sdb in these instructions.

Before beginning, note the SDB and login nodes' IDs as you will need them throughout the install process.

 crayadm@smw> ssh root@boot-p1
 boot# grep sdb /etc/hosts
 10.128.0.32   nid00031       c0-0c0s0n3     sdb001  sdb002
 10.131.255.253 sdb sdb-p1 syslog syslog-p1 ufs ufs-p1
 
 boot# grep login /etc/hosts
 10.128.0.3    nid00002       c0-0c0s1n0     login   login-p1       login1  castor-p1

In this example, and throughout this page, the login node has NID 2 and the SDB has NID 31.

Copy Moab/Torque software to SMW and boot node

workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/torque-6.1.1.tar.gz crayadm@smw:/home/crayadm/<yourusername>
workstation> scp -p /cray/css/release/cray/build/batch/moab-torque/moab-9.1.1-<OS>.tar.gz crayadm@smw:/home/crayadm/<yourusername>
 
crayadm@smw> cd /home/crayadm//<yourusername>
crayadm@smw> scp -p torque-6.1.1.tar.gz root@boot-p1:/rr/current/software
crayadm@smw> scp -p moab-9.1.1-<OS>.tar.gz root@boot-p1:/rr/current/software

Install Torque. SSH to the boot node and unpack the Torque tarball within xtopview.

crayadm@smw> ssh root@boot-p1
boot# xtopview -m "Installing Torque"
default/:/# cd /software/
default/:/# tar -zxvf torque-6.1.1.tar.gz

(or, if installing on an esMS)
esms# cd /path/to/software/
esms# tar -zxvf torque-6.1.1.tar.gz

Configure, build, and install Torque within xtopview.

default/:/# cd torque-6.1.1
default/:/# ./configure --prefix=/opt/torque/6.1.1 --with-server-home=/var/spool/torque --with-default-server=sdb-p1 --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/opt/modulefiles --with-job-create CFLAGS="-DCRAY_MOAB_PASSTHRU"

(or, if installing on an esMS)
esms# cd /path/to/software/torque-6.1.1
esms# ./configure --prefix=/opt/torque/6.1.1 --with-server-home=/var/spool/torque --with-default-server=this-esms --enable-syslog --disable-gcc-warnings --with-debug --with-modulefiles=/cm/local/modulefiles CFLAGS="-DCRAY_MOAB_PASSTHRU"

The server name in the example is sdb-p1. Change this to sdb on a non-partitioned system.

Make and install Torque.

 default/:/# make
 default/:/# make packages
 default/:/# make install
 default/:/# ln -sf /opt/torque/Torque /opt/torque/default # The previous default symlink might need to be deleted first if it exists
 default/:/# exit

Copy the Torque server directory to the Moab server host.
```
 boot# cd /rr/current/var/spool
 boot# cp -pr torque /snv/31/var/spool
 boot# cp -pr torque /snv/2/var/spool
```
In this example, the SDB node has NID 31, and the login node NID 2.

Set up Torque on the SDB node.

boot# ssh sdb-p1
sdb# export PATH=/opt/torque/default/sbin:/opt/torque/default/bin:$PATH
sdb# cd /software/torque-6.1.1
sdb# ./torque.setup root
root
pbs_server port is: 15001
trqauthd daemonized - port 15005
trqauthd successfully started
initializing Torque (admin: root@boot)
 
You have selected to start pbs_server in create mode.
If the server database exists it will be overwritten.
do you wish to continue y/(n)?
 
# Type y
 
sdb# qmgr
Qmgr: set server keep_completed = 60 # Number of seconds to keep completed jobs in qstat
unset queue batch resources_default.nodes
set server acl_host_enable = true
set server acl_hosts += nid00002
set server acl_hosts += castor-p1 # Where castor-p1 is the hostname of the login node
set server acl_hosts += sdb-p1
set server submit_hosts += login
set server submit_hosts += login-p1 # Only needed on partitioned systems
set server submit_hosts += castor-p1 # Where castor-p1 is the hostname of the login node
set server submit_hosts += nid00002 # Where nid00002 is the NID of the login node
set server tcp_timeout = 30
set server query_other_jobs = True
set server disable_server_id_check = True
set queue batch resources_default.mppnppn=16  # On Cascade systems with aprun -j1 set as default
set server cray_enabled = True
set server resources_default.partition = castor # Where "castor" is the same as the RMCFG[clustername] in moab.cfg
exit

Stop Torque.
```
sdb# qterm
```

Install the torque_server init.d script on the SDB node.

There is a known issue with some of the init scripts included with Torque. If the included init script doesn't work for you, use the process below to create a workaround torque_server init.d script for your system.

boot# xtopview -n 31 -m "torque_server init.d"
node/31:/ # touch /etc/init.d/torque_server
node/31:/ # xtspec -n 31 /etc/init.d/torque_server
node/31:/ # chmod a+x /etc/init.d/torque_server
node/31:/ # vi /etc/init.d/torque_server

#!/bin/sh
#
# pbs_server    This script will start and stop the PBS Server
#
### BEGIN INIT INFO
# Provides:       pbs_server
# Required-Start: $local_fs network
# Should-Start:
# Required-Stop:
# Should-Stop:
# Default-Start:  2 3 5
# Default-Stop:
# Description: Torque is a versatile batch system for SMPs and clusters
### END INIT INFO
 
PBS_DAEMON=/opt/torque/default/sbin/pbs_server
PBS_HOME=/var/spool/torque
PIDFILE=$PBS_HOME/server_priv/server.lock
export PBS_DAEMON PBS_HOME PIDFILE
 
# Source the library functions
. /etc/rc.status
rc_reset
 
[ -f /etc/sysconfig/pbs_server ] && . /etc/sysconfig/pbs_server
[ -x $PBS_DAEMON ] || exit
 
 
# How were we called
case "$1" in
      start)
              echo -n "Starting Torque Server: "
              ulimit -c unlimited
              if [ -r $PBS_HOME/server_priv/serverdb ]
              then
                      startproc $PBS_DAEMON $SERVER_ARGS
              else
                      startproc $PBS_DAEMON -t create $DAEMON_ARGS
              fi
              rc_status -v
              ;;
      stop)
              echo -n "Shutting down Torque Server: "
              killproc -p $PIDFILE $PBS_DAEMON
              rc_status -v
              ;;
      status)
              echo -n "Checking Torque Server: "
              checkproc -p $PIDFILE pbs_server
              rc_status -v
              ;;
      restart)
              $0 stop
              $0 start
              rc_status
              ;;
      try-restart)
              $0 status >/dev/null && $0 restart
              rc_status
              ;;
      reload|force-reload)
              echo -n "Reloading Torque Server: "
              killproc -p $PIDFILE pbs_server -HUP
              rc_status -v
              ;;
      *)
              echo "Usage: torque_server {start|stop|status|try-restart|restart|force-reload|reload}"
              exit 1
esac
rc_exit
node/31:/ # chkconfig torque_server on
node/31:/ # service torque_server start

Get the number of nodes available on the system.

sdb# echo Node count is $(($(apstat -v | grep XT | awk "{print \$3}")))

In the nodes / nodect commands, put in your system's number of nodes.

sdb# qmgr
set server resources_available.nodes = 20
set server resources_available.nodect = 20
set queue batch resources_available.nodes = 20
set queue batch resources_available.nodect = 20
exit

Create Torque nodes file.
```
sdb# vi /var/spool/torque/server_priv/nodes
 
   castor-p1 alps_login np=1000 # Where "castor-p1" is the hostname of your login node
   sdb-p1 alps_reporter
 
 sdb# exit
```
The np attribute is the number of processes that can be running at once. This number should be set appropriately high, depending on the number of nodes on the system.

Install the torque_mom init.d script on the SDB (or other node that runs the alps_reporter service for Torque) and login nodes.

There is a known issue with some of the init scripts included with Torque. If the included init script doesn't work for you, use the process below to create a workaround torque_mom init.d script for your system.

Example N-50: SDB node

boot# xtopview -n 31 -m "torque_mom init.d"
node/31:/ #  touch /etc/init.d/torque_mom
node/31:/ #  xtspec -n 31 /etc/init.d/torque_mom
node/31:/ #  chmod +x /etc/init.d/torque_mom
node/31:/ #  vi /etc/init.d/torque_mom

#!/bin/sh
#
# pbs_mom       This script will start and stop the PBS Mom
#
### BEGIN INIT INFO
# Provides:       pbs_mom
# Required-Start: $local_fs
# Should-Start: pbs_server pbs_sched
# Required-Stop:
# Should-Stop:
# Default-Start:  2 3 5
# Default-Stop:
# Description: Torque is a versatile batch system for SMPs and clusters
### END INIT INFO
 
PBS_DAEMON=/opt/torque/default/sbin/pbs_mom
PBS_HOME=/var/spool/torque
PIDFILE=$PBS_HOME/mom_priv/mom.lock
export PBS_DAEMON PBS_HOME PIDFILE
 
ulimit -n 32768
# Source the library functions
. /etc/rc.status
rc_reset
 
[ -f /etc/sysconfig/pbs_mom ] && . /etc/sysconfig/pbs_mom
[ -x $PBS_DAEMON ] || exit
 
args=""
if [ -z "$PREVLEVEL" ];then
 # being run manually, don't disturb jobs
 args="-p"
fi

# How were we called
case "$1" in
      start)
              echo -n "Starting Torque Mom: "
              #ulimit -c unlimited
              /sbin/startproc $PBS_DAEMON $args $DAEMON_ARGS
              rc_status -v
              ;;
      purge)
              [ -f /var/lock/subsys/pbs_mom ] && $0 stop
              echo -n "Starting Torque Mom with purge: "
              startproc $PBS_DAEMON -r $DAEMON_ARGS
              rc_status -v
              ;;
      stop)
              echo -n "Shutting down Torque Mom: "
              /sbin/killproc -p $PIDFILE $PBS_DAEMON
              rc_status -v
              ;;
      status)
              echo -n "Checking Torque Mom: "
              checkproc -p $PIDFILE $PBS_DAEMON
              rc_status -v
              ;;
      restart)
              $0 stop
              sleep 1
              $0 start -p
              rc_status
              ;;
      try-restart)
              $0 status >/dev/null && $0 restart
              rc_status
              ;;
      reload|force-reload)
              echo -n "Re-reading Torque Mom config file: "
              killproc -p $PIDFILE -HUP pbs_mom
              rc_status -v
              ;;
      *)
              echo "Usage: torque_mom {start|stop|status|try-restart|restart|force-reload|reload|purge}"
              exit 1
esac
node/31:/ # chkconfig torque_mom on
node/31:/ # service torque_mom start

Example N-51: Login nodes

boot# xtopview -c login -m "torque_mom init.d"
class/login:/ # touch /etc/init.d/torque_mom
class/login:/ # xtspec -c login /etc/init.d/torque_mom
class/login:/ # chmod +x /etc/init.d/torque_mom
class/login:/ # vi /etc/init.d/torque_mom
							
# Use the same script as the SDB node above
class/login:/ # chkconfig torque_mom on
class/login:/ # service torque_mom start

Create the MOM configuration file. This must be done on every login node and also the alps_reporter node (typically the SDB) specified in the Torque server_priv/nodes file.

First, determine if the ALPS path needs to be configured in the MOM configuration file.

login# which apbasil
/usr/bin/apbasil
# No configuration change needed

login# which apbasil
/opt/cray/alps/5.0.2-2.0500.7827.1.1.ari/bin/apbasil
# MOM configuration change is needed. This path needs to be declared in the mom_priv/config file.

Create and populate the MOM configuration file on the nodes.

login and sdb# vi /var/spool/torque/mom_priv/config
 
$usecp *:/ufs /ufs
$usecp *:/home /home
$usecp *:/home/users /home/users
$usecp *:/scratch /scratch
$usecp *:/lus /lus
$usecp *:/extlus /extlus
$login_node true # For login node
$reporter_mom true # For SDB node
$apbasil_protocol 1.2
$prologalarm 120
$apbasil_path /opt/cray/alps/default/bin/apbasil # Only if needed. Use the path discovered above.

If needed, you can add CPR information to the MOM configuration on the login nodes .

$checkpoint_run_exe /opt/cray/blcr/default/bin/cr_run
$checkpoint_script /opt/cray/cprbatchutils/default/libexec/checkpoint.torque
$restart_script /opt/cray/cprbatchutils/default/libexec/restart.torque
$remote_checkpoint_dirs /lus/scratch/BLCR_checkpoint_dir

Create the torque.cfg file (useful if having issues with LDAP users submitting jobs) on the SDB node.
```
sdb# vi /var/spool/torque/torque.cfg
 
   QSUBSENDUID true
   VALIDATEPATH FALSE
```

Install the trqauthd init.d script on the SDB and login nodes.

boot# xtopview -n <SDB or login> -m "trqauthd"
node/<SDB or login>:/ # cp /software/torque-6.1.1/contrib/init.d/suse.trqauthd /etc/init.d/trqauthd
node/<SDB or login>:/ # chmod +x /etc/init.d/trqauthd
node/<SDB or login>:/ # vi /etc/init.d/trqauthd
 
   PBS_DAEMON=/opt/torque/default/sbin/trqauthd

Start the trqauth daemon on the SDB and login nodes.
```
<SDB or login># /etc/init.d/trqauthd start
```

N.77.5 Enabling Node Features for Cray Compute Nodes

Node features can be set for Cray compute nodes. To add node features to a Cray compute node, use the cray_compute keyword on designated nodes in the nodes file.:

# node_id cray_compute feature_name  
2 cray_compute bigmem

N.77.6 Configuring Torque for ALPS 1.3

To configure Torque for ALPS 1.3, configure the apbasil_protocol parameter in mom_priv/config and set the nppcu server parameter. The nppcu parameter has three options that determine whether to use Hyper-Threading:

Table N-1: nppcu values

Value	Description
0	Allow ALPS to choose
1	Hyper-Threading disabled (default)
2	Hyper-Threading enabled

When nppcu is set to 0 or 2, pbs_nodes reports twice as many cores.

apbasil_protocol:

$apbasil_protocol 1.3
$loglevel 3

nppcu:

qmgr -c 'set server nppcu=1'

N.77.7 Installing Moab Workload Manager

Unpack the Moab tarball within xtopview.

boot# xtopview -m "Installing Moab"
default/:/# cd /software/
default/:/# tar -zxvf moab-9.1.1-<OS>.tar.gz
default/:/# cd moab-9.1.1

Configure, build, and install Moab within xtopview.

default/:/# ./configure --prefix=/opt/moab/9.1.1 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/opt/modulefiles --with-xt4
 
(or, if installing on an esMS)
default/:/# ./configure --prefix=/opt/moab/9.1.1 --with-homedir=/var/spool/moab --with-torque=/opt/torque/default --with-modulefiles=/cm/local/modulefiles --with-xt4

default/:/# make install
default/:/# ln -sf /opt/moab/9.1.1 /opt/moab/default # The previous default symlink may need to be deleted first if it exists
default/:/# exit

Configure the moab.cfg file.

boot# cd /rr/current/var/spool/moab/etc
boot# vi moab.cfg
 
   Change the value of SCHEDCFG[Moab] to SERVER=sdb-p1:42559 # Leave the port number as whatever default is present
   Change ADMINCFG[1] USERS=root to USERS=root,crayadm  # Where "crayadm" is the administrative user
   If applicable, change TOOLSDIR from /opt/moab/9.1.1/tools to /opt/moab/default/tools
   
   Change RMCFG[boot]    TYPE=PBS to
          RMCFG[castor]  TYPE=TORQUE # "castor" can be any logical name for the partition
   
   For Moab version 6.x, use TYPE=NATIVE:XT4
   
   Example:
       RMCFG[tuna]  TYPE=TORQUE SUBMITCMD=/opt/torque/default/bin/qsub FLAGS=asyncstart
   
   
   Add:
   RMPOLLINTERVAL        00:00:10
   DEFERTIME             00:05:00
   JOBNODEMATCHPOLICY    EXACTNODE
   NODECFG[DEFAULT]      OS=linux ARCH=XT
   NODEACCESSPOLICY      SINGLEJOB
   JOBMIGRATEPOLICY      IMMEDIATE
   NODEALLOCATIONPOLICY  PRIORITY
   NODECFG[DEFAULT]      PRIORITYF='PRIORITY'
   NODECFG[castor-p1]    Partition=login # Use a logical name such as "login" to keep the MOM nodes in a separate
                                         # partition from the compute nodes
                                         # "castor-p1" in this case is the hostname of the login node
   CLIENTCFG[DEFAULT]    DEFAULTSUBMITPARTITION=castor # Where "castor" is the name of the partition (see RMCFG[castor] above)
   JOBMAXTASKCOUNT        <total number of processors>
   
   
   # Comment out USEDATABASE   INTERNAL
   
   # If using a re-purposed compute node as the alps_reporter MOM, add:
   NODECFG[nid00060]    Partition=login # Where nid00060 is the nid of the RCN
   
   # If necessary to ignore nodes (such as 24 core nodes on a primarily 32 core system), add:
   IGNORENODES   57,58 # Where "57" and "58" are node hostnames

Configure the config.xt4.pl file.

boot# vi config.xt4.pl

$basilProtocol = "1.2";
   
# Uncomment the %loginReplaceTable line, and update the hostnames:
%loginReplaceTable = (nid00002 => "login-p1"); # Where "login-p1" is the exact hostname of the login node, for example, "castor-p3"
   
# Uncomment:
$topologyOrdering = 1;
   
# If on a system that has undergone ALPS standardization (See the apbasil notes above), replace the $torquePath line with:
my $torquePath = "/opt/torque/default/bin:/usr/bin:/opt/cray/alps/default/bin";

Copy Moab to the SDB node.

boot# cd /rr/current/var/spool/
boot# cp -pr moab /snv/31/var/spool/

For Moab version 6.x only:
boot# mv /snv/31/var/spool/moab/etc/moab.cfg /snv/31/var/spool/moab/

For all versions:
boot# mkdir -p /snv/2/var/spool/moab/etc /snv/2/var/spool/moab/log
boot# cp moab/etc/moab.cfg /snv/2/var/spool/moab/etc/
boot# cp moab/etc/config.xt4.pl /snv/2/var/spool/moab/etc/

Install the moab init.d script

There is a known issue with some of the init scripts included with Moab. If the included init script doesn't work for you, use the process below to create a workaround moab init.d script for your system.

boot# xtopview -n 31 -m "Moab init.d"
node/31:/ # touch /etc/init.d/moab
node/31:/ # xtspec -n 31 /etc/init.d/moab
node/31:/ # chmod a+x /etc/init.d/moab
node/31:/ # vi /etc/init.d/moab
#!/bin/bash
#
# Starts the Moab daemon
#
# chkconfig: 345 96 6
# description: Moab Workload Manager
# processname: moab
#
### BEGIN INIT INFO
# Provides: Moab
# Required-Start: $local_fs $syslog $network $named
# Required-Stop: $local_fs $syslog $network $named
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: Moab daemon management
# Description: Start Moab Workload Manager
### END INIT INFO
#
# 1. This file should be installed as /etc/init.d/moab
#
# 2. Start Moab with:
#
#      /etc/init.d/moab start
#
# Source function library.
[ -f /etc/rc.status ] || exit 0
. /etc/rc.status
 
 
export MOABHOMEDIR=/var/spool/moab
export MOABPARCLEANUP=Full
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/torque/default/lib
 
 
prog=moab
path=/opt/moab/default/sbin
RETVAL=0
 
function start()
{
      echo -n "Starting $prog: "
      ulimit -s unlimited    # Increase stack size to unlimited
      ulimit -c unlimited  # Uncomment to preserve core files
 
      export MOABNOALLOCMASTER=1
      sleep 2
      startproc $path/moab
 
      RETVAL=$?
      echo
      [ $RETVAL -eq 0 ] && touch /var/lock/subsys/moab
      return $RETVAL
}
 
function stop()
{
      echo -n "Shutting down $prog: "
      killproc moab
      RETVAL=$?
      echo
      [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/moab
      return $RETVAL
}
 
function restart()
{
      stop
      sleep 2
      start
}
 
function condrestart()
{
      if [ -f /var/lock/subsys/capi ] ; then
              restart
      fi
}
 
function reload()
{
      echo -n $"Reloading $prog: "
      schedctl -R
      RETVAL=$?
      echo
      return $RETVAL
}
 
case "$1" in
  start)
              start
              rc_status -v
              ;;
  status)
              status moab
              RETVAL=$?
              rc_status -v
              ;;
  stop)
              stop
              rc_status -v
              ;;
  restart)
              restart
              rc_status -v
              ;;
  condrestart)
              condrestart
              ;;
  reload)
              reload
              rc_status -v
              ;;
  *)
      echo "Usage: $prog {start|stop|restart|reload|status|condrestart}"
      exit 1
esac
 
exit $RETVAL

For versions of Moab earlier than 7.1.3 and Torque 4.1.4, root must be allowed to submit jobs.

boot# ssh sdb
sdb# module load moab torque
sdb# qmgr
 qmgr> set server acl_roots = root
 qmgr> exit
sdb# /etc/init.d/torque_server restart
 
sdb# vi /var/spool/moab/etc/moab.cfg # or /var/spool/moab/moab.cfg for Moab version 6
 
 Add
 ALLOWROOTJOBS TRUE
 
sdb# /etc/init.d/moab restart
   or
sdb# mschedctl -R

For versions of Moab newer than 7.1.3, the root user can submit jobs as another user.

sdb# qmgr
 qmgr> set server managers += root@*
 qmgr> exit

sdb# ssh login

login# qsub -I -l mppwidth=1 -P <otherusername>

Copy your Moab license file to the SDB node in the /var/spool/moab/ directory.