Server configuration

See these topics for details:

• Server configuration overview
• Name service configuration
• Configuring job submission hosts
• Configuring TORQUE on a multi-homed server
• Architecture specific notes
• Specifying non-root administrators
• Setting up email
• Using MUNGE authentication
• Setting up the MOM hierarchy

Server configuration overview

There are several steps to ensure that the server and the nodes are completely aware of each other and able to communicate directly. Some of this configuration takes place within TORQUE directly using the qmgr command. Other configuration settings are managed using the pbs_server nodes file, DNS files such as /etc/hosts and the /etc/hosts.equiv file.

Name service configuration

Each node, as well as the server, must be able to resolve the name of every node with which it will interact. This can be accomplished using /etc/hosts, DNS, NIS, or other mechanisms. In the case of /etc/hosts, the file can be shared across systems in most cases.

A simple method of checking proper name service configuration is to verify that the server and the nodes can "ping" each other.

Configuring job submission hosts

Using RCmd authentication

When jobs can be submitted from several different hosts, these hosts should be trusted via the R* commands (such as rsh and rcp). This can be enabled by adding the hosts to the /etc/hosts.equiv file of the machine executing the pbs_server daemon or using other R* command authorization methods. The exact specification can vary from OS to OS (see the man page for ruserok to find out how your OS validates remote users). In most cases, configuring this file is as simple as adding a line to your /etc/hosts.equiv file, as in the following:

/etc/hosts.equiv:

Either of the hostname or username fields may be replaced with a wildcard symbol (+). The (+) may be used as a stand-alone wildcard but not connected to a username or hostname, e.g., +node01 or +user01. However, a (-) may be used in that manner to specifically exclude a user.

For example, the following lines will not work or will not have the desired effect:

These lines will work:

The most restrictive rules must precede more permissive rules. For example, to restrict user tsmith but allow all others, follow this format:

Please note that when a hostname is specified, it must be the fully qualified domain name (FQDN) of the host. Job submission can be further secured using the server or queue acl_hosts and acl_host_enabled parameters (for details, see Queue attributes).

Using the "submit_hosts" service parameter

Trusted submit host access may be directly specified without using RCmd authentication by setting the server submit_hosts parameter via qmgr as in the following example:

Allowing job submission from compute hosts

If preferred, all compute nodes can be enabled as job submit hosts without setting .rhosts or hosts.equiv by setting the allow_node_submit parameter to true.

Configuring TORQUE on a multi-homed server

If the pbs_server daemon is to be run on a multi-homed host (a host possessing multiple network interfaces), the interface to be used can be explicitly set using the SERVERHOST parameter.

Architecture specific notes

With some versions of Mac OS/X, it is required to add the line $restricted *.<DOMAIN> to the pbs_mom configuration file. This is required to work around some socket bind bugs in the OS.

Specifying non-root administrators

By default, only root is allowed to start, configure and manage the pbs_server daemon. Additional trusted users can be authorized using the parameters managers and operators. To configure these parameters use the qmgr command, as in the following example:

All manager and operator specifications must include a user name and either a fully qualified domain name or a host expression.

Setting up email

Moab relies on emails from TORQUE about job events. To set up email, do the following:

To set up email

1. Use the --with-sendmail configure option at configure time. TORQUE needs to know where the email application is. If this option is not used, TORQUE tries to find the sendmail executable. If it isn't found, TORQUE cannot send emails.

> ./configure --with-sendmail=<path_to_executable>

2. Set mail_domain in your server settings. If your domain is clusterresources.com, execute:

> qmgr -c 'set server mail_domain=clusterresources.com'

3. (Optional) You can override the default mail_body_fmt and mail_subject_fmt values via qmgr:

> qmgr -c 'set server mail_body_fmt=Job: %i \n Name: %j \n On host: %h \n \n %m \n \n %d'

> qmgr -c 'set server mail_subject_fmt=Job %i - %r'

By default, users receive e-mails on job aborts. Each user can select which kind of e-mails to receive by using the qsub -m option when submitting the job. If you want to dictate when each user should receive e-mails, use a submit filter (for details, see Job submission filter ("qsub wrapper")).

Using MUNGE authentication

MUNGE is an authentication service that creates and validates user credentials. It was developed by Lawrence Livermore National Laboratory (LLNL) to be highly scalable so it can be used in large environments such as HPC clusters. To learn more about MUNGE and how to install it, see http://code.google.com/p/munge/.

Configuring TORQUE to use MUNGE is a compile time operation. When you are building TORQUE, use -enable-munge-auth as a command line option with ./configure.

You can use only one authorization method at a time. If -enable-munge-auth is configured, the privileged port ruserok method is disabled.

TORQUE does not link any part of the MUNGE library into its executables. It calls the MUNGE and UNMUNGE utilities which are part of the MUNGE daemon. The MUNGE daemon must be running on the server and all submission hosts. The TORQUE client utilities call MUNGE and then deliver the encrypted credential to pbs_server where the credential is then unmunged and the server verifies the user and host against the authorized users configured in serverdb.

Authorized users are added to serverdb using qmgr and the authorized_users parameter. The syntax for authorized_users is authorized_users=<user>@<host>. To add an authorized user to the server you can use the following qmgr command:

The previous example adds user1 and user2 from hosta to the list of authorized users on the server. Users can be removed from the list of authorized users by using the -= syntax as follows:

Users must be added with the <user>@<host> syntax. The user and the host portion can use the '*' wildcard to allow multiple names to be accepted with a single entry. A range of user or host names can be specified using a [a-b] syntax where a is the beginning of the range and b is the end.

This allows user1 through user10 on hosta to run client commands on the server.

Setting up the MOM hierarchy

The MOM hierarchy allows you to override the compute nodes' default behavior of reporting status updates directly to the pbs_server. Instead, you configure compute nodes so that each node sends its status update information to another compute node. The compute nodes pass the information up a tree or hierarchy until eventually the information reaches a node that will pass the information directly to pbs_server. This can significantly reduce traffic and time required to keep the cluster status up to date.

The name of the file that contains the configuration information is named mom_hierarchy. By default, it is located in the /var/spool/torque/server_priv directory. The file uses syntax similar to XML:

<path>
  <level> comma-separated node list </level>
  <level> comma-separated node list </level>
  ...
</path>
...

The <path></path> tag pair identifies a group of compute nodes. The <level></level> tag pair contains a comma-separated list of compute node names. Multiple paths can be defined with multiple levels within each path.

Within a <path> tag pair the levels define the hierarchy. All nodes in the top level communicate directly with the server. All nodes in lower levels communicate to the first available node in the level directly above it. If the first node in the upper level goes down, the nodes in the subordinate level will then communicate to the next node in the upper level. If no nodes are available in an upper level then the node will communicate directly to the server.

If an upper level node has fallen out and then becomes available, the lower level nodes will eventually find that the node is available and start sending their updates to that node.

<path>
  <level>hostname:mom_manager_port</level>
  ...
</path>
...

Putting the MOM hierarchy on the MOMs

You can put the MOM hierarchy file directly on the MOMs. This way, the pbs_server doesn't have to send the hierarchy to all the MOMs during each pbs_server startup. The hierarchy file still has to exist on the pbs_server and if the file versions conflict, the pbs_server version overwrites the local MOM file. Due to this, it is recommended that the hierarchy file either be symlinked to the MOMs or put on a global NFS share.

Once the hierarchy file exists on the MOMs, start pbs_server with the -n option which tells pbs_server to not send the hierarchy file on startup. Instead, pbs_server waits until a MOM requests it.

Related topics 

• Advanced configuration

© 2014 Adaptive Computing