2.23 Torque NUMA-Aware Configuration

This topic provides instructions for enabling NUMA-aware, including cgroups, and requires Torque 6.0 or later. For instructions on NUMA-support configurations, see 2.24 Torque NUMA-Support Configuration. This topic assumes you have a basic understanding of cgroups. See RedHat Resource Management Guideor cgroups on kernel.orgfor basic information on cgroups.

Torque uses cgroups to better manage cpu and memory accounting, memory enforcement, cpuset management, and binding jobs to devices such as MICs and GPUs.

Be aware of the following:

• If you are building with cgroups enabled, you must have boost version 1.41 or later.
• The pbs_mom daemon is the only Torque binary that uses cgroups.
• Beginning with Torque 6.0.2, Cray-enabled Torque may be configured with cgroups. On the login node, each job will have all of the cpus and all of the memory controllers in it's cgroup.

Prerequisites

1. Install the prerequisites found in Installing Torque Resource Manager.
2. hwloc version 1.9 or later is required. Version 1.11 is needed if installing with NVIDIA K80 or newer GPU hardware
• download hwloc-1.9.tar.gz from: https://www.open-mpi.org/software/hwloc/v1.9
• perform the following command line actions:

$ tar -xzvf hwloc-1.9.tar.gz

$ cd hwloc-1.9.tar.gz

$ sudo ./configure

• You do not need to overwrite the default installation of hwloc. By default hwloc will install to the /usr/local directory. You can also configure hwloc with the --prefix option to have it install to a location of your choosing. If you do not install hwloc to /usr directory you can tell Torque where to find the version you want it to use at configure time using the --with-hwloc-path option. For example:
./configure --enable-cgroups --with-hwloc-path=/usr/local
• Run make
• sudo make install

Installation Instructions

Do the following:

1. Install the libcgroup package.
   

   Red Hat-based Systems must use libcgroup version 0.40.rc1-16.el6 or later; SUSE-based systems need to use a comparative libcgroup version.

   • Red Hat-based systems

     yum install libcgroup-tools libcgroup

   • SUSE-based systems

     zypper install libcgroup-tools

2. Enable Torque to access cgroups.

   $ ./configure --enable-cgroups

3. Run lssubsys -am to determine whether your system has mounted cgroups.
   1. If cgroups are not mounted, you will see:

      $ lssubsys -am
      
      nsperf_event
      net_prio
      cpuset
      cpu
      cpuacct
      memory
      devices
      freezer
      net_cls blkio

   2. If cgroups are mounted, you will see:

      $ lssubsys -am
      
      ns
      perf_event
      net_prio
      cpuset,cpu,cpuacct /cgroup/cpu
      memory /cgroup/memory
      devices /cgroup/devices
      freezer /cgroup/freezer
      net_cls /cgroup/net_cls
      blkio /cgroup/blkio

4. If you determined that cgroups are not mounted, take one of the following actions; otherwise, proceed to the next step.
   • Follow the cgroup mounting instructions for your Red Hat operating system.
   • Manually mount cgroups from the command line.

     mount -t cgroup -o <subsystem>[,<subsystem>,...] name <dir path>/name 

     The name parameter will be the name of the hierarchy.

     

     The following commands create five hierarchies, one for each subsystem.

     mount -t cgroup -o cpuset cpuset /var/spool/torque/cgroup/cpuset
     mount -t cgroup -o cpu cpu /var/spool/torque/cgroup/cpu
     mount -t cgroup -o cpuacct cpuacct /var/spool/torque/cgroup/cpuacct
     mount -t cgroup -o memory memory /var/spool/torque/cgroup/memory
     mount -t cgroup -o devices devices /var/spool/torque/cgroup/devices

   Once you have mounted the cgroups, run lssubsys -am again. You should now see:

   cpuset /var/spool/torque/cgroup/cpuset
   cpu /var/spool/torque/cgroup/cpu
   cpuacct /var/spool/torque/cgroup/cpuacct
   memory /var/spool/torque/cgroup/memory
   devices /var/spool//torque/cgroup/devices
   freezer
   blkio
   perf_event

2.23.1 Multiple cgroup Directory Configuration

If your system has more than one cgroup directory configured, you must create the trq‑cgroup‑paths file in the $TORQUE_HOME directory. This file has a list of the cgroup subsystems and the mount points for each subsystem in the syntax of <subsystem> <mount point>.

All five subsystems used by pbs_mom must be in the trq‑cgroup‑paths file. In the example that follows, a directory exists at /cgroup with subdirectories for each subsystem. Torque uses this file first to configure where it will look for cgroups.

cpuset  /cgroup/cpuset

cpuacct /cgroup/cpuacct

cpu     /cgroup/cpu

memory  /cgroup/memory

devices /cgroup/devices

2.23.2 Change Considerations for pbs_mom

In order to improve performance when removing cgroup hierarchies and job files Torque 6.0.0 added a new MOM configuration parameter $thread_unlink_calls. This parameter puts the job file cleanup processes on their own thread which increases the performance of the MOM. However, the addition of new threads also increases the size of pbs_mom from around 50 mb to 100mb.

$thread_unlink_calls is true by default which will thread job deletion. If pbs_mom is too large for your configuration set $thread_unlink_calls to false and jobs will be deleted within the main pbs_mom thread.

© 2016 Adaptive Computing