2.23 Torque NUMA-Aware Configuration

This topic provides instructions for enabling NUMA-aware, including cgroups, and requires Torque 6.0 or later. For instructions on NUMA-support configurations, see 2.24 Torque NUMA-Support Configuration.

Torque uses cgroups to better manage cpu and memory accounting, memory enforcement, cpuset management, and binding jobs to devices such as MICs and GPUs.

The pbs_mom daemon is the only Torque binary that uses cgroups.

This topic assumes you have a basic understanding of cgroups. See RedHat Resource Management Guide or cgroups on kernel.org for basic information on cgroups.

Do the following:

1. Install the libcgroup package.
   

   Red Hat-based Systems must use libcgroup version 0.40.rc1-16.el6 or later; SUSE-based systems need to use a comparative libcgroup version.

   • Red Hat-based systems

     sudo yum install libcgroup

     yum install libcgroup-tools

   • SUSE-based systems

     zypper install libcgroup-tools

2. Enable Torque to access cgroups.

   $ ./configure --enable-cgroups

3. Run lssubsys -am to determine whether your system has mounted cgroups.
   1. If cgroups are not mounted, you will see:

      $ lssubsys -am
      
      nsperf_event
      net_prio
      cpuset
      cpu
      cpuacct
      memory
      devices
      freezer
      net_cls blkio

   2. If cgroups are mounted, you will see:

      $ lssubsys -am
      
      ns
      perf_event
      net_prio
      cpuset,cpu,cpuacct /cgroup/cpu
      memory /cgroup/memory
      devices /cgroup/devices
      freezer /cgroup/freezer
      net_cls /cgroup/net_cls
      blkio /cgroup/blkio

4. If you determined that cgroups are not mounted, take one of the following actions; otherwise, proceed to the next step.
   • Follow the cgroup mounting instructions for your Red Hat operating system.
   • Manually mount cgroups from the command line.

     mount -t cgroup -o <subsystem>[,<subsystem>,...] name <dir path>/name 

     The name parameter will be the name of the hierarchy.

     

     The following commands create five hierarchies, one for each subsystem.

     mount -t cgroup -o cpuset cpuset /var/spool/torque/cgroup/cpuset
     mount -t cgroup -o cpu cpu /var/spool/torque/cgroup/cpu
     mount -t cgroup -o cpuacct cpuacct /var/spool/torque/cgroup/cpuacct
     mount -t cgroup -o memory memory /var/spool/torque/cgroup/memory
     mount -t cgroup -o devices devices /var/spool/torque/cgroup/devices

   Once you have mounted the cgroups, run lssubsys -am again. You should now see:

   cpuset /var/spool/torque/cgroup/cpuset
   cpu /var/spool/torque/cgroup/cpu
   cpuacct /var/spool/torque/cgroup/cpuacct
   memory /var/spool/torque/cgroup/memory
   devices /var/spool//torque/cgroup/devices
   freezer
   blkio
   perf_event

2.23.1 Multiple cgroup Directory Configuration

If your system has more than one cgroup directory configured, you must create the trq‑cgroup‑paths file in the $TORQUE_HOME directory. This file has a list of the cgroup subsystems and the mount points for each subsystem in the syntax of <subsystem> <mount point>.

All five subsystems used by pbs_mom must be in the trq‑cgroup‑paths file. In the example that follows, a directory exists at /cgroup with subdirectories for each subsystem. Torque uses this file first to configure where it will look for cgroups.

cpuset  /cgroup/cpuset

cpuacct /cgroup/cpuacct

cpu     /cgroup/cpu

memory  /cgroup/memory

devices /cgroup/devices

© 2015 Adaptive Computing