N.103
This chapter introduces NUMA-aware scheduling and identifies Moab-, Torque-, and MAM-related functions.
NUMA-aware scheduling is available with Moab Workload Manager 9.0 and later and requires Torque Resource Manager 6.0 and later.
In this chapter:
- N.103.1 NUMA Process
- N.103.2 Installation and Configuration
- N.103.3 Moab and NUMA Resources
- N.103.4 Track Dedicated NUMA Resources
The following image provides an example of the NUMA process.
|
|
Click to enlarge |
N.103.2 Installation and Configuration
Moab does not require any special installation or configuration processes to support NUMA-aware with Torque.
N.103.3 Moab and NUMA Resources
Moab uses generic resources to natively understand the concept of "socket", "numanode", "core" and "thread". The msub/qsub "-L" syntax for job submissions lets you request placement or allocation of these specific resources.
See -L NUMA Resource Request for more information on using the -L syntax.
For the -L syntax, submit using qsub unless your system requires submissions using msub.
- A job requesting a placement of a numanode is requesting exclusive access to the entire numa node and all of its resources including cores, threads, memory, gpus, and mics.
- A job requesting a placement of a socket is requesting exclusive access to the entire socket including numanodes, cores, threads, memory, gpus and mics.
Moab does not require a configuration change to support NUMA-aware scheduling (no new Moab configuration parameters). However, you may need to increase MAXGRES to accommodate the additional resources. See MAXGRES.
When using NUMA-aware, the following occurs:
- Moab imports NUMA resources from Torque and treats them as a special case of generic resources.
- When a job requests a NUMA resource, such as a socket or numanode, Moab will schedule exclusive access to those resources for the job. If exclusive access to the NUMA resource is not available on a particular node, Moab will look for another node or schedule the job out into the future if no resources are available at the time.
Moab assumes that the NUMA resources on a particular node are balanced. This means that each socket has the same amount of resources, including numanodes, cores, threads, memory, and gpus.
Moab supports advanced multi-req resource requests within the same job using the msub/qsub "-L" syntax.
In addition, any job that requests NUMA resources will receive a per-task default memory requirement if a memory requirement is not specified by the user. .
Adaptive Computing provides these recommendations for jobs:
- Jobs that request NUMA resources can share nodes with non-NUMA jobs, but it is not recommended. It is recommended that you enforce the separation using policies (queues, reservations, partitions, node-sets, etc.).
-
GPU jobs that request cores and processors should not share nodes with non-GPU jobs. For example, a job requesting a numanode may be given a numanode with a gpu and thus block other jobs from consuming that GPU. This can be prevented with proper queue policies and placement.
- A job may run across heterogeneous NUMA resources. For example, a socket on one node may contain more memory than a socket on another node. Use the NODEALLOCATIONPOLICY Moab parameter to enable running a job across homogenous NUMA resources.
N.103.4 Track Dedicated NUMA Resources
If Moab Accounting Manager is part of your configuration, you can configure MAM to track dedicated NUMA resources ( sockets, numanodes, cores, threads).
As the MAM Admin, run the commands for the individual resources you wish to track. The follow example shows the commands for all of the available resources.
[mam]$ mam-shell Attribute Create Object=UsageRecord Name=Sockets DataType=Integer Description="\"Number of Sockets Dedicated\""
[mam]$ mam-shell Attribute Create Object=UsageRecord Name=NumaNodes DataType=Integer Description="\"Number of Numa Nodes Dedicated\""
[mam]$ mam-shell Attribute Create Object=UsageRecord Name=Cores DataType=Integer Description="\"Number of Cores Dedicated\""
[mam]$ mam-shell Attribute Create Object=UsageRecord Name=Threads DataType=Integer Description="\"Number of Threads Dedicated\""
NUMA resources are only reported to MAM when they are dedicated to the job. As you can specify the placement rules (the NUMA resources that are dedicated), it is not recommended to charge for any NUMA resources; use Processors instead.