3.7.2 TORQUE NVIDIA GPGPUs

The pbs_mom file can now query for GPU hardware information and report status to the pbs_server. gpustatus will appear in pbsnodes output. New commands allow for setting GPU modes and for resetting GPU ECC error counts.

This feature is only available in TORQUE 2.5.6, 3.0.2, and later.

This document assumes that you have installed the NVIDIA CUDA ToolKit and the NVIDIA development drivers on a compute node with a NVIDAI GPU. (Both can be downloaded from http://developer.nvidia.com/category/zone/cuda-zone).

You will want to download the latest version if you run into problems compiling.

If the pbs_server does not have GPUs, it only needs to be configured with --enable-nvidia-gpus. All other systems that have NVIDIA GPUs will need:

• --enable-nvidia-gpus
• --with-nvml-include=DIR (include path for nvml.h)


nvml.h is only found in the NVIDIA CUDA ToolKit.

• --with-nvml-lib=DIR (*lib path for libnvidia-ml)

For example, you would configure the a PBS_SERVER that does not have GPUs, but will be managing compute nodes with NVIDIA GPUs in this way:

Server

Compute nodes (with NVIDIA GPUs)

If all of the compute nodes have the same hardware and software configuration, you can choose to compile on one compute node and then run make packages.

The package files are self-extracting packages that can be copied and executed on your production machines. (Use --help for options.)

For more information, see Compute nodes.

When updating, it is good practice to stop the pbs_server and make a backup of the TORQUE home directory. You will also want to backup the output of qmgr -c 'p s'. The update will only overwrite the binaries.

For further details, see these topics:

• TORQUE configuration
• GPU modes for NVIDIA 260.x driver
• GPU Modes for NVIDIA 270.x driver
• gpu_status
• New NVIDIA GPU support

TORQUE configuration

There are three configuration (./configure) options available for use with Nvidia GPGPUs:

• --enable-nvidia-gpus
• --with-nvml-lib=DIR
• --with-nvml-include=DIR

--enable-nvidia-gpus is used to enable the new features for the Nvidia GPGPUs. By default, the pbs_moms use the nvidia_smi command to interface with the Nvidia GPUs.

To use the NVML (NVIDIA Management Library) API instead of nvidia-smi, configure TORQUE using --with-nvml-lib=DIR and --with-nvml-include=DIR. These commands specify the location of the libnvidia-ml library and the location of the nvml.h include file.

By default, when TORQUE is configured with --enable-nvidia-gpus the $TORQUE_HOME/nodes file is automatically updated with the correct GPU count for each MOM node.

GPU modes for NVIDIA 260.x driver

• 0 – Default - Shared mode available for multiple processes
• 1 – Exclusive - Only one COMPUTE thread is allowed to run on the GPU
• 2 – Prohibited - No COMPUTE contexts are allowed to run on the GPU

GPU Modes for NVIDIA 270.x driver

• 0 – Default - Shared mode available for multiple processes
• 1 – Exclusive Thread - Only one COMPUTE thread is allowed to run on the GPU (v260 exclusive)
• 2 – Prohibited - No COMPUTE contexts are allowed to run on the GPU
• 3 – Exclusive Process - Only one COMPUTE process is allowed to run on the GPU

gpu_status

New NVIDIA GPU support

qsub allows specifying required compute mode when requesting GPUs. If no GPU mode is requested, it will default to "exclusive" for Nvidia driver version 260 or "exclusive_thread" for Nvidia driver version 270 and above.

• qsub -l nodes=1:ppn=1:gpus=1
• qsub -l nodes=1:gpus=1
• qsub -l nodes=1:gpus=1:exclusive_thread
• qsub -l nodes=1:gpus=1:exclusive_process
• qsub -l nodes=1:gpus=1:reseterr
• qsub -l nodes=1:gpus=1:reseterr:exclusive_thread (exclusive_thread:reseterr)
• qsub -l nodes=1:gpus=1:reseterr:exclusive_process

Related topics 

• Scheduling GPUs
• Using GPUs with NUMA

© 2012 Adaptive Computing