TORQUE Resource Manager

3.6 Scheduling Cores

In TORQUE 2.4 and later, users can request specific cores on a node at job submission by using geometry requests. To use this feature, users specify the procs_bitmap resource request of qsub -l at job submission.

3.6.1 Geometry Request Configuration

A Linux kernel of 2.6, or later, is required to use geometry requests since this feature uses Linux cpusets in its implementation. In order to use this feature, the cpuset directory has to be mounted. For more information on how to mount the cpuset directory, see Linux Cpuset Support. If the operating evnironment is suitable for geometry requests, configure TORQUE with the --enable-geometry-requests option.

> ./configure --prefix=/home/john/torque --enable-geometry-requests

TORQUE is configured to install to /home/john/torque and to enable the geometry requests feature.

Note The geometry request feature uses a subset of the cpusets feature. When you configure TORQUE using --enable-cpuset and --enable-geometry-requests at the same time, and use -l procs_bitmap=X, the job will get the requested cpuset. Otherwise, the job is treated as if only --enable-cpuset was configured.

3.6.2 Geometry Request Usage

Once enabled, users can submit jobs with a geometry request by using the procs_bitmap=<string> resource request. procs_bitmap requires a numerical string made up of 1's and 0's. A 0 in the bitmap means the job can not run on the core that matches the 0's index in the bitmap. The index is in reverse order of the number of cores available. If a job is submitted with procs_bitmap=1011, then the job requests a node with four free cores, and uses only cores one, two, and four.

Note The geometry request feature requires a node that has all cores free. A job with a geometry request cannot run on a node that has cores that are busy, even if the node has more than enough cores available to run the job.

qsub -l procs_bitmap=0011 ossl.sh

The job ossl.sh is submitted with a geometry request of 0011.

In the above example, the submitted job can run only on a node that has four cores. When a suitable node is found, the job runs exclusively on cores one and two.

3.6.3 Geometry Request Considerations

As stated above, jobs with geometry requests require a node with all of its cores available. After the job starts running on the requested cores, the node cannot run other jobs, even if the node has enough free cores to meet the requirements of the other jobs. Once the geometry requesting job is done, the node is available to other jobs again.