Requesting Resources

Specify the CPU clock frequency for each node requested for this job. A cpuclock request applies to every processor on every node in the request. Specifying varying CPU frequencies for different nodes or different processors on nodes in a single job request is not supported.

Not all processors support all possible frequencies or ACPI states. If the requested frequency is not supported by the CPU, the nearest frequency is used.

ALPS 1.4 or later is required when using cpuclock on Cray.

The clock frequency can be specified via:

• a number that indicates the clock frequency (with or without the SI unit suffix).

  qsub -l cpuclock=1800,nodes=2 script.sh

  qsub -l cpuclock=1800mhz,nodes=2 script.sh

  This job requests 2 nodes and specifies their CPU frequencies should be set to 1800 MHz.

• a Linux power governor policy name. The governor names are:

  • performance: This governor instructs Linux to operate each logical processor at its maximum clock frequency.

    This setting consumes the most power and workload executes at the fastest possible speed.

  • powersave: This governor instructs Linux to operate each logical processor at its minimum clock frequency.

    This setting executes workload at the slowest possible speed. This setting does not necessarily consume the least amount of power since applications execute slower, and may actually consume more energy because of the additional time needed to complete the workload's execution.

  • ondemand: This governor dynamically switches the logical processor's clock frequency to the maximum value when system load is high and to the minimum value when the system load is low.

    This setting causes workload to execute at the fastest possible speed or the slowest possible speed, depending on OS load. The system switches between consuming the most power and the least power.

    

    The power saving benefits of ondemand might be non-existent due to frequency switching latency if the system load causes clock frequency changes too often.

    This has been true for older processors since changing the clock frequency required putting the processor into the C3 "sleep" state, changing its clock frequency, and then waking it up, all of which required a significant amount of time.

    Newer processors, such as the Intel Xeon E5-2600 Sandy Bridge processors, can change clock frequency dynamically and much faster.

  • conservative: This governor operates like the ondemand governor but is more conservative in switching between frequencies. It switches more gradually and uses all possible clock frequencies.

    This governor can switch to an intermediate clock frequency if it seems appropriate to the system load and usage, which the ondemand governor does not do.

  qsub -l cpuclock=performance,nodes=2 script.sh

  This job requests 2 nodes and specifies their CPU frequencies should be set to the performance power governor policy.

• an ACPI performance state (or P-state) with or without the P prefix. P-states are a special range of values (0-15) that map to specific frequencies. Not all processors support all 16 states, however, they all start at P0. P0 sets the CPU clock frequency to the highest performance state which runs at the maximum frequency. P15 sets the CPU clock frequency to the lowest performance state which runs at the lowest frequency.

  qsub -l cpuclock=3,nodes=2 script.sh
  qsub -l cpuclock=p3,nodes=2 script.sh

  This job requests 2 nodes and specifies their CPU frequencies should be set to a performance state of 3.

When reviewing job or node properties when cpuclock was used, be mindful of unit conversion. The OS reports frequency in Hz, not MHz or GHz.

Specifies a user owned epilogue script which will be run before the system epilogue and epilogue.user scripts at the completion of a job. The syntax is epilogue=<file>. The file can be designated with an absolute or relative path.

For more information, see Appendix G: Prologue and Epilogue Scripts.

Specifies a property or feature for the job. Feature corresponds to TORQUE node properties and Moab features.

qsub script.sh -l procs=10,feature=bigmem

Maximum amount of physical memory used by the job. Ignored on Darwin, Digital Unix, Free BSD, HPUX 11, IRIX, NetBSD, and SunOS. Not implemented on AIX and HPUX 10.

The mem resource will only work for single-node jobs. If your job requires multiple nodes, use pmem instead.

The number of processors in one task where a task cannot span nodes.

You cannot request both ncpus and nodes in the same job.

Number and/or type of nodes to be reserved for exclusive use by the job. The value is one or more node_specs joined with the + (plus) character: node_spec[+node_spec...]. Each node_spec is a number of nodes required of the type declared in the node_spec and a name of one or more properties desired for the nodes. The number, the name, and each property in the node_spec are separated by a : (colon). If no number is specified, one (1) is assumed. The name of a node is its hostname. The properties of nodes are:

• ppn=# - Specify the number of virtual processors per node requested for this job.

  The number of virtual processors available on a node by default is 1, but it can be configured in the $TORQUE_HOME/server_priv/nodes file using the np attribute (see Server Node File Configuration). The virtual processor can relate to a physical core on the node or it can be interpreted as an "execution slot" such as on sites that set the node np value greater than the number of physical cores (or hyper-thread contexts). The ppn value is a characteristic of the hardware, system, and site, and its value is to be determined by the administrator.

• gpus=# - Specify the number of GPUs per node requested for this job.

  The number of GPUs available on a node can be configured in the $TORQUE_HOME/server_priv/nodes file using the gpu attribute (see Server Node File Configuration). The GPU value is a characteristic of the hardware, system, and site, and its value is to be determined by the administrator.

• property - A string assigned by the system administrator specifying a node's features. Check with your administrator as to the node names and properties available to you.

TORQUE does not have a TPN (tasks per node) property. You can specify TPN in Moab Workload Manager with TORQUE as your resource manager, but TORQUE does not recognize the property when it is submitted directly to it via qsub.

See qsub -l nodes for examples.

By default, the node resource is mapped to a virtual node (that is, directly to a processor, not a full physical compute node). This behavior can be changed within Maui or Moab by setting the JOBNODEMATCHPOLICY parameter. See Appendix A: Moab Parameters for more information.

Allows a user to specify site specific information. This resource is provided for use by the site's scheduling policy. The allowable values and effect on job placement is site dependent.

This does not work for msub using Moab and Maui.

(Applicable in version 2.5.0 and later.) The number of processors to be allocated to a job. The processors can come from one or more qualified node(s). Only one procs declaration may be used per submitted qsub command.

> qsub -l nodes=3 -1 procs=2

A string made up of 1's and 0's in reverse order of the processor cores requested. A procs_bitmap=1110 means the job requests a node that has four available cores, but the job runs exclusively on cores two, three, and four. With this bitmap, core one is not used.

For more information, see Scheduling Cores.

Specifies a user owned prologue script which will be run after the system prologue and prologue.user scripts at the beginning of a job. The syntax is prologue=<file>. The file can be designated with an absolute or relative path.

For more information, see Appendix G: Prologue and Epilogue Scripts.

For TORQUE, this resource has no meaning. It is passed on to the scheduler for interpretation. In the Moab scheduler, the size resource is intended for use in Cray installations only.