8.1 Monitoring Resources
8.1.1 Resource Overview
A primary task of any resource manager is to monitor the state, health, configuration, and utilization of managed resources. TORQUE is specifically designed to monitor compute hosts for use in a batch environment. TORQUE is not designed to monitor non-compute host resources such as software licenses, networks, file systems, and so forth, although these resources can be integrated into the cluster using some scheduling systems.
With regard to monitoring compute nodes, TORQUE reports about a number of attributes broken into three major categories:
8.1.1.1 Configuration
Configuration includes both detected hardware configuration and specified batch attributes.
| Attribute | Description | Details |
|---|---|---|
| Architecture (arch) | operating system of the node | The value reported is a derivative of the operating system installed. |
| Node Features (properties) | arbitrary string attributes associated with the node | No node features are specified by default. If required, they are set using the nodes file located in the TORQUE_HOME/server_priv directory. They may specify any string and are most commonly used to allow users to request certain subsets of nodes when submitting jobs. |
| Local Disk (size) | configured local disk | By default, local disk space is not monitored. If the MOM configuration size parameter is set, TORQUE will report, in kilobytes, configured disk space within the specified directory. |
| Memory (physmem) | local memory/RAM | Local memory/RAM is monitored and reported in kilobytes. |
| Processors (ncpus/np) | real/virtual processors | The number of processors detected by TORQUE is reported via the ncpus attribute. However, for scheduling purposes, other factors are taken into account. In its default configuration, TORQUE operates in dedicated mode with each node possessing a single virtual processor. In dedicated mode, each job task will consume one virtual processor and TORQUE will accept workload on each node until all virtual processors on that node are in use. While the number of virtual processors per node defaults to 1, this may be configured using the nodes file located in the TORQUE_HOME/server_priv directory. An alternative to dedicated mode is timeshared mode. If TORQUE's timeshared mode is enabled, TORQUE will accept additional workload on each node until the node's maxload limit is reached. |
| Swap (totmem) | virtual memory/Swap | Virtual memory/Swap is monitored and reported in kilobytes. |
8.1.1.2 Utilization
Utilization includes information regarding the amount of node resources currently in use as well as information about who or what is consuming it.
| Attribute | Description | Details |
|---|---|---|
| Disk (size) | local disk availability | By default, local disk space is not monitored. If the MOM configuration size parameter is set, TORQUE will report configured and currently available disk space within the specified directory in kilobytes. |
| Memory (availmem) | real memory/RAM | Available real memory/RAM is monitored and reported in kilobytes. |
| Network (netload) | local network adapter usage | Reports total number of bytes transferred in or out by the network adapter. |
| Processor Utilization (loadave) | node's cpu load average | Reports the node's 1 minute bsd load average. |
8.1.1.3 Node States
State information includes administrative status, general node health information, and general usage status.
| Attribute | Description | Details |
|---|---|---|
| Idle Time (idletime) | time since local keyboard/mouse activity has been detected | Time in seconds since local keyboard/mouse activity has been detected. |
| State (state) | monitored/admin node state |
A node can be in one or more of the following states:
|