Node policies within Moab allow specification of not only how the node's load should be managed, but who can use the node, and how the node and jobs should respond to various events. These policies allow a site administrator to specify on a node by node basis what the node will and will not support. Node policies may be applied to specific nodes or applied system-wide using the specification NODECFG[DEFAULT] ....

12.3.1 Node Usage/Throttling Policies

MAXJOB

This policy constrains the number of total independent jobs a given node may run simultaneously. It can only be specified via the NODECFG parameter.

If node pools are specified via the node's POOL attribute, the MAXJOB limit is applied across the aggregate pool as in the following example: NODECFG[node01-a] POOL=node01 NODECFG[node01-b] POOL=node01 NODECFG[node01-c] POOL=node01 NODECFG[node01-d] POOL=node01 NODECFG[node02-a] POOL=node02 NODECFG[node02-b] POOL=node02 NODECFG[node02-c] POOL=node02 NODECFG[node02-d] POOL=node02 # only allow one job to run per node pool NODECFG[DEFAULT] MAXJOB=1

On Cray XT systems, use the NID (node id) instead of the node name. For more information, see Configuring the moab.cfg file.

MAXJOBPERUSER

Constrains the number of total independent jobs a given node may run simultaneously associated with any single user. It can only be specified via the NODECFG parameter.

MAXJOBPERGROUP

Constrains the number of total independent jobs a given node may run simultaneously associated with any single group. It can only be specified via the NODECFG parameter.

MAXLOAD

MAXLOAD constrains the CPU load the node will support as opposed to the number of jobs. This maximum load policy can also be applied system wide using the parameter NODEMAXLOAD.

MAXPE

This policy constrains the number of total dedicated processor-equivalents a given node may support simultaneously. It can only be specified via the NODECFG parameter.

MAXPROC

This policy constrains the number of total dedicated processors a given node may support simultaneously. It can only be specified via the NODECFG parameter.

MAXPROCPERUSER

This policy constrains the number of total processors a given node may have dedicated to any single user. It can only be specified via the NODECFG parameter.

MAXPROCPERGROUP

This policy constrains the number of total processors a given node may have dedicated to any single group. It can only be specified via the NODECFG parameter.

Node throttling policies are used strictly as constraints. If a node is defined as having a single processor or the NODEACCESSPOLICY is set to SINGLETASK, and a MAXPROC policy of 4 is specified, Moab will not run more than one task per node. A node's configured processors must be specified so that multiple jobs may run and then the MAXJOB policy will be effective. The number of configured processors per node is specified on a resource manager specific basis. PBS, for example, allows this to be adjusted by setting the number of virtual processors with the np parameter for each node in the PBS nodes file.

Example:

NODECFG[node024] MAXJOB=4 MAXJOBPERUSER=2
NODECFG[node025] MAXJOB=2
NODECFG[node026] MAXJOBPERUSER=1
NODECFG[DEFAULT] MAXLOAD=2.5
...

12.3.2 Desktop Management Policies (Desktop Harvesting)

KBDDETECTPOLICY, MINRESUMEKBDIDLETIME, and MINPREEMPTLOAD

Sometimes a site administrator would like to add normal desktops to their cluster but are afraid that jobs might overrun normal desktop usage. Using these three parameters desktop support can be enabled and configured based on a particular site administrator's needs. KBDDETECTPOLICY tells the scheduler what to do when local keyboard usage is detected. When it is set to "DRAIN" any jobs currently running on the node are allowed to finish but no new jobs will run on the node until MINRESUMEKBDIDLETIME is reached. When KBDDETECTPOLICY is set to "PREEMPT" any jobs running on the system are preempted and the full system reverts to desktop usage until no keyboard activity is detected for MINRESUMEKBDIDLETIME. Desktop support can be further configured using MINPREEMPTLOAD. This parameter tells Moab to continue scheduling jobs on a node even when local keyboard usage is detected as long as the total system load remains below the specified value. These three parameters can be configured globally or for each particular node.

Example:

NODECFG[node024] KBDDETECTPOLICY=DRAIN     # when local keyboard is active jobs will be allowed to finish
NODECFG[node024] MINRESUMEKBDIDLETIME=600  # no jobs will be scheduled on node until keyboard has been 
                                           # idle for 10 minutes
NODECFG[node024] MINPREEMPTLOAD=0.5        # as long as system load remains below 0.5 jobs will be scheduled, 
                                           # even if keyboard is active
...

NODECFG[DEFAULT] KBDDETECTPOLICY=PREEMPT   # any jobs running on any nodes will be preempted if local 
                                           # keyboard usage is detected
NODECFG[DEFAULT] MINRESUMEKBDIDLETIME=600  # no jobs will be scheduled any node until local keyboard has been 
                                           # idle for 10 minutes
NODECFG[DEFAULT] MINPREEMPTLOAD=0.5        # as long as the local system load remains below 0.5, jobs will be 
                                           # scheduled on the node, even if keyboard is active
...

12.3.3 Node Access Policies

While most sites require only a single cluster wide node access policy (commonly set using NODEACCESSPOLICY), it is possible to specify this policy on a node by node basis using the ACCESSattributes of the NODECFG parameter. This attribute may be set to any of the valid node access policy values listed in the Node Access Policies section.

Example

To set a global policy of SINGLETASK on all nodes except nodes 13 and 14, use the following:

# by default, enforce dedicated node access on all nodes
NODEACCESSPOLICY  SINGLETASK
# allow nodes 13 and 14 to be shared
NODECFG[node13]   ACCESS=SHARED
NODECFG[node14]   ACCESS=SHARED

See Also

• mnodectl