12.3 Node Specific Policies
Node policies within Moab allow specification of not only how the node's load should be managed, but who can use the node, and how the node and jobs should respond to various events. These policies allow a site administrator to specify on a node by node basis what the node will and will not support. Node policies may be applied to specific nodes or applied system-wide using the specification NODECFG[DEFAULT] ....
12.3.1 Node Usage/Throttling Policies
This policy constrains the number of total independent jobs a given node may run simultaneously. It can only be specified via the NODECFG parameter.
 |
If node pools are specified via the node's POOL attribute, the MAXJOB limit is applied across the aggregate pool as in the following example:
NODECFG[node01-a] POOL=node01 NODECFG[node01-b] POOL=node01 NODECFG[node01-c] POOL=node01 NODECFG[node01-d] POOL=node01 NODECFG[node02-a] POOL=node02 NODECFG[node02-b] POOL=node02 NODECFG[node02-c] POOL=node02 NODECFG[node02-d] POOL=node02 # only allow one job to run per node pool NODECFG[DEFAULT] MAXJOB=1 |
This policy constrains the number of total independent jobs a given node may run simultaneously associated with any single user. It can only be specified via the NODECFG parameter.
This policy constrains the number of total independent jobs a given node may run simultaneously associated with any single group. It can only be specified via the NODECFG parameter.
This policy constrains the CPU load the node will support as opposed to the number of jobs. If the node's load exceeds the MAXLOAD limit and the NODELOADPOLICY parameter is set to ADJUSTSTATE, the node is marked busy. This maximum load policy can also be applied system wide using the parameter NODEMAXLOAD.
This policy constrains the number of total dedicated processor-equivalents a given node may support simultaneously. It can only be specified via the NODECFG parameter.
This policy constrains the number of total dedicated processors a given node may support simultaneously. It can only be specified via the NODECFG parameter.
This policy constrains the number of total processors a given node may have dedicated to any single user. It can only be specified via the NODECFG parameter.
This policy constrains the number of total processors a given node may have dedicated to any single group. It can only be specified via the NODECFG parameter.
|
Node throttling policies are used strictly as constraints. If a node is defined as having a single processor or the NODEACCESSPOLICY is set to SINGLETASK, and a MAXPROC policy of 4 is specified, Moab will not run more than one task per node. A node's configured processors must be specified so that multiple jobs may run and then the MAXJOB policy will be effective. The number of configured processors per node is specified on a resource manager specific basis. PBS, for example, allows this to be adjusted by setting the number of virtual processors with the np parameter for each node in the PBS nodes file. |
Example:
NODECFG[node024] MAXJOB=4 MAXJOBPERUSER=2 NODECFG[node025] MAXJOB=2 NODECFG[node026] MAXJOBPERUSER=1 NODECFG[DEFAULT] MAXLOAD=2.5 ...
12.3.2 Desktop Management Policies (Desktop Harvesting)
KBDDETECTPOLICY, MINRESUMEKBDIDLETIME, and MINPREEMPTLOAD
Sometimes a site administrator would like to add normal desktops to their cluster but are afraid that jobs might overrun normal desktop usage. Using these three parameters desktop support can be enabled and configured based on a particular site administrator's needs. KBDDETECTPOLICY tells the scheduler what to do when local keyboard usage is detected. When it is set to "DRAIN" any jobs currently running on the node are allowed to finish but no new jobs will run on the node until MINRESUMEKBDIDLETIME is reached. When KBDDETECTPOLICY is set to "PREEMPT" any jobs running on the system are preempted and the full system reverts to desktop usage until no keyboard activity is detected for MINRESUMEKBDIDLETIME. Desktop support can be further configured using MINPREEMPTLOAD. This parameter tells Moab to continue scheduling jobs on a node even when local keyboard usage is detected as long as the total system load remains below the specified value. These three parameters can be configured globally or for each particular node.
Example:
NODECFG[node024] KBDDETECTPOLICY=DRAIN # when local keyboard is active jobs will be allowed to finish
NODECFG[node024] MINRESUMEKBDIDLETIME=600 # no jobs will be scheduled on node until keyboard has been
# idle for 10 minutes
NODECFG[node024] MINPREEMPTLOAD=0.5 # as long as system load remains below 0.5 jobs will be scheduled,
# even if keyboard is active
...
OR:
NODECFG[DEFAULT] KBDDETECTPOLICY=PREEMPT # any jobs running on any nodes will be preempted if local
# keyboard usage is detected
NODECFG[DEFAULT] MINRESUMEKBDIDLETIME=600 # no jobs will be scheduled any node until local keyboard has been
# idle for 10 minutes
NODECFG[DEFAULT] MINPREEMPTLOAD=0.5 # as long as the local system load remains below 0.5, jobs will be
# scheduled on the node, even if keyboard is active
...
12.3.3 Node Access Policies
While most sites require only a single cluster wide node access policy (commonly set using NODEACCESSPOLICY), it is possible to specify this policy on a node by node basis using the ACCESS attributes of the NODECFG parameter. This attribute may be set to any of the valid node access policy values listed in the Node Access Policies section.
Example
To set a global policy of SINGLETASK on all nodes except nodes 13 and 14, use the following:
# by default, enforce dedicated node access on all nodes NODEACCESSPOLICY SINGLETASK # allow nodes 13 and 14 to be shared NODECFG[node13] ACCESS=SHARED NODECFG[node14] ACCESS=SHARED
See Also