Moab Adaptive Computing Suite Administrator's Guide 5.4
Home Up Previous Next

2.8 Automating Resource Access Requests

Moab can be set up so that the local system automatically allocates and de-allocates utility computing resources based on workload, time of day, resource failures, or other conditions. This allows organizations to schedule their cluster size according to current or anticipated needs.

2.8.1 Load Based Allocation

Load based allocation allows organizations to adjust their available resources based on the nature and quantity of queued workload. This can be set up dynamically so that as more jobs are added and workload response times increase, resources are automatically allocated to improve performance.

Configuring this capability consists of defining the allocation performance thresholds and the default allocation attributes. In all cases, these attributes are defined on the hosting center resource manager object using the RMCFG parameter.

Defining Thresholds

  • UCTHRESHOLD - The maximum backlog (in seconds) allowed before utility computing resources are allocated.
    Note If QoS based utility computing is enabled, only the backlog of eligible jobs is considered.
  • UCTHRESHOLDDURATION - The duration of time the threshold must be exceeded before utility computing resources are allocated—used to prevent temporary workload spikes from triggering an undesired allocation.

Defining Allocations

  • UCALLOCDURATION - The default duration of the utility computing allocation.
  • UCALLOCSIZE - The default size of the utility computing allocation in processors.
    Note If not specified, Moab allocates up to UCMAXSIZE processors if required to meet current workload needs.
  • UCMAXSIZE - The maximum number of processors allowed to be allocated from the utility computing center regardless of workload processors.
    Note If not specified, Moab allocates up to UCMAXSIZE processors if required to meet current workload needs.
  • UCNODEPROFILE - Node profile defining what resources should be allocated.
    Note If not specified, Moab allocates a node with attributes that satisfy the needs of the highest priority eligible job that is authorized to access utility resources.

It is also possible to allocate resources for particular credentials. The following example automatically allocates nodes when the backlog of the specified QoS reaches the specified limit:

QOSCFG[high] TRIGGER=atype=internal,action=RM:odm:modify:nodes:+=1,etype=start,threshold=backlog>100,multifire=true,failoffset=1:00
QOSCFG[high] TRIGGER=atype=internal,action=RM:odm:modify:nodes:-=1,etype=start,threshold=backlog<100,failoffset=1:00,multifire=true,requires=UC,sets=!done
QOSCFG[high] TRIGGER=atype=exec,action="/bin/hostname",etype=start,requires=done.UC,unsets=UC,multifire=true
...

2.8.2 Time Based Allocation

If you have predictable workload fluctuations, you may want to consider time based utility computing allocations. This is particularly valuable if significant provisioning or customization time is required to prepare the resources, or if large data sets are associated with the workload. In this model, organizations automatically send an allocation request to a utility computing organization on a periodic basis using the Moab Standing Trigger capability. This facility allows great flexibility in how, when, and under what conditions these requests are sent. For example, one site may request resources from 9:00 a.m. until 6:00 p.m. on business days. Another site may want resources for two hours at the start of every month. Standing triggers allow the specification of the period and timeframe which the triggers activate. They also allow specification of other consideration factors allowing a policy such as allocate resources every Wednesday and Friday at 11:30 a.m. if the Genesis data set is available and the estimated local queue time is over four hours.

With each trigger, the action can be set to execute the mrmctl -x command with information regarding the type and quantity of resources to allocate.

2.8.3 Failure Based Allocation

As with time based allocation, failure based allocation is trigger based. However, in this case, a trigger is attached to one or more key resource manager interfaces. If a failure is detected, and the resource manager does not recover within the resource manager's MAXFAILTIME, the trigger will be launched, and additional resources will be provisioned to handle the workload.

RMCFG[TORQUE]          TRIGGER=atype=internal,action=RM:ODM:modify:nodes:1,etype=start,threshold=availability<300,multifire=true,failoffset=1:00
...

In addition to resource manager based triggers, a trigger can be attached to all or specific nodes to automate allocation of new resources should existing resources fail. The following example shows how to automate the allocation of new resources when a node fails. In this example, should node01 ever report a failure state for 1 minute, Moab automatically allocates a new node with at least the same architecture, operating system, and memory as the failed node:

NODECFG[node01]     TRIGGER=atype=internal,action=RM:odm:modify:nodes:+=1,etype=failure,failoffset=1:00,multifire=false
...
Home Up Previous Next