3.2 Scheduling Environment
3.2.1 Scheduling Objects
Moab functions by manipulating a number of elementary objects, including jobs, nodes, reservations, QoS structures, resource managers, and policies. Multiple minor elementary objects and composite objects are also used; these objects are defined in the scheduling dictionary.
3.2.1.1 Jobs
Job information is provided to the Moab scheduler from a resource manager such as Loadleveler, PBS, Wiki, or LSF. Job attributes include ownership of the job, job state, amount and type of resources required by the job, and a wallclock limit indicating how long the resources are required. A job consists of one or more task groups, each of which requests a number of resources of a given type; for example, a job may consist of two task groups, the first asking for a single master task consisting of 1 IBM SP node with at least 512 MB of RAM and the second asking for a set of slave tasks such as 24 IBM SP nodes with at least 128 MB of RAM. Each task group consists of one or more tasks where a task is defined as the minimal independent unit of resources. By default, each task is equivalent to one processor. In SMP environments, however, users may wish to tie one or more processors together with a certain amount of memory and other resources.
3.2.1.1.1 Job States
The job's state indicates its current status and eligibility for execution and can be any of the values listed in the following tables:
| State | Definition |
|---|---|
| deferred | Job that has been held by Moab due to an inability to schedule the job under current conditions. Deferred jobs are held for DEFERTIME before being placed in the idle queue. This process is repeated DEFERCOUNT times before the job is placed in batch hold. |
| hold | Job is idle and is not eligible to run due to a user, (system) administrator, or batch system hold (also, batchhold, systemhold, userhold). |
| idle | Job is currently queued and eligible to run but is not executing (also, notqueued). |
| migrated | Job that has been migrated to a remote peer, but have not yet begun execution. Migrated jobs show can show up on any grid system |
| staged | Job has been migrated to another scheduler but has not yet started executing. Staged jobs show up only when using data staging. |
| State | Definition |
|---|---|
| starting | Batch system has attempted to start the job and the job is currently performing pre-start tasks that may include provisioning resources, staging data, or executing system pre-launch scripts. |
| running | Job is currently executing the user application. |
| suspended | Job was running but has been suspended by the scheduler or an administrator; user application is still in place on the allocated compute resources, but it is not executing. |
| canceling | Job has been canceled and is in process of cleaning up. |
3.2.1.1.2 Task Group (or Req)
A job task group (or req) consists of a request for a single type of resources. Each task group consists of the following components:
- Task Definition — A specification of the elementary resources that compose an individual task.
- Resource Constraints — A specification of conditions that must be met for resource matching to occur. Only resources from nodes that meet all resource constraints may be allocated to the job task group.
- Task Count — The number of task instances required by the task group.
- Task List — The list of nodes on which the task instances are located.
- Task Group Statistics — Statistics tracking resource utilization.
3.2.1.2 Nodes
Moab recognizes a node as a collection of resources with a particular set of associated attributes. This definition is similar to the traditional notion of a node found in a Linux cluster or supercomputer wherein a node is defined as one or more CPUs, associated memory, and possibly other compute resources such as local disk, swap, network adapters, and software licenses. Additionally, this node is described by various attributes such as an architecture type or operating system. Nodes range in size from small uniprocessor PCs to large symmetric multiprocessing (SMP) systems where a single node may consist of hundreds of CPUs and massive amounts of memory.
In many cluster environments, the primary source of information about the configuration and status of a compute node is the resource manager. This information can be augmented by additional information sources including node monitors and information services. Further, extensive node policy and node configuration information can be specified within Moab via the graphical tools or the configuration file. Moab aggregates this information and presents a comprehensive view of the node configuration, usages, and state.
While a node in Moab in most cases represents a standard compute host, nodes may also be used to represent more generalized resources. The GLOBAL node possesses floating resources that are available cluster wide, and created virtual nodes (such as network, software, and data nodes) track and allocate resource usage for other resource types.
For additional node information, see General Node Administration.
3.2.1.3 Advance Reservations
An advance reservation dedicates a block of specific resources for a particular use. Each reservation consists of a list of resources, an access control list, and a time range for enforcing the access control list. The reservation ensures the matching nodes are used according to the access controls and policy constraints within the time frame specified. For example, a reservation could reserve 20 processors and 10 GB of memory for users Bob and John from Friday 6:00 a.m. to Saturday 10:00 p.m. Moab uses advance reservations extensively to manage backfill, guarantee resource availability for active jobs, allow service guarantees, support deadlines, and enable metascheduling. Moab also supports both regularly recurring reservations and the creation of dynamic one-time reservations for special needs. Advance reservations are described in detail in the Advance Reservations overview.
3.2.1.4 Policies
A configuration file specifies policies controls how and when jobs start. Policies include job prioritization, fairness policies, fairshare configuration policies, and scheduling policies.
3.2.1.5 Resources
Jobs, nodes, and reservations all deal with the abstract concept of a resource. A resource in the Moab world is one of the following:
- processors — specify with a simple count value
- memory — specify real memory or RAM in megabytes (MB)
- swap — specify virtual memory or swap in megabytes (MB)
- disk — specify local disk in megabytes (MB)
In addition to these elementary resource types, there are two higher level resource concepts used within Moab: task and the processor equivalent, or PE.
3.2.1.6 Task
A task is a collection of elementary resources that must be allocated together within a single node. For example, a task may consist of one processor, 512 MB of RAM, and 2 GB of local disk. A key aspect of a task is that the resources associated with the task must be allocated as an atomic unit, without spanning node boundaries. A task requesting 2 processors cannot be satisfied by allocating 2 uniprocessor nodes, nor can a task requesting 1 processor and 1 GB of memory be satisfied by allocating 1 processor on 1 node and memory on another.
In Moab, when jobs or reservations request resources, they do so in terms of tasks typically using a task count and a task definition. By default, a task maps directly to a single processor within a job and maps to a full node within reservations. In all cases, this default definition can be overridden by specifying a new task definition.
Within both jobs and reservations, depending on task definition, it is possible to have multiple tasks from the same job mapped to the same node. For example, a job requesting 4 tasks using the default task definition of 1 processor per task, can be satisfied by 2 dual processor nodes.
3.2.1.7 PE
The concept of the processor equivalent, or PE, arose out of the need to translate multi-resource consumption requests into a scalar value. It is not an elementary resource but rather a derived resource metric. It is a measure of the actual impact of a set of requested resources by a job on the total resources available system wide. It is calculated as follows:
PE = MAX(ProcsRequestedByJob / TotalConfiguredProcs,
MemoryRequestedByJob / TotalConfiguredMemory,
DiskRequestedByJob / TotalConfiguredDisk,
SwapRequestedByJob / TotalConfiguredSwap) * TotalConfiguredProcs
For example, if a job requested 20% of the total processors and 50% of the total memory of a 128-processor MPP system, only two such jobs could be supported by this system. The job is essentially using 50% of all available resources since the system can only be scheduled to its most constrained resource—memory in this case. The processor equivalents for this job should be 50% of the processors, or PE = 64.
Another example: Assume a homogeneous 100-node system with 4 processors and 1 GB of memory per node. A job is submitted requesting 2 processors and 768 MB of memory. The PE for this job would be calculated as follows:
PE = MAX(2/(100*4), 768/(100*1024)) * (100*4) = 3.
This result makes sense since the job would be consuming 3/4 of the memory on a 4-processor node.
The calculation works equally well on homogeneous or heterogeneous systems, uniprocessor or large SMP systems.
3.2.1.8 Class (or Queue)
A class (or queue) is a logical container object that implicitly or explicitly applies policies to jobs. In most cases, a class is defined and configured within the resource manager and associated with one or more of the following attributes or constraints:
| Attribute | Description |
|---|---|
| Default Job Attributes | A queue may be associated with a default job duration, default size, or default resource requirements. |
| Host Constraints | A queue may constrain job execution to a particular set of hosts. |
| Job Constraints | A queue may constrain the attributes of jobs that may submitted, including setting limits such as max wallclock time and minimum number of processors. |
| Access List | A queue may constrain who may submit jobs into it based on such things as user lists and group lists. |
| Special Access | A queue may associate special privileges with jobs including adjusted job priority. |
As stated previously, most resource managers allow full class configuration within the resource manager. Where additional class configuration is required, the CLASSCFG parameter may be used.
Moab tracks class usage as a consumable resource allowing sites to limit the number of jobs using a particular class. This is done by monitoring class initiators that may be considered to be a ticket to run in a particular class. Any compute node may simultaneously support several types of classes and any number of initiators of each type. By default, nodes will have a one-to-one mapping between class initiators and configured processors. For every job task run on the node, one class initiator of the appropriate type is consumed. For example, a 3-processor job submitted to the class batch consumes three batch class initiators on the nodes where it runs.
Using queues as consumable resources allows sites to specify various policies by adjusting the class initiator to node mapping. For example, a site running serial jobs may want to allow a particular 8-processor node to run any combination of batch and special jobs subject to the following constraints:
- only 8 jobs of any type allowed simultaneously
- no more than 4 special jobs allowed simultaneously
To enable this policy, the site may set the node's MAXJOB policy to 8 and configure the node with 4 special class initiators and 8 batch class initiators.
In virtually all cases jobs have a one-to-one correspondence between processors requested and class initiators required. However, this is not a requirement, and with special configuration, sites may choose to associate job tasks with arbitrary combinations of class initiator requirements.
In displaying class initiator status, Moab signifies the type and number of class initiators available using the format [<CLASSNAME>:<CLASSCOUNT>]. This is most commonly seen in the output of node status commands indicating the number of configured and available class initiators, or in job status commands when displaying class initiator requirements.
3.2.1.9 Resource Manager (RM)
While other systems may have more strict interpretations of a resource manager and its responsibilities, Moab's multi-resource manager support allows a much more liberal interpretation. In essence, any object that can provide environmental information and environmental control can be used as a resource manager, including sources of resource, workload, credential, or policy information such as scripts, peer services, databases, web services, hardware monitors, or even flats files. Likewise, Moab considers to be a resource manager any tool that provides control over the cluster environment whether that be a license manager, queue manager, checkpoint facility, provisioning manager, network manager, or storage manager.
Moab aggregates information from multiple unrelated sources into a larger more complete world view of the cluster that includes all the information and control found within a standard resource manager such as TORQUE, including node, job, and queue management services. For more information, see the Resource Managers and Interfaces overview.
Arbitrary Resource
Nodes can also be configured to support various arbitrary resources. Use the NODECFG parameter to specify information about such resources. For example, you could configure a node to have 256 MB RAM, 4 processors, 1 GB Swap, and 2 tape drives.