Moab Workload Manager

7.1.6 Personal Reservations (a.k.a User Reservations) - Enabling Reservations for End Users

By default, advance reservations are only available to scheduler administrators. While administrators may create and manage reservations to provide resource access to end-users, end-users cannot create, modify, or destroy these reservations. Moab extends the ability to manage reservations to end-users and provides control facilities to keep these features manageable. Reservations created by end-users are called personal reservations or user reservations.

7.1.6.1 Enabling Personal Reservation Management

User, or personal reservations can be enabled on a per QoS basis by setting the ENABLEUSERRSV flag as in the following example:

QOSCFG[titan]    QFLAGS=ENABLEUSERRSV # allow 'titan' QOS jobs to create user reservations
USERCFG[DEFAULT] QDEF=titan           # allow all users to access 'titan' QOS
...

If set, end-users are allowed to create, modify, cancel, and query reservations they own. As with jobs, users may associate a personal reservation with any QoS or account to which they have access. This is accomplished by specifying per reservation accountable credentials as in the following example:

> mrsvctl -c -S AQOS=titan -h node01 -d 1:00:00 -s 1:30:00

Note:  reservation test.126 created

As in the preceding example, a non-administrator user who wants to create a reservation must ALWAYS specify an accountable QoS with the mrsvctl -S flag. This specified QoS must have the ENABLEUSERRSV flag. By default, a personal reservation is created with an ACL of only the user who created it.

Example: Allow All Users in Engineering Group to Create Personal Reservations

QOSCFG[rsv]     QFLAGS=ENABLEUSERRSV # allow 'rsv' QOS jobs to create user reservations
GROUPCFG[sales] QDEF=rsv             # allow all users in group sales to access 'rsv' QOS
...

Example: Allow Specific Users to Create Personal Reservations

# special qos has higher job priority and ability to create user reservations
QOSCFG[special] QFLAGS=ENABLEUSERRSV
QOSCFG[special] PRIORITY=1000

# allow betty and steve to use the special qos
USERCFG[betty]  QDEF=special
USERCFG[steve]  QLIST=fast,special,basic QDEF=rsv
...

7.1.6.2 Reservation Accountability

Personal reservations must be configured with a set of accountable credentials. These credentials (user, group, account, and so forth) indicate who is responsible for the resources dedicated by the reservation. If resources are dedicated by a reservation but not consumed by a job, these resources can be charged against the specified accountable credentials. Administrators are allowed to create reservations and specify any accountable credentials for that reservation. While end-users can also be allowed to create and otherwise modify personal reservations, they are only allowed to create reservations with accountable credentials to which they have access. Further, while administrators may manage any reservation, end-users may only control reservations they own.

Like jobs, reservation accountable credentials specify which credentials are charged for reservation usage and what policies are enforced as far as usage limits and allocation management is concerned. (See the mrsvctl command documentation for more information on setting personal reservation credentials.) While similar to jobs, personal reservations do have a separate set of usage limits and different allocation charging policies.

7.1.6.2.1 Reservation Allocation and Charging

If an allocation management system is used, Moab can be configured to charge users for both the creation of the reservation and the reservation itself. Using the QoS RSVCREATIONCOST and DEDRESCOST attributes, a site may specify a per-QoS cost for each reservation created. The RSVCREATIONCOST cost is a flat charge in allocation units applied regardless of whether the reservation is used. The DEDRESCOST tracks reserved (or dedicated) resources allowing sites to charge by the quantity of resources (in processor-seconds) that are actually reserved. As with jobs, these charges are applied against the personal reservation account.

7.1.6.2.2 Setting Reservation Default Attributes

Organizations can use reservation profiles to set default attributes for personal reservations. These attributes can include reservation aspects such as management policies, charging credentials, ACLs, host constraints, and time frame settings.

7.1.6.3 Reservation Limits

Allowing end-users the ability to create advance reservations can lead to potentially unfair and unproductive resource usage. This results from the fact that by default, there is nothing to prevent a user from reserving all resources in a given system or reserving resources during time slots that would greatly impede the scheduler's ability to schedule jobs efficiently. Because of this, it is highly advised that sites initially place either usage or allocation based constraints on the use of personal reservations.

While time spanning throttling policies are a significant step in the direction of end-user advance reservation management, it is important to track actual site usage of the advance reservation facility. It is still likely that further usage policies are required at each site to prevent reservation misuse and provide an optimally useful system.

7.1.6.3.1 Reservation Usage Limits

Personal reservation usage limits may be enforced using the attributes specified in the following table. These limits can be used to constrain the resources reserved in any single reservation as well as the total quantity of resources that may be reserved cumulatively across all reservations.

Reservation usage limits are independent of job limits imposed on users, groups, and other credentials.

Limit Description
RMAXDURATION Limits the duration (in seconds) of any single personal reservation.
RMAXPROC Limits the size (in processors) of any single personal reservation.
RMAXPS Limits the size (in processor-seconds) of any single personal reservation.
RMAXCOUNT Limits the total number of personal reservations a credential may have active at any given moment.
RMAXTOTALDURATION Limits the total duration of personal reservations a credential may have active at any given moment.
RMAXTOTALPROC Limits the total number of processors a credential may reserve active at any given moment.
RMAXTOTALPS Limits the total number of processor-seconds a credential may reserve active at any given moment.

Limit	Description
RMAXDURATION	Limits the duration (in seconds) of any single personal reservation.
RMAXPROC	Limits the size (in processors) of any single personal reservation.
RMAXPS	Limits the size (in processor-seconds) of any single personal reservation.
RMAXCOUNT	Limits the total number of personal reservations a credential may have active at any given moment.
RMAXTOTALDURATION	Limits the total duration of personal reservations a credential may have active at any given moment.
RMAXTOTALPROC	Limits the total number of processors a credential may reserve active at any given moment.
RMAXTOTALPS	Limits the total number of processor-seconds a credential may reserve active at any given moment.

Example: Constrain Size/Duration of Personal Reservations

# special qos can only make reservations up to 5 procs and 2 hours
QOSCFG[special] QFLAGS=ENABLEUSERRSV RMAXPROC=5 RMAXDURATION=2:00:00

7.1.6.3.2 Reservation Masks (a.k.a. Personal Reservation Sandbox)

Additional control over when and where personal reservations can be created can be enabled through the use of a mask reservation. To create a mask reservation, a standard standing reservation can be created with the flag ALLOWPRSV set. If such a reservation exists, the scheduler constrains all personal reservation requests to only use resources within this reservation. This effectively allows a site to constrain the times of the day and days of the week during which a personal reservation can be created and limits the requests to a specific subset of resources. Note that reservation masks can be used in conjunction with reservation usage limits to provide a finer level of control. Also note that reservation masks still support the concept of the reservation ACL, which can constrain exactly who or what may request the resources within the reservation mask. To enable global access to a standing reservation with the flag ALLOWPRSV, the ACL USER==ALL+ is assigned to the reservation by default.

Personal reservation masks/sandboxes only impact users who are not Moab administrators. Moab administrators are allowed to create personal reservations on any resource they can normally access.

Example: Create Personal Reservation Sandbox

# allow personal reservations to only reserve nodes 1-8 on Tues and Weds
SRCFG[sandbox] HOSTLIST=node0[1-8] FLAGS=ALLOWPRSV 
SRCFG[sandbox] PERIOD=DAY DAYS=TUE,WED

7.1.6.4 Reservation and Job Binding

Moab allows job-to-reservation binding to be configured at an administrator or end-user level. This binding constrains how job to reservation mapping is allowed.

7.1.6.4.1 Constraining a job to only run in a particular reservation

Jobs may be bound to a particular reservation at submit time (using the RM extension ADVRES) or dynamically using the mjobctl command. (See Job to Reservation Mapping.) In either case, once bound to a reservation, a job may only run in that reservation even if other resources may be found outside of that reservation. The mjobctl command may also be used to dynamically release a job from reservation binding.

Example 1: Bind job to reservation

> mjobctl -m flags+=advres:grid.3 job1352

Example 2: Release job from reservation binding

> mjobctl -m flags-=advres job1352

7.1.6.4.2 Constraining a Reservation to Only Accept Certain Jobs

Binding a job to a reservation is independent of binding a reservation to a job. For example, a reservation may be created for user steve. User steve may then submit a number of jobs including one that is bound to that reservation using the ADVRES attribute. However, this binding simply forces that one job to use the reservation, it does not prevent the reservation from accepting other jobs submitted by user steve. To prevent these other jobs from using the reserved resources, reservation to job binding must occur. This binding is accomplished by specifying either general job binding or specific job binding.

General job binding is the most flexible form of binding. Using the BYNAME attribute, a reservation may be created that only accepts jobs specifically bound to it.

Specific job binding is more constraining. This form of binding causes the reservation to only accept specific jobs, regardless of other job attributes and is set using the JOB reservation ACL.

Example 1: Configure a reservation to accept only jobs that are bound to it

> mrsvctl -m flags+=byname grid.3

Example 2: Remove general reservation to job binding

> mrsvctl -m flags-=byname grid.3

Example 3: Configure a reservation to accept a specific job

> mrsvctl -m -a JOB=3456 grid.3

Example 4: Remove a specific reservation to job binding

> mrsvctl -m -a JOB=3456 grid.3 --flags=unset