Administrative Reservations

7.0 Controlling Resource Access - Reservations, Partitions, and QoS Facilities > Advance Reservations > Administrative Reservations

Conventions

7.1.2 Administrative Reservations

Annotating Administrative Reservations
Using Reservation Profiles
Optimizing Maintenance Reservations

Administrative reservations behave much like standing reservations but are generally created to address non-periodic, one-time issues. All administrative reservations are created using the mrsvctl -c (or setres) command and are persistent until they expire or are removed using the mrsvctl -r (or releaseres) command.

7.1.2-A Annotating Administrative Reservations

Reservations can be labeled and annotated using comments allowing other administrators, local users, portals and other services to obtain more detailed information regarding the reservations. Naming and annotations are configured using the -n and -D options of the mrsvctl command respectively, as in the following example:

> mrsvctl -c -D 'testing infiniband performance' -n nettest -h 'r:agt[15-245]'

7.1.2-B Using Reservation Profiles

You can set up reservation profiles to avoid manually and repetitively inputting standard reservation attributes. Profiles can specify reservation names, descriptions, ACLs, durations, host lists, triggers, flags, and other aspects that are commonly used. With a reservation profile defined, a new administrative reservation can be created that uses this profile by specifying the -P flag as in the following example.

Example 7-1:

RSVPROFILE[mtn1] TRIGGER=AType=exec,Action="/tmp/trigger1.sh",EType=start
RSVPROFILE[mtn1] USERLIST=steve,marym
RSVPROFILE[mtn1] HOSTEXP="r:50-250"

> mrsvctl -c -P mtn1 -s 12:00:00_10/03 -d 2:00:00

Example 7-2: Non-Blocking System Reservations with Scheduler Pause

RSVPROFILE[pause] TRIGGER=atype=exec,etype=start,action="/opt/moab/bin/mschedctl -p"
RSVPROFILE[pause] TRIGGER=atype=exec,etype=cancel,action="/opt/moab/bin/mschedctl -r"
RSVPROFILE[pause] TRIGGER=atype=exec,etype=end,action="/opt/moab/bin/mschedctl -r"

> mrsvctl -c -P pause -s 12:00:00_10/03 -d 2:00:00

7.1.2-C Optimizing Maintenance Reservations

Any reservation causes some negative impact on cluster performance as it further limits the scheduler's ability to optimize scheduling decisions. You can mitigate this impact by using flexible ACLs and triggers.

In particular, a maintenance reservation can be configured to reduce its effective reservation shadow by allowing overlap with checkpointable/preemptible jobs until the time the reservation becomes active. This can be done using a series of triggers that perform the following actions:

Modify the reservation to disable preemption access.
Preempt jobs that may overlap the reservation.
Cancel any jobs that failed to properly checkpoint and exit.

The following example highlights one possible configuration:

RSVPROFILE[adm1] JOBATTRLIST=PREEMPTEE
RSVPROFILE[adm1] DESCRIPTION="regular system maintenance"
RSVPROFILE[adm1] TRIGGER=EType=start,Offset=-300,AType=internal,Action="rsv:-:modify:acl:jattr-=PREEMPTEE"
RSVPROFILE[adm1] TRIGGER=EType=start,Offset=-240,AType=jobpreempt,Action="checkpoint"
RSVPROFILE[adm1] TRIGGER=EType=start,Offset=-60,AType=jobpreempt,Action="cancel"

> mrsvctl -c -P adm1 -s 12:00:00_10/03 -d 8:00:00 -h ALL

This reservation reserves all nodes in the cluster for a period of eight hours. Five minutes before the reservation starts, the reservation is modified to remove access to new preemptible jobs. Four minutes before the reservation starts, preemptible jobs that overlap the reservation are checkpointed. One minute before the reservation, all remaining jobs that overlap the reservation are canceled.

Reservations can also be used to evacuate virtual machines from a nodelist. To do this, you can configure a reservation profile in the moab.cfg file that calls an internal trigger to enable the evacuate VM logic. For example:

RSVPROFILE[evacvms]     TRIGGER=EType=start,AType=internal,action=node:$(HOSTLIST):evacvms

> mrsvctl -c -P evacvms -s 12:00:00_10/03 -d 8:00:00 -h ALL

Please note that Moab gives its best effort in evacuating VMs; however, if other reservations and policies prevent Moab from locating an alternate location for the VMs to be migrated to, then no action will occur. Administrators can attach additional triggers to the reservation profile to add evacuation logic where needed.

If your organization uses Viewpoint 7.1 or later, there is an option when creating reservations in Viewpoint to evacuate VMs from reserved nodes. This functionality assumes the reservation profile in Moab is named "evacvms." For Cloud customers, the evacvms reservation profile already exists in your moab.cfg file configuration by default.

You can also manually create a reservation that evacuates VMs from a nodelist by using the EVACVMS reservation flag. For example:

> mrsvctl -c -F EVACVMS -s 12:00:00_10/03 -d 8:00:00 -h ALL