Moab Workload Manager
Home Up Previous Next

20.1 Configuring VPC Profiles

20.1.1 Configuring VPC Profiles

Use the VCPROFILE parameter to configure a VPC profile. At its simplest, a VPC profile is a single line that gives the profile a name. The profile can then be requested by name and the user can fill in any other details required. The DESCRIPTION parameter is helpful, but not required. Users will be able to see the description in the listing of VPCs.

Example: A simple VPC with a description

VCPROFILE[example1]  DESCRIPTION="A Dedicated Compute Node"

The VPC can then be requested on the command line or through Moab Access Portal. Moab Access Portal provides a customizable portal where users can create, modify and manage VPCs. For more information, see Creating VPCs.

ACL
Format: List of zero or more comma delimited <OTYPE>==<OID>[:<OID>] pairs where <OTYPE> is one of the following:
USER, GROUP, ACCT, CLASS, or QOS
Default: ---
Details: Specifies which credentials can view this profile and can create VPCs using this profile.
Example: 
VCPROFILE[basic] ACL=USER==steve:john,GROUP==staff

Users steve and john and any member of the group staff can use VPC resources.

Note Inputting USER==steve,USER==john will only allow john to use the virtual cluster (as ==john overwrites ==steve). For both to use resources, input USER==steve,USER+=john.

   
DESCRIPTION
Format: <STRING>
Default: ---
Details: Arbitrary human readable string describing VPC profile usage.

Note This string will be provided to end users via application and web portals to help them select the appropriate resources.)

Example: 
VCPROFILE[dataset3] DESCRIPTION="custom 1 TB data mining environment with level 3 security"

End-users will see this description when specifying which VPC package to select.

   
NODEHOURCHARGERATE
Format: <DOUBLE>
Default: ---
Details: Per node chargerate multiplier associated with using resources and services in the specified VPC. See the Allocation Management overview for more information.
Example: 
VCPROFILE[dataset1] NODEHOURCHARGERATE=1.5
VCPROFILE[dataset2] NODEHOURCHARGERATE=4.0
VCPROFILE[dataset3] NODEHOURCHARGERATE=6.0

Accounts associated with the creation of each data VPC will be charged for VPC utilization with the associated charge multiplier.

   
NODESETLIST
Format: <FEATURE>[,<FEATURE>]...
Default: ---
Details: Nodeset constraints to be applied to explicitly specified reqs in format.

Note The VC profile NODESETLIST attribute only supports blocking node feature based node sets with the ONEOF policy enforced.

Example: 
VCPROFILE[dataset3] NODESETLIST=fastio,myrinet

Resources will either be allocated entirely from the fastio or the myrinet node set.

   
NODESETUPCOST
Format: <DOUBLE>
Default: ---
Details: Per node cost in credits associated with provisioning resources and services to enable the associated VPC. See the Allocation Management overview for more information.
Example: 
VCPROFILE[dataset1] NODESETUPCOST=20.5
VCPROFILE[dataset2] NODESETUPCOST=20.5
VCPROFILE[dataset3] NODESETUPCOST=65.0

Accounts associated with the creation of a new VPC will be assessed a per-node charge for each node in a newly created VPC associated with the overhead of setting up the needed environment.

   
OPRSVPROFILE
Format: <RSVPROFILEID>
Default: ---
Details: The default reservation profile for resource requirements that are explicitly specified by the user.
Example: 
VCPROFILE[dataset3] OPRSVPROFILE=fastp

Resource requirements that are explicitly specified by the requestor and that do not have an explicit reservation profile will use the fastp reservation profile.

   
QUERYDATA
Format: <RESOURCE_QUERY_STRING> (See mshow command usage.)
Default: ---
Details: List of requirements (reqs) to be co-allocated along with explicitly specified requirements.
Example: 
VCPROFILE[pkgA] QUERYDATA=minprocs=1,nodefeature=ionodes,duration=00:30:00
VCPROFILE[pkgA] QUERYDATA=mintasks=1,gres=vlan:1,duration=01:00:00

One ionode will be allocated for the first 30 minutes of the VPC and one vlan generic resource will be allocated for the first hour.

Note If a QUERYDATA attribute has a nodefeature constraint of $common, the co-allocation query will import the nodefeature constraint of the first explicitly specified requirement.

   
REQENDPAD
Format: [[[DD:]HH:]MM:]SS
Default: ---
Details: Amount of additional time required after each explicitly specified requirement.
Example: 
VCPROFILE[premium] REQENDPAD=00:05:00

Five minutes of additional walltime will be appended to the end of each explicitly specified requirement.

   
REQDEFATTR
Format: <RESOURCE_LIST>
Default: ---
Details: A list of comma-delimited resource requirements that are merged with the resource request requirements in the corresponding mshow -a command.
Example: 
VCPROFILE[database]    REQDEFATTR=duration=1:00:00:00

Indicates a database profile with a default duration of one day. User can specify a shorter or longer duration when the VPC is requested.

   
REQSETATTR
Format: <RESOURCE_LIST>
Default: ---
Details: A list of comma-delimited resource requirements that are merged with the resource request requirements in the corresponding mshow -a command.
Example: 
VCPROFILE[Large]    REQSETATTR=mintasks=4,gres=tape:2

A request for four tasks and two generic resources named tape are appended to the user-specified resource request.

   
REQSTARTPAD
Format: [[[DD:]HH:]MM:]SS
Default: ---
Details: Amount of additional time required before each explicitly specified req.
Example: 
VCPROFILE[premium] REQSTARTPAD=00:15:00

Fifteen minutes of additional walltime will be prepended to the start of each explicitly specified requirement.

The reservation profiles of the VPC set with the QUERYDATA or OPRSVPROFILE attributes can be used to manage provisioning of VPC resources. This provisioning is generally enabled via the use of triggers although it may also be enabled through a resource manager interface.

20.1.2 Associating Resources with a VPC Profile

Any number of resources or other attributes can be associated with a VPC profile. Examples include the number of nodes required, the duration of the VPC, or required node features. Default attributes that can be overridden when the VPC is requested are specified with REQDEFATTR. Required attributes that can not be overridden are specified with REQSETATTR.

For resources that need to be allocated once per request (coallocation), see Coallocation of Additional Resources.

Example: Database server that will only run on specific nodes

VCPROFILE[database]     DESCRIPTION="Database Server with Fast IO"
VCPROFILE[database]     REQSETATTR=nodefeature=fastio
VCPROFILE[database]     REQDEFATTR=duration=1:00:00:00

The preceding example indicates a database profile with a default duration of one day. The user can specify a shorter or longer duration when the VPC is requested. The database profile will only run on nodes with the fastio feature, which cannot be overridden by the user.

20.1.3 Using Reservation Profiles with VPCs

When a VPC is created, a reservation is created on the cluster. A reservation profile can be specified for the reservation. Setup and teardown actions, as well as other attributes, can be configured as part of the reservation profile. Reservation profiles are specified using the OPRSVPROFILE parameter.

Setup and teardown actions are specified using triggers. Triggers allow actions ranging from simple notification to complex workflows. If the setup time for the VPC is significant, the REQSTARTPAD and REQENDPAD parameters can be used to compensate the user for lost time. For actions that take place when the VPC is cancelled, a special variable (ACTIVE) can be passed to the trigger script. ACTIVE is set to the string TRUE if the VPC is currently active, or UNDEFINED if it is not. For more information see trigger variables.

Reservation profiles also allow a variety of flags to be specified. For more information see 7.1.2.2 Using Reservation Profiles.

Example: A VPC that provisions an operating system and logs information

VCPROFILE[test]     DESCRIPTION="Test Environment"
VCPROFILE[test]     OPRSVPROFILE=test

RSVPROFILE[test]     FLAGS=SYSTEMJOB
RSVPROFILE[test]     TRIGGER=ATYPE=exec,Action="$HOME/logdate.pl TEST START $VPCHOSTLIST $OID $HOSTLIST $ACTIVE",EType=start
RSVPROFILE[test]     TRIGGER=ATYPE=exec,Action="$HOME/installos.pl $HOSTLIST rhel",EType=start
RSVPROFILE[test]     TRIGGER=ATYPE=exec,Action="$HOME/logdate.pl TEST END $VPCHOSTLIST $OID $HOSTLIST $ACTIVE",EType=end
RSVPROFILE[test]     TRIGGER=ATYPE=exec,Action="$HOME/logdate.pl TEST CANCEL $VPCHOSTLIST $OID $HOSTLIST $ACTIVE",EType=cancel

In the preceding example, a logging script (logdate.pl) is called when the VPC starts, ends, or is cancelled. An operating system is installed on all nodes in the VPC when it starts.

20.1.4 Coallocation of Additional Resources

Additional units of other resources can also be packaged with VPC on a per instance basis. For example, if a user requests a four-node cluster, a head node (in addition to the four requested nodes) can be allocated automatically. This is done using the QUERYDATA parameter.

Example: Database profile with a single server and multiple client nodes

VCPROFILE[database]  DESCRIPTION="Database client/server package"
VCPROFILE[database]  QUERYDATA=minnodes=1,label=server,rsvprofile=dbserver
VCPROFILE[database]  OPRSVPROFILE=dbclient

In the preceding example, a database profile is configured that will give the user the number of database clients that they request (the dbclient profile), plus a single database server.

Example: LAMP stack profile with multiple parts

VCPROFILE[lampstack]         DESCRIPTION="LAMP stack"

VCPROFILE[lampstack]         OPRSVPROFILE=lampstack
VCPROFILE[lampstack]         NODEHOURCHARGERATE=6.6 NODESETUPCOST=66.6
VCPROFILE[lampstack]         FLAGS=AUTOACCESS


VCPROFILE[lampstack]         QUERYDATA=minnodes=1,label=apache_lb,rsvprofile=apache_lb
VCPROFILE[lampstack]         QUERYDATA=minnodes=3,label=apache_pool,rsvprofile=apache_pool
VCPROFILE[lampstack]         QUERYDATA=minnodes=1,label=db_lb,rsvprofile=db_lb
VCPROFILE[lampstack]         QUERYDATA=minnodes=2,label=db_pool,rsvprofile=db_pool
VCPROFILE[lampstack]         QUERYDATA=minnodes=1,label=db_backup,rsvprofile=db_backup
VCPROFILE[lampstack]         QUERYDATA=minnodes=2,label=script_compute,rsvprofile=script_compute

RSVPROFILE[lampstack]        FLAGS=EXCLUDEMYGROUP,SYSTEMJOB

RSVPROFILE[lampstack]        TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/lampstack.py          --etype=$ETYPE --vpcid=$VPCID  --hostlist=$HOSTLIST",EType=start,sets=^lampstack,requires=apache_lb.db_lb
RSVPROFILE[lampstack]        TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/lampstack_shepherd.py --vpcid=$VPCID",EType=start,MultiFire=TRUE,RearmTime=05:00,sets=!badlamp,requires=lampstack
RSVPROFILE[lampstack]        TRIGGER=ATYPE=mail,Action="There was a problem with this vpc: $VPCID",EType=start,requires=badlamp,unsets=badlamp

RSVPROFILE[lampstack]        TRIGGER=ATYPE=exec,Action="mschedctl -d vpc:$VPCID",EType=start,requires=selfdestruct

RSVPROFILE[apache_lb]        TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/apache_lb_ctl.py      --etype=$ETYPE --vpcid=$VPCID --hostlist=$HOSTLIST --pool=$apache_pool",EType=start,sets=^apache_lb,requires=apache_pool.script_compute
RSVPROFILE[apache_pool]      TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/apache_pool_ctl.py    --etype=$ETYPE --vpcid=$VPCID --hostlist=$HOSTLIST",EType=start,sets=^apache_pool
RSVPROFILE[script_compute]   TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/script_compute.py     --etype=$ETYPE  --db_lb=$db_lb  --vpcid=$VPCID  --hostlist=$HOSTLIST",EType=start,requires=db_lb,sets=^script_compute

RSVPROFILE[db_lb]            TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/db_lb_ctl.py          --etype=$ETYPE --vpcid=$VPCID --hostlist=$HOSTLIST --pool=$db_pool --backup=$db_backup --active=$ACTIVE",EType=start,sets=^db_lb,requires=db_pool.db_backup

RSVPROFILE[db_backup]        TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/db_backup_ctl.py      --etype=$ETYPE --vpcid=$VPCID --hostlist=$HOSTLIST",EType=start,sets=^db_backup

RSVPROFILE[db_pool]          TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/db_pool_zero.py       --etype=$ETYPE --vpcid=$VPCID",EType=start,sets=db_pool_go
RSVPROFILE[db_pool]          TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/db_pool_ctl.py        --etype=$ETYPE --vpcid=$VPCID --hostlist=$HOSTLIST",EType=start,MultiFire=TRUE,sets=^db_pool.!db_pool_retry,unsets=db_pool_go.!db_pool_go,requires=db_pool_go
RSVPROFILE[db_pool]          TRIGGER=ATYPE=exec,Action="/opt/moab/tools/lamp/db_pool_retry.py      --etype=$ETYPE --vpcid=$VPCID",EType=start,MultiFire=TRUE,sets=^db_pool_go.!^selfdestruct,unsets=db_pool_retry.!db_pool_retry,requires=db_pool_retry

In the preceding example, a profile is configured that makes a request for a LAMP stack that is comprised of an Apache load balancer, a pool of three Apache worker nodes, two nodes used to offload script-based computing, a database load balancer, a pool of two database nodes, and a database backup node.

The example shows how to handle two failure conditions. First, a shepherd trigger—a trigger that check the condition of a running job—is assigned to watch over the health of the lampstack in 5-minute intervals. If this trigger fails (returns -1), Moab sends an email to the administrator indicating the VPCID of the culprit VPC.

The second form of failure is illustrated using the db_pool example. The db_pool_zero script runs at VPC startup, resets the counter associated with the given VPCID (in this case stored in an SQLite3 database), and sets a variable allowing the second db_pool trigger to fire. If this trigger is successful, normal workflow conditions exist, and the rest of the VPC LAMP stack is created. If it fails, it sets a flag that the db_pool_retry script responds to. The db_pool_retry script increments the counter associated with the current VPCID, and resets the trigger variables to a state that allows the db_pool_ctl trigger to refire. If the db_pool_retry script gets called more than a prescribed number of times, this trigger fails, setting a variable that destroys the VPC.

20.1.5 Access Control

Administrators can specify the credentials that are allowed to create each VPC profile using the ACL parameter. The ACL parameter uses the Moab ACL syntax.

Example: A VPC that can only be created by members of the test group

VCPROFILE[test]  DESCRIPTION="Testing Environment"
VCPROFILE[test]  OPRSVPROFILE=test
VCPROFILE[test]  ACL=GROUP:test

The preceding example specifies that only members of the test group may request the test profile.

When a VPC is requested, an ACL can be specified for the reservation associated with the VPC. This allows users to give others access to their VPCs. This also allows users to submit batch jobs into the VPC that they have created. This capability can be used to allow trusted users to create limited reservations on the system and submit batch jobs into them. It also allows automatic charging for these reservations. If the AUTOACCESS flag is specified on the VPC, the owner of that VPC is automatically added to the ACL of its reservation.

Example: A profile that allows the owner and administrators to submit batch jobs into it

VCPROFILE[private] DESCRIPTION="Private cluster access"
VCPROFILE[private] REQSETATTR=nodefeature=dedicated,acl=group=administrators
VCPROFILE[private] FLAGS=AUTOACCESS

In the preceding example, the specified profile creates a reservation on the cluster that only administrators and the owner of the VPC may access. Only nodes with the dedicated feature may be reserved in this way.

Example: A user creates a VPC and submits jobs into it

To submit a job, the user specifies the VPC's reservation on the command line using -l advres.

$ mshow -a -i -x -o --flags=tid,summary,future -p private -w minnodes=1,duration=10000 
Partition     Tasks  Nodes      Duration   StartOffset       StartDate 
---------     -----  -----  ------------  ------------  -------------- 
ALL               2      1       2:46:40      00:00:00  16:19:14_03/10  TID=13 
ALL               2      1       2:46:40       2:46:40  19:05:54_03/10  TID=14 
ALL               2      1       2:46:40       5:33:20  21:52:34_03/10  TID=15 
$ mschedctl -c vpc -a resources=13 

vpc.5 

$ mschedctl -l vpc:vpc.5 
 
VPC vpc.5 (active) --------- 
User:                  user1
Owner:                 user:user1 
Size:                  2 
Task Resources:        PROCS: [ALL] 
Available Time:        16:19:14_03/10 (-00:01:56) 
Available Duration:    2:46:40 
Provision Start Time:  16:19:22_03/10 (-00:01:48) 
Cleanup End Time:      19:05:54_03/10 (2:44:44) 
PurgeTime:             19:05:54_03/11 
Variables:             VPCHOSTLIST=node009;VPCID=vpc.5; 
Rsv Group:             PRIVATE.5 (PRIVATE.5) 
VPC Profile:           PRIVATE
Total Charge:          0.00 
 
$ echo sleep 30 | msub -l advres=PRIVATE.5,walltime=30,nodes=1 
 
moab.2

20.1.6 Charging for VPCs

VPC charging is configured with the NODECHARGERATE and NODESETUPCOST parameters.

Example: Charging

VCPROFILE[dbclient]  NODEHOURCHARGERATE=25.0 NODESETUPCOST=50.0

See Also

Home Up Previous Next