Charging and Allocation Management

6.0 Managing Fairness - Throttling Policies, Fairshare, and Allocation Management > Charging and Allocation Management

6.4 Charging and Allocation Management

Charging and Allocation Management Overview

Configuring the Allocation Manager Interface
Allocation Management Policies
Charge Metrics

Allocation Manager Types
- MAM (Moab Accounting Manager)
- Native Allocation Manager

6.4-A Charging and Allocation Management Overview

Either Moab HPC Suite 7.0 - Enterprise Edition or Moab Cloud Suite 7.0 are required for support of charging and allocation management capabilities.

An allocation manager is a software system that manages resource allocations. A resource allocation grants a job a right to use a particular amount of resources. While full details of each allocation manager may be found within its respective documentation, the following brief review highlights a few of the values of using such a system.

An allocation manager functions much like a bank in that it provides a form of currency that allows jobs to run on an HPC system. The owners of the resource (cluster/supercomputer) determine how they want the system to be used (often via an allocations committee) over a particular time frame, often a month, quarter, or year. To enforce their decisions, they distribute allocations to various projects via accounts. These allocations can be used for particular clusters or globally. They can also have time frames associated with them to establish an allocation cycle. All transaction information is typically stored in a database or directory server allowing extensive statistical and allocation tracking.

When using an allocation manager, each job must be associated with an account. To accomplish this with minimal user impact, the allocation manager could be set up to handle default accounts on a per-user basis. However, as is often the case, some users may be active on more than one project and thus have access to more than one account. In these situations, a mechanism such as a job command file keyword should be provided to allow a user to specify which account should be associated with the job.

The amount of each job's allocation charge is directly associated with the amount of resources used (processors) by that job and the amount of time it was used. Optionally, the allocation manager can also be configured to charge accounts varying amounts based on the QoS desired by the job, the type of compute resources used, and the time when the resources were used.

The allocation manager interface provides near real-time allocation management, giving a great deal of flexibility and control over how available compute resources are used over the medium- and long-term, and works hand-in-hand with other job management features such as Moab's usage limit policies and fairshare mechanism.

The ENFORCEACCOUNTACCESS parameter controls whether the scheduler enforces account constraints.

Supported allocation managers include MAM (Moab Accounting Manager) and Native.

Moab Accounting Manager is a commercial charge-back accounting system that can be used in cloud or HPC environments. It is based on the Gold Allocation Manager.

The MAM allocation manager type (AMCFG[mam] TYPE=MAM) uses the direct SSS wire protocol originally used by Gold.

The Native allocation manager type (AMCFG[mam] TYPE=Native) invokes scripts to create a customization layer between Moab Workload Manager and Moab Accounting Manager. The interface is used for cloud contexts and can be used in some hpc contexts where greater accounting customization is required. The native interface can also be customized to interact with third-party allocation manager system. Moab makes calls to the scripts that handle the interaction with the external system.

The MAM interface is faster while the native interface allows a higher level of customization.

Configuring the Allocation Manager Interface

Configure Moab to use the Moab Accounting Manager by running ./configure with the applicable options when installing Moab:

--with-am[=TYPE] - enables accounting management with the specified accounting manager type (mam or native) [mam]

--with-am-dir=DIR - uses the specified prefix directory for the accounting manager if installed in a non-default location
--with-cloud[=EDITION] - specifies cloud edition [xcat]

The --with-am option specifies the accounting manager type that you want to use as either mam, which is the default, or native. Specifying this option will add the necessary entries into the moab.cfg file and cause the install process to copy configuration files, scripts, and libraries into place.

Use --with-am-dir to specify the prefix directory for Moab Accounting Manager if the native type is being used and it has been installed in a non-default location.

The --with-cloud option specifies that you are installing Moab in a cloud context (HPC is the default context) and makes some adjustments to the configuration files and interface scripts necessary for contextually appropriate charging behaviors. This option includes automatically setting the --with-am=native option, since the mam accounting manager type is not supported in the cloud context. If you are specifying a cloud context but do not wish to use the accounting manager, use the --without-am configure option.

The following is an example of configuring Moab charging for HPC:

./configure --with-am

The following is an example of configuring Moab charging for cloud:

./configure --with-cloud

If using a native allocation manager type, it will also be necessary to run make perldeps as root when installing so that the prerequisite bundled Perl modules are installed.

make perldeps

If you want to configure one of the other types of allocation manager, follow the instructions in the appropriate section.

Moab's allocation manager interface(s) are defined using the AMCFG parameter. This parameter allows specification of key aspects of the interface as shown in the following table:

VALIDATEJOBSUBMISSION

WIREPROTOCOL

CHARGEPOLICY
Format	One of DEBITALLWC, DEBITALLCPU, DEBITALLPE, DEBITALLBLOCKED, DEBITSUCCESSFULWC, DEBITSUCCESSFULCPU, DEBITSUCCESSFULPE, or DEBITSUCCESSFULBLOCKED
Default	DEBITSUCCESSFULWC
Description	Specifies how consumed resources should be charged against the consumer's credentials. See Charge Policy Overview for details. When you use the Native accounting manager interface, Moab ignores the configured CHARGEPOLICY and instead uses DEBITALLWC to calculate charges. The DEBITSUCCESSFUL* policies require TORQUE to work. Additionally, the job scripts must return a negative number as the exit code on failure in order to be ignored for charging.
Example	AMCFG[mam] CHARGEPOLICY=DEBITALLCPU Allocation charges are based on actual CPU usage only, not dedicated CPU resources. If the LOCALCOST flag (`AMCFG[] FLAGS=LOCALCOST`) is set, Moab uses the information gathered with CHARGEPOLICY to calculate charges. If LOCALCOST is not set, Moab sends this information to the allocation manager to calculate charges.

CREATEFAILUREACTION
Format	<AMFailureAction>[,<FundsFailureAction>] where the action is one of CANCEL, DEFER, HOLD, or IGNORE
Default	IGNORE,IGNORE
Description	Before creating a job that should be tracked or charged within the accounting manager, Moab contacts the accounting manager for authorization. If the job creation is rejected due to lack of funds, Moab applies the FundsFailureAction to the job. For any other rejection reason including a connection problem, Moab applies the AMFailureAction to the job. If you do not specify a FundsFailureAction, Moab will apply the AMFailureAction for an insufficient funds failure. If the action is set to CANCEL, Moab cancels the job; DEFER, defers the job; HOLD, puts the job on hold; and IGNORE, ignores the failure and continues to start the job. This parameter applies to both the mam and native accounting manager types.
Example	AMCFG[mam] CREATEFAILUREACTION=HOLD A job will be placed on hold when submitted if there are insufficient funds for it to start.

CREATEURL
Format	exec://<fullPathToCreateScript>
Default	---
Description	If you use the native accounting manager interface, Moab runs this script to create a chargeable job or reservation in order to determine whether it should be created. For jobs, the CREATEFAILUREACTION parameter specifies the action that should be taken if the authorization fails (such as for insufficient funds).
Example	AMCFG[mam] CREATEURL=exec://$TOOLSDIR/mam/usage.create.mam.pl Moab calls the usage.create.mam.pl script for authorization before starting a job or reservation.

DELETEURL
Format	exec://<fullPathToDeleteScript>
Default	---
Description	If you use the native accounting manager interface, Moab runs this script as needed to clean up after an interrupted job or reservation life-cycle. The default behavior is to remove outstanding liens.
Example	AMCFG[mam] DELETEURL=exec://$TOOLSDIR/mam/usage.delete.mam.pl Moab calls the usage.delete.mam.pl script to clean up after an interrupted job or reservation.

ENDURL
Format	exec://<fullPathToEndScript>
Default	---
Description	If you use the native accounting manager interface, Moab runs this script after the end of a chargeable job or reservation in order to make a final charge or update the accounting record. The default behavior is to make a prorated charge for the job or reservation.
Example	AMCFG[mam] ENDURL=exec://$TOOLSDIR/mam/usage.end.mam.pl Calls the usage.end.mam.pl script to make the final charge for a job or reservation.

FALLBACKACCOUNT
Format	STRING
Default	---
Description	If specified, Moab verifies adequate allocations for all new jobs. If adequate allocations are not available in the job's primary account, Moab changes the job's credentials to use the fallback account. If not specified, Moab places a hold on jobs that do not have adequate allocations in their primary account.
Example	AMCFG[mam] FALLBACKACCOUNT=freecycle Moab assigns the account freecycle to jobs that do not have adequate allocations in their primary account.

FALLBACKQOS
Format	STRING
Default	---
Description	If specified, Moab verifies adequate allocations for all new jobs. If adequate allocations are not available in the job's primary QoS, Moab changes the job's credentials to use the fallback QoS. If not specified, Moab places a hold on jobs that do not have adequate allocations in their primary QoS.
Example	AMCFG[mam] FALLBACKQOS=freecycle Moab assigns the QoS freecycle to jobs that do not have adequate allocations in their primary QoS.

FLAGS
Format	<STRING>
Default	---
Description	AMCFG flags are used to enable special services.
Example	AMCFG[mam] FLAGS=LOCALCOST Moab calculates the charge for the job locally and sends that as a charge to the allocation manager, which then charges that amount for the job.

FLUSHINTERVAL
Format	HOUR, DAY, WEEK, MONTH (or NONE)
Default	NONE
Description	Indicates the amount of time between allocation manager debits for long running reservation and job based charges. Only accepts intervals such as HOUR, DAY, WEEK, MONTH, or NONE.
Example	AMCFG[mam] FLUSHINTERVAL=DAY Moab updates its charges every 24 hours for long running jobs and reservations.

NODECHARGEPOLICY
Format	One of AVG, MAX, or MIN
Default	MIN
Description	When charging for resource usage, the allocation manager will charge by node allocation according to the specified policy. For AVG, MAX, and MIN, the allocation manager will charge by the average, maximum, and minimum node charge rate of all allocated nodes. (Also see CHARGEPOLICY.) This feature can only be used in conjunction with the AMCFG[] LOCALCOST flag which limits its use to cases where Moab calculates the full charge to be used by Moab Accounting Manager.
Example	NODECFG[node01] CHARGERATE=1.5 NODECFG[node02] CHARGERATE=1.75 AMCFG[mam] NODECHARGEPOLICY=MAX Allocation management charges jobs by the maximum allocated node's charge rate.

PAUSEURL
Format	EXEC://<fullPathToPauseScript>
Default	---
Description	If you use the native accounting manager interface, Moab runs this script after preempting a job that might be resumed later. The default behavior is to make an incremental charge but not create a fresh lien.
Example	AMCFG[mam] PAUSEURL=exec://$TOOLSDIR/mam/usage.pause.mam.pl Moab calls the usage.pause.mam.pl script after pausing a job.

QUOTEURL
Format	exec://<fullPathToQuoteScript>
Default	---
Description	If you use the native accounting manager interface, Moab runs the specified script to determine the amount the accounting manager will charge for a job or reservation.
Example	AMCFG[mam] QUOTEURL=exec://$TOOLSDIR/mam/usage.quote.mam.pl Moab calls the usage.quote.mam.pl script when it needs to determine the cost for a job or reservation.

SERVER
Format	URL
Default	N/A
Description	Specifies the type and location of the allocation manager service. If the keyword ANY is specified instead of a URL, Moab will use the local service directory to locate the allocation manager.
Example	AMCFG[mam] SERVER=mam://tiny.supercluster.org:4368

SOCKETPROTOCOL
Format:	One of SUTCP, SSS-HALF, HTTP, or SSS-CHALLENGE
Default:	SSS-HALF
Description:	Specifies the socket protocol to be used for scheduler-allocation manager communication.
Example:	AMCFG[mam] SOCKETPROTOCOL=SSS-CHALLENGE

STARTFAILUREACTION
Format:	<AMFailureAction>[,<FundsFailureAction>] where the action is one of CANCEL, DEFER, HOLD, IGNORE, or RETRY
Default:	IGNORE,IGNORE
Description:	Moab applies <FundsFailureAction> to any job that should be tracked or charged within the account manager if it is rejected due to insufficient funds for the given user and account. Moab applies <AMFailureAction> to a job if the account manager rejects it for any other reason. If you do not specify a <FundsFailureAction>, Moab will apply the <AMFailureAction> for an insufficient funds failure. If the action is set to CANCEL, Moab cancels the job; DEFER, defers the job; HOLD, puts the job on hold; IGNORE, ignores the failure and continues to start the job; and RETRY, does not start the job on this attempt but attempts to start the job at the next opportunity. STARTFAILUREACTION applies to both the mam and native accounting manager type.
Example:	AMCFG[mam] STARTFAILUREACTION=HOLD A job will be placed on hold if there are insufficient funds when it is time for it to start.

STARTURL
Format:	exec://<fullPathToStartScript>
Default:	---
Description:	If you use the native accounting manager interface, Moab runs this script on a chargeable job or reservation to determine whether it should start. For jobs, the STARTFAILUREACTION attribute specifies the action that Moab should take if the authorization fails (such as for insufficient funds).
Example:	AMCFG[mam] STARTURL=exec://$TOOLSDIR/mam/usage.start.mam.pl Moab calls the usage.start.mam.pl script for authorization before starting a job or reservation.

TIMEOUT
Format:	[[[DD:]HH:]MM:]SS
Default:	15
Description:	Specifies the maximum delay allowed for scheduler-allocation manager communications.
Example:	AMCFG[mam] TIMEOUT=30

UPDATEURL
Format	exec://<fullPathToUpdateScript>
Description	If you use the native accounting manager interface and have FLUSHINTERVAL set, Moab runs this script every flush interval for each chargeable job or reservation to determine if it should continue. The default behavior is to charge for the previous charge interval and create a lien for the next.
Example	AMCFG[mam] UPDATEURL=exec://$TOOLSDIR/mam/usage.update.mam.pl Moab calls the usage.update.mam.pl script for authorization to continue a job or reservation.

VALIDATEJOBSUBMISSION
Format:	<BOOLEAN>
Default:	FALSE
Description:	If set to TRUE, when a new job is submitted, Moab will execute the CREATEURL script (for TYPE=Native) or seek a job quote from Moab Accounting Manager (TYPE=MAM) before allowing the job to be submitted. Otherwise, the fund validation step is just utilized by reservations and fallback account checks. If the call fails (for example, if the user's account does not have sufficient funds or specifies an invalid account.), Moab applies the CREATEFAILUREACTION.
Example:	AMCFG[mam] VALIDATEJOBSUBMISSION=TRUE Moab calls the usage.update.mam.pl script for authorization to continue a job or reservation.

WIREPROTOCOL
Format:	One of AVP, HTML, SSS2, or XML
Default:	XML
Description:	Specifies the wire protocol to be used for scheduler-allocation manager communication.
Example:	AMCFG[mam] WIREPROTOCOL=SSS2

The first step to configure the allocation manager involves specifying where the allocation service can be found. This is accomplished by setting the AMCFG parameter's SERVER attribute to the appropriate URL.

In the case of the Moab Accounting Manager, after the interface URL is specified, secure communications between scheduler and allocation manager must be enabled. As with other interfaces, this is configured using the CLIENTCFG parameter within the moab-private.cfg file as described in the Security Appendix. The KEY and AUTHTYPE attributes should be set to values defined during initial allocation manager build and configuration, as in the following example:

CLIENTCFG[AM:mam] KEY=secret_key AUTHTYPE=HMAC64

AMCFG Flags

AMCFG flags can be used to enable special services and to disable default services. These services are enabled/disabled by setting the AMCFG FLAGS attribute.

Flag Name	Description
ACCOUNTFAILASFUNDS	When this flag is set, logic failures within the allocation manager are treated as fund failures and are canceled. When ACCOUNTFAILASFUNDS is not set, Allocation Manager failures are treated as a server failure and the result is a job which requests an account to which the user does not have access.
LOCALCOST	Moab calculates the charge for the job locally and sends that as a charge to the allocation manager, which then charges the amount for the job.
STRICTQUOTE	Sends an estimated process count to the allocation manager when an initial quote is requested for a newly-submitted job.

Allocation Management Policies

In most cases, the scheduler interfaces with a peer service. With all peer services based allocation managers, the scheduler checks with the allocation manager before starting any job. For allocation tracking to work, however, each job must specify an account to charge or the allocation manager must be set up to handle default accounts on a per user basis.

Under this configuration, when Moab starts a job, it contacts the allocation manager and requests an allocation reservation (or lien) be placed on the associated account. This allocation reservation is equivalent to the total amount of allocation that could be consumed by the job (based on the job's wallclock limit) and is used to prevent the possibility of allocation over subscription. Moab then starts the job. When the job completes, Moab debits the amount of allocation actually consumed by the job from the job's account and then releases the allocation reservation, or lien.

These steps should be transparent to users. Only when an account has insufficient allocations to run a requested job will the presence of the allocation manager be noticed. If preferred, an account may be specified for use when a job's primary account is out of allocations. This account, specified using the AMCFG parameter's FALLBACKACCOUNT attribute, is often associated with a low QoS privilege and priority, and is often configured to run only when no other jobs are present.

The scheduler can also be configured to charge for reservations. One of the big hesitations with dedicating resources to a particular group is that if the resources are not used by that group, they go idle and are wasted. By configuring a reservation to be chargeable, sites can charge every idle cycle of the reservation to a particular project. When the reservation is in use, the consumed resources will be associated with the account of the job using the resources. When the resources are idle, the resources will be charged to the reservation's charge account. In the case of standing reservations, this account is specified using the parameter SRCFG, attribute CHARGEACCOUNT. In the case of administrative reservations, this account is specified via a command line flag to the setres command.

Moab only interfaces to the allocation manager when running in NORMAL mode.

Charge Metrics

The allocation manager interface allows a site to charge accounts in a number of different ways. Some sites may wish to charge for all jobs regardless of whether the job completed successfully. Sites may also want to charge based on differing usage metrics, such as dedicated wallclock time or processors actually used. Moab supports the following charge policies specified via the CHARGEPOLICY attribute:

DEBITALLWC - Charges all jobs regardless of job completion state using processor weighted wallclock time dedicated as the usage metric.
DEBITALLCPU - Charges all jobs based on processors used by job.
DEBITALLPE - Charges all jobs based on processor-equivalents dedicated to job.
DEBITALLBLOCKED - Charges all jobs based on processors dedicated and blocked according to node access policy or QoS node exclusivity.
DEBITSUCCESSFULWC - Charges only jobs that successfully complete using processor weighted wallclock time dedicated as the usage metric. This is the default metric.
DEBITSUCCESSFULCPU - Charges only jobs that successfully complete using CPU time as the usage metric.
DEBITSUCCESSFULPE - Charges only jobs that successfully complete using PE weighted wallclock time dedicated as the usage metric.
DEBITSUCCESSFULBLOCKED - Charges only jobs that successfully complete based on processors dedicated and blocked according to node access policy or QoS node exclusivity.

When you use the Native accounting manager interface, Moab ignores the configured CHARGEPOLICY and instead uses DEBITALLWC to calculate charges.

On systems where job wallclock limits are specified, jobs that exceed their wallclock limits and are subsequently canceled by the scheduler or resource manager are considered to have successfully completed as far as charging is concerned, even though the resource manager may report these jobs as having been removed or canceled.

If machine-specific allocations are created within the allocation manager, the allocation manager machine name should be synchronized with the Moab resource manager name as specified with the RMCFG parameter, such as the name orion in RMCFG[orion] TYPE=PBS.

To control how jobs are charged when heterogeneous resources are allocated and per resource charges may vary within the job, use the NODECHARGEPOLICY attribute.

When calculating the cost of the job, Moab will use the most restrictive node access policy. See NODEACCESSPOLICY for more information.

6.4-B Allocation Manager Types

Moab supports two allocation manager types: MAM (Moab Accounting Manager) and Native.

MAM (Moab Accounting Manager)

Moab Accounting Manager is an accounting management system that provides usage tracking, charge accounting, and allocation enforcement for resource or service usage in cloud and technical computing environments. It acts like a bank in which credits are deposited into accounts with constraints designating which entities may access the account. As resources or services are utilized, accounts are charged and usage recorded. MAM supports familiar operations such as deposits, withdrawals, transfers, and refunds and provides balance and usage feedback to users, managers, and system administrators.

To configure Moab to use the MAM interface for allocation management, use the configure options as described in the Configuring the Allocation Manager Interface section.

Example 6-7:

./configure -with-am=mam ...

Consequently, make install will add the essential configuration and connection entries into the moab.cfg and moab-private.cfg files.

The following are typical entries in the Moab configuration files for using the MAM interface in the HPC context:

moab.cfg:

AMCFG[mam] TYPE=MAM HOST=localhost PORT=7112

AMCFG[mam] STARTFAILUREACTION=HOLD CHARGEPOLICY=DEBITALLWC

moab-private.cfg:

CLIENTCFG[AM:mam] KEY=secret_key AUTHTYPE=HMAC64

Synchronize the secret key with Moab Accounting Manager by copying the value of the token.value parameter from the MAM_PREFIX/etc/site.conf file which is randomly generated during the Moab Accounting Manager install process.

Moab Accounting Manager should be installed, started, and initialized. See the Getting Started chapter of the Moab Accounting Manager Administrator Guide for examples of how to initialize MAM for your initial mode of operation.

Native

When utilizing the Native allocation manager interface, Moab calls scripts to perform its accounting and allocation functions instead of communicating directly with the allocation manager, as is the case with MAM. The default scripts are designed to interact with Moab Accounting Manager and are typically used in the cloud context. You can customize these scripts to the needs of your site, either to provide additional flexibility in accounting and allocation management with Moab Accounting Manager or to interact with a third-party accounting or allocation management system.

Moab will invoke the Native Allocation Manager Interface (NAMI) scripts (see the AMCFG parameter documentation for QUOTEURL, CREATEURL, STARTURL, UPDATEURL, PAUSEURL, ENDURL, DELETEURL) by passing the job or reservation information via XML to the standard input of the script. The script should return a return code (zero for success), data on standard out and messages on standard error. A failure in CREATEURL, STARTURL, or UPDATEURL should result in the application of the CREATEFAILUREACTION, STARTFAILUREACTION or UPDATEFAILUREACTION, respectively.

To configure Moab to use the Native interface for allocation management, use the configure options as described in the Configuring the Allocation Manager Interface section.

Example 6-8:

./configure --with-am=native ...

Consequently, make install will add the essential allocation manager entries into moab.cfg and install the accounting-related scripts ($PREFIX/tools/mam/usage.*.mam.pl) and configuration files ($MOABHOMEDIR/etc/nami.cfg) in the correct locations.

The following are typical entries in the Moab configuration files for using the Native interface for the HPC context:

moab.cfg:

AMCFG[mam] TYPE=NATIVE
AMCFG[mam] CREATEURL=exec://$TOOLSDIR/mam/usage.create.mam.pl
AMCFG[mam] STARTURL=exec://$TOOLSDIR/mam/usage.start.mam.pl
AMCFG[mam] UPDATEURL=exec://$TOOLSDIR/mam/usage.update.mam.pl
AMCFG[mam] PAUSEURL=exec://$TOOLSDIR/mam/usage.pause.mam.pl
AMCFG[mam] ENDURL=exec://$TOOLSDIR/mam/usage.end.mam.pl
AMCFG[mam] DELETEURL=exec://$TOOLSDIR/mam/usage.delete.mam.pl
AMCFG[mam] STARTFAILUREACTION=HOLD

nami.cfg:

Context                             hpc
LOG[config]                         path=/opt/mam/log/nami.log
LOG[config]                         loglevel=debug
LOG[config]                         includeGoldLog=false
LOG[config]                         permissions=666
LOG[config]                         maxSize=10000000
LOG[config]                         rolloverLimit=7)

To view your current URL and FAILUREACTION information, run mdiag -R -v (even more information may be available in mdiag -R -v --xml). The following shows sample output from running the mdiag command:

AM[mam]  Type: native  State: 'Active'
ValidateJobSubmission:     FALSE
FlushInterval:  1:00:00
ChargePolicy:   DEBITALLWC
Quote URL:
Create URL:     CREATEURL=exec://$TOOLSDIR/mam/usage.create.mam.pl
Start URL:      STARTURL=exec://$TOOLSDIR/mam/usage.start.mam.pl
Update URL:     UPDATEURL=exec://$TOOLSDIR/mam/usage.update.mam.pl
Pause URL:      PAUSEURL=exec://$TOOLSDIR/mam/usage.pause.mam.pl
End URL:        ENDURL=exec://$TOOLSDIR/mam/usage.end.mam.pl
Delete URL:     DELETEURL=exec://$TOOLSDIR/mam/bank.delete.mam.pl

Moab Accounting Manager should be installed, started, and initialized. The simplest procedure is to install it on the same server as Moab Workload Manager so that the Moab Accounting Manager can share libraries and configuration files with the Moab Workload Manager and Moab Accounting Manager scripts. See the Getting Started chapter of the Moab Accounting Manager User Guide for examples of how to initialize MAM for your initial mode of operation.