(Click to open topic with navigation)

You are here: 5 Managing Fairness - Throttling Policies, Fairshare, and Allocation Management > Accounting, Charging, and Allocation Management

5.5 Accounting, Charging, and Allocation Management

In this topic:

For a complete list of and additional information on the AMCFG parameters and flags, see 5.6 AMCFG Parameters and Flags.

5.5.1 Accounting Manager Overview

An accounting manager is a software system that enables tracking and charging for job resource usage. Moab Accounting Manager is a commercial charge-back accounting system that has built-in integration with Moab Workload Manager. Moab Accounting Manager can be used in a variety of accounting modes such as for usage tracking, notional charging or allocation enforcement.

When used for usage tracking only, the accounting manager simply records workload usage details. When configured additionally to perform charging, resource charge rates are used to impute a charge for each job. When configured to enforce resource allocation limits, jobs are charged against allocations and new jobs may be blocked from running if their account runs out of funds. See Accounting Mode and see Select an Appropriate Accounting Mode in the Moab Accounting Manager Administrator Guide for more details on supported accounting modes.

In a typical allocation enforcement use case, credits are allocated to accounts for designated time periods; establishing limits on the use of compute resources. The base currency credits can be defined in terms of system resource units (e.g. Processor-Seconds) or a real currency (e.g. U.S. dollars). Charge rates are established for the use of resources. Accounts are created and users are given access to the appropriate accounts. Deposits are made into funds associated with the account’s creating allocations. An allocation cycle can be established whereby funds are reset on a regular periodic basis (such as yearly, quarterly, or monthly) and where allocations are renewed for accepted accounts. Before a job is started, Moab Workload Manager will verify that the user has sufficient credits to run the job by attempting to place a hold against their funds (referred to as a lien). When a job completes, the user's funds will be debited via a charge, usage information will be recorded for the job, and the lien will be removed.

5.5.2 Accounting Mode

The accounting mode (specified via the AMCFG[] MODE parameter) modifies the way in which accounting-relevant job and reservation stages (e.g. create, start, end, etc.) are processed. See 5.5.6 Accounting Stages for more information on the behaviors of the different values of the accounting mode.

The following table describes the valid values for the accounting mode.

Value Description
strict‑allocation Use this mode if you wish to strictly enforce allocation limits. Under this mode, holds (called liens) will be placed against allocations in order to prevent multiple jobs from starting up on the same funds. Jobs and reservations may be prevented from running if the end-users do not have sufficient funds. This is the default.
fast‑allocation

Use this mode if you wish to debit allocations, but need higher throughput by eliminating the lien and quote operations of strict-allocation mode. Under this mode, jobs and reservations check a cached account balance, and may be prevented from running after the balance has become zero or negative.

If you are using fast-allocation, funds are assumed to have account-based constraints only. Moab will reject funds having no constraints or having non-account constraints. It is highly recommended that you enable ENFORCEACCOUNTACCESS to TRUE and AMCFG[] CREATECRED=TRUE with an appropriate refresh period (via AMCFG[] REFRESHPERIOD) so that Moab can prevent jobs from running under accounts that the user does not belong to (this is enforced via liens in the strict-allocation accounting mode). Also, the configured refresh period will apply to both credential updates and account balance updates. See Moab Parameters for more information on the ENFORCEACCOUNTACCESS parameter.
notional‑charging Use this mode if you wish to calculate and record charges for workload usage, but not keep track of fund balances or allocation limits.
usage‑tracking Use this mode if you wish to record workload usage details, but not to calculate a charge nor keep track of fund balances or allocation limits.

5.5.3 Accounting Manager Interface Types

Moab Workload Manager supports two accounting manager interface types: MAM and Native.

5.5.3.A MAM

The MAM accounting manager interface type enables direct communication between Moab Workload Manager and Moab Accounting Manager. This often results in the fastest accounting performance. Use this interface type if you do not need to customize the interaction with the accounting manager.

To configure Moab to use the MAM accounting manager interface, run configure using the --with-am option.

Example 5-8:  

./configure -with-am=mam ...

Consequently, make install will add the essential configuration and connection entries into the moab.cfg and moab-private.cfg files.

The following are typical entries in the Moab configuration files for using the MAM interface:

Synchronize the secret key with Moab Accounting Manager by copying the value of the token.value parameter from the MAM_PREFIX/etc/mam-site.conf file which is randomly generated during the Moab Accounting Manager install process.

When using the MAM accounting manager interface, by default Moab will communicate directly with Moab Accounting Manager via the SSS wire protocol. However, it is possible to enable a hybrid model and override individual accounting actions by specifying the exec protocol and the path of a custom script to the appropriate AMCFG[] *URL parameters.

Moab Accounting Manager should be installed, started, and initialized. See Initial Setup in the Moab Accounting Manager Administrator Guide for examples of how to initialize MAM for your initial mode of operation.

5.5.3.B Native

The Native accounting manager interface type provides a customization layer between Moab Workload Manager and Moab Accounting Manager. This interface can be used where greater accounting customization is required. The native interface can also be customized to interact with third-party accounting manager systems. Moab passes job accounting details to scripts that handle the interaction with the external system.

To configure Moab to use the MAM accounting manager interface, run configure using the --with-am=native option.

Additionally, you may need to use the --with-am-dir configure option to specify the prefix directory for Moab Accounting Manager if MAM has been installed in a non-default location.

Example 5-9:  

./configure --with-am=native ...

Consequently, make install will add the essential accounting manager entries into moab.cfg and install the accounting-related scripts ($PREFIX/tools/mam/usage.*.mam.pl) in the correct locations..

Moab will default to using a set of stock scripts for the accounting stages. To view the scripts that are currently in use, run mdiag -R -v (even more information may be available in mdiag -R -v --xml). The following shows sample output from running the mdiag -R -v command.

AM[mam]  Type:  Native  State: 'Active'
  Timeout:                  15
  Thread Pool Size:         0
  Accounting Mode:          strict-allocation
  Create URL:               exec:///opt/moab-accounting/tools/mam/usage.quote.mam.pl
  Start URL:                exec:///opt/moab-accounting/tools/mam/usage.reserve.mam.pl
  Pause URL:                exec:///opt/moab-accounting/tools/mam/usage.charge.mam.pl
  Resume URL:               exec:///opt/moab-accounting/tools/mam/usage.reserve.mam.pl
  Update URL:               exec:///opt/moab-accounting/tools/mam/usage.charge.mam.pl
  Continue URL:             exec:///opt/moab-accounting/tools/mam/usage.reserve.mam.pl
  End URL:                  exec:///opt/moab-accounting/tools/mam/usage.charge.mam.pl
  Delete URL:               exec:///opt/moab-accounting/tools/mam/lien.delete.mam.pl
  Query URL:                exec:///opt/moab-accounting/tools/mam/account.query.mam.pl
  Charge Policy:            DEBITSUCCESSFULWC

Moab will invoke the native accounting manager scripts by passing the job or reservation information via XML to the standard input of the script. You may override any of the default scripts with a custom script by specifying the appropriate AMCFG URL parameter in the moab server configuration file. See 5.6 AMCFG Parameters and Flags for CREATEURL, STARTURL, PAUSEURL, RESUMEURL, UPDATEURL, CONTINUEURL, ENDURL, DELETEURL, and QUERYURL values for more information.

The XML sent to the scripts is in the form of an SSS Request that is identical to the Request sent to MAM when you use the MAM Accounting Manager Interface type. For example, the XML sent to the usage.charge.mam.pl script in a final charge consists of an encapsulating Request element with an action attribute that has a value of "Charge"; an object element with a value of "UsageRecord"; one or more optional Option elements; and a Data element. The Data element has a single UsageRecord element with property elements describing the job or reservation properties. For example:

<Request action="Charge"><Object>UsageRecord</Object><Option name="Duration">1234</Option><Data><UsageRecord><Type>Job</Type><Instance>Moab.165</Instance><User>amy</User><Group>staff</Group><Account>chemistry</Account><Class>batch</Class><QualityOfService>high</QualityOfService><Machine>colony</Machine><Nodes>1</Nodes><NodeType>Fast</NodeType><NodeCharge>2.000000</NodeCharge><Partition>Torque</Partition><Processors consumptionRate="0.50">2</Processors><Memory>2048</Memory><Matlab>2</Matlab><StartTime>1398805354</StartTime><EndTime>1398805357</EndTime><CompletionCode>0</CompletionCode><OpSys>CentOS 6</Opsys><Temp>87.00</Temp></UsageRecord></Data></Request> *

In the sample XML above, Matlab is an example of a generic resource, Opsys is an example of a job variable, and Temp is an example of a generic metric.

A reservation charge, or quote or lien, is very similar. For example:

<Request action="Charge"><Object>UsageRecord</Object><Option name="Duration">7200</Option><Data><UsageRecord><Type>Reservation</Type><Instance>reservation.7</Instance><User>amy</User><Machine>colony</Machine><Nodes>1</Nodes><ProcessorsconsumptionRate="0.76">12</Processors><Duration>7200</Duration><StartTime>1398797430</StartTime><EndTime>1398804630</EndTime></UsageRecord></Data></Request>

The majority of the scripts use this same basic XML format; for instance, usage.quote.mam.pl, usage.reserve.mam.pl, and usage.charge.mam.pl.

The XML sent to the lien.delete.mam.pl script to clean up after a failure consists of an encapsulating Request element with an action attribute that has a value of "Delete"; an object element with the value of "Lien"; and a condition (Where) element indicating the lien instance to delete. For example:

<Request action="Delete"><Object>Lien</Object><Where name="Instance">Moab.127</Where></Request>

The script should return a return code (zero for success), data on standard out and messages on standard error. A failure in CREATEURL, STARTURL, RESUMEURL, or CONTINUEURL should result in the application of the CREATEFAILUREACTION, STARTFAILUREACTION, RESUMEFAILUREACTION, or CONTINUEFAILUREACTION respectively.

Moab Accounting Manager should be installed, started, and initialized. The simplest procedure is to install it on the same server as Moab Workload Manager so that the Moab Accounting Manager can share libraries and configuration files with the Moab Workload Manager and Moab Accounting Manager scripts. See Initial Setup in the Moab Accounting Manager Administrator Guide for examples of how to initialize MAM for your initial mode of operation.

5.5.4 Accounting Properties Reported to Moab Accounting Manager

When you set the accounting manager TYPE to MAM, Moab can send the following information to Moab Accounting Manager via charging actions:

5.5.4.A For Jobs

Property name in MAM Usage Record Description of property value recorded in MAM Usage Record
Account Account name
Charge If the AMCFG LOCALCOST flag is set, Moab will calculate and pass the Charge amount to MAM. If it is not, MAM will calculate the charge based on the transmitted job properties.
Class Class/queue name
CompletionCode* Exit code
CPUTime CPU time
Duration Moab sends the wallclock time for the job charge(s) in seconds. This is aggregated in MAM as Duration.
EndTime Job end time
(Generic Metrics)* The property name is the name of the generic metric. The property value is the average value of the generic metric across the nodes of the job and time.
(Generic Resources)*

The property name is the name of the generic resource. The property value is the number of generic resources consumed by the job.

 

NUMA-specific generic resources include sockets, numanodes, cores, and threads and represent the NUMA resources dedicated by the job.
Group Group name
Instance Job ID
Machine Cluster (RM) name
Memory Dedicated or utilized memory in megabytes
NodeCharge* Aggregate node charge rate. See NODECHARGEPOLICY and CHARGERATE for more information.
Nodes Node count
NodeType* Node type. See NODETYPE for more information.
Partition Partition name
Processors Processor count; this property may also have a consumptionRate attribute (scale multiplier) indicating how much the processors value should be scaled by when charging for them.
QualityOfService QoS name
Stage Accounting stage
StartTime Job start time
SubmitTime Job submission time
Type Set to "Job"
User User name
(Variables)* The property name is the name of the job variable. The property value is the value of the job variable.

* For this property to be recorded in the MAM Usage Record, you must define a custom usage record attribute in MAM for it. See Customizing the Usage Record Object in the Moab Accounting Manager Administrator Guide for more information.

5.5.4.B For Reservations

Property name in MAM Usage Record Description of property value recorded in MAM Usage Record
Account Charge account
Duration Moab sends the wallclock time for the reservation in seconds. This is aggregated in MAM as Duration.
EndTime Reservation end time
Instance Reservation ID
Machine Cluster (RM) name
Nodes Node count allocated to the reservation
Partition Partition name
Processors Processor count allocated to the reservation. This property may also have a consumptionRate attribute which is the ratio of idle processor seconds to total processor seconds. For instance, for a reservation in which a total processor seconds was utilized 25% by running jobs, the consumption rate would be transmitted as 0.75, meaning that 75% of the total reservation processor seconds are to be charged for being idle.
Stage Accounting stage
StartTime Reservation start time
Type Set to "Reservation"
User Charge user or reservation owner

5.5.5 Accounting Policies

When using an accounting mode of strict-allocation, before Moab starts a job, it contacts the accounting manager and requests an allocation reservation (or lien) be placed on the associated account. The lien amount is equivalent to the total amount of allocation that could be consumed by the job (based on the job's wallclock limit) and is used to prevent the possibility of allocation over subscription. Moab then starts the job. When the job completes, Moab debits the allocation by the amount actually consumed by the job and then releases the lien.

These steps should be transparent to users. Only when an account has insufficient allocations to run a requested job will the presence of the accounting manager be noticed. If desired, a fallback account may be specified for use when a job's primary account is out of allocations. This account, specified using the AMCFG parameter's FALLBACKACCOUNT attribute, is often associated with a low QoS privilege and priority, and is often configured to run only when no other jobs are present.

The scheduler can also be configured to charge for reservations. One of the hesitations with dedicating resources to a particular group is that if the resources are not used by that group, they go idle and are wasted. By configuring a reservation to be chargeable, sites can charge every idle cycle of the reservation to a particular account. When the reservation is in use, the consumed resources will be charged to the job using the resources. When the resources are idle, the resources will be charged to the reservation's charge account. In the case of standing reservations, this account is specified using the parameter SRCFG[X], attribute CHARGEACCOUNT. In the case of administrative reservations, this account is specified via the -S account flag to the mrsvctl -c command (see mrsvctl).

5.5.5.A Charge Metrics

The accounting manager interface allows a site to charge accounts in a number of different ways. Some sites may wish to charge for all jobs regardless of whether the job completed successfully. Sites may also want to charge based on differing usage metrics, such as dedicated wallclock time or processors actually used. Moab supports the following charge policies specified via the CHARGEPOLICY attribute.

DEBITALLBLOCKED or DEBITSUCCESSFULBLOCKED should only be used with policies that allow only a single job to dedicate a node such as with a Node Access Policy of SINGEJOB or SINGLETASK, or using a QOS with the DEDICATED flag. Using DEBITALLBLOCKED or DEBITSUCCESSFULBLOCKED with any policy allowing more than one job to dedicate a node (such as a Node Access Policy of SINGLEUSER, SINGLECLASS, SINGLEACCOUNT or UNIQUEUSER) is not supported.

On systems where job wallclock limits are specified, jobs that exceed their wallclock limits and are subsequently canceled by the scheduler or resource manager are considered to have successfully completed as far as charging is concerned, even though the resource manager may report these jobs as having been removed or canceled.

If machine-specific allocations are created within the accounting manager, the accounting manager machine name should be synchronized with the Moab resource manager name as specified with the RMCFG parameter, such as the name orion in RMCFG[orion] TYPE=PBS.

To control how jobs are charged when heterogeneous resources are allocated and per resource charges may vary within the job, use the NODECHARGEPOLICY attribute.

When calculating the cost of the job, Moab will use the most restrictive node access policy. See NODEACCESSPOLICY for more information.

5.5.6 Accounting Stages

The accounting manager performs various actions throughout different stages of a job or reservation lifetime. For a stock configuration (meaning you have not overridden the accounting actions with custom scripts), the following describes the stages and the respective actions that occur at these stages depending on the accounting mode:

5.5.7 Accounting Events

You can add accounting events to the event log by specifying one or more of the following with RECORDEVENTLIST. See 12.2.4.A Event Log Format for more information.

Event Description
AMCREATE Record accounting events trigger when an object is created; for example, when a balance check occurs at job submission.
AMDELETE Record accounting events triggered when an object's normal accounting lifecycle is interrupted; for example, when the lifecycle is interrupted to clean up reservations for a failed job start.
AMEND Record accounting events triggered when an object ends; for example, when a charge occurs at the end of a job.
AMPAUSE Record accounting events triggered when an object is paused; for example, when a partial charge occurs when a job is paused.
AMQUOTE Record accounting events triggered when an object requires a quote amount.
AMRESUME Record accounting events triggered when an object is resumed; for example, when a lien is made when a job is resumed.
AMSTART Record accounting events triggered when an object is started; for example, when a lien is made when a job starts.
AMUPDATE Record accounting events triggered when an object continues past a flush interval; for example, when a partial charge occurs and new lien is made for a job.

5.5.8 Blocking Versus Non-Blocking Accounting Actions

Moab uses a thread pool to perform non-blocking actions. Instead of blocking the scheduling thread, the request is added to a queue that is serviced by the accounting thread pool. Using the thread pool to perform non-blocking accounting actions can result in faster aggregate scheduling and better client response times, though individual actions can, in some cases, be shortly delayed. By default, Moab uses non-blocking calls for the final charge only. The default behavior for individual accounting actions (such as Create, Start, End) can be overridden via the associated parameter (CONTINUEISBLOCKING, CREATEISBLOCKING, DELETEISBLOCKING, ENDISBLOCKING, PAUSEISBLOCKING, RESUMEISBLOCKING, STARTISBLOCKING).

For best performance when using non-blocking accounting actions, it is recommended to specify an RM poll interval with a minimum poll time of zero (such as RMPOLLINTERVAL=0,30). Setting a non-zero minimum poll time can prevent Moab from responding quickly to accounting actions and can result in increased latency in job scheduling.

When using the fast-allocation accounting mode, if the charge action is set to be non-blocking (which is the default), Moab's account balance cache is not updated with the effects of the charge until the iteration after the charge is issued.

5.5.9 Retrying Failed Charges

If the AMCFG[] RETRYFAILEDCHARGES parameter is set to true (this is the default), job charges will be retried if they have failed due to a connection failure. When a job charge or usage record update (such as might occur when a job is suspended, at the periodic charge interval, or when a job completes) results in a connection failure between Moab and the accounting manager, then the charge request will be saved to a file in SPOOLDIR/am/retrying/. Once Moab detects that the connection with the accounting manager has been restored, the charge will be retried up to CHARGERETRYCOUNT times.

Charges that fail due to reasons other than a connection failure, or connection failures that surpass the CHARGERETRYCOUNT, will be saved to files in SPOOLDIR/am/failed/. Although these failures generally represent permanent failures, in some cases it may be possible to reissue some of these charges with a slight modification. For example, a user may have been moved from one account to another after the job started causing the final charge to fail. For such circumstances, a script has been provided (TOOLSDIR/mam/mam-charge-retry.pl) to facilitate the re-issuance of a failed usage charge from a failed charge retry file.

[root]# /software/moab-accounting/tools/mam/mam-charge-retry.pl --help

The mam-charge-retry.pl script mimics the mam-charge command in making a charge to MAM. The specified command-line options will override the original values contained in the failed charge file. The --dry-run option can be used to issue the retry as a quote rather than a charge in order to see if the charge would be successful. The --delete-on-success option can be used to delete the retry file after a successful charge. This script cannot be used to rerun a command when the accounting action uses a native script. In such cases, the modified request XML from the charge retry file can be passed as the standard input to the native script to reissue a charge.

5.5.9.A Using the Script

This section provides synopsis information and an example on using the TOOLSDIR/mam/mam-charge-retry.pl script

Synopisis

mam-charge-retry {[--filename] <retry_filename>} [-J <instance_name>] [-j <usage_record_id>] [-q <quote_id>] [-l <lien_id>] [-T <usage_record_type>] [-u <user_name>] [-g <group_name>] [-a <account_name>] [-o <organization_name>] [-c <class_name>] [-Q <quality_of_service>] [-m <machine_name>] [-N <nodes>] [-P <processors>] [-C <cpu_time>] [-M <memory>] [-D <disk>] [--stage <lifecycle_stage>] [-X, --extension <property>=<value>]... [-t <charge_duration>] [-s <charge_start_time>] [-e <charge_end_time>] [-d <charge_description>] [-z <charge_amount>]] [-f <fund_id>]] [--incremental] [-R <charge_rate_name>][{<charge_rate_value>]}]=<charge_rate_amount>],...]... [--hours] [--itemize] [--delete-on-success] [--dry-run] [--debug] [--site <site_name>]] [--help] [--man] [--quiet] [--verbose] [--version]

 

Reissuing a charge that has failed example

First we will list the files in the SPOOLDIR/am/failed directory to see if there are any "permanently" failed charges that we might want to reissue.

[root]# cd /opt/moab/spool/am/failed 
[root]# ls
job.250

We see there is a failed charge for job 250. It may be useful to check the charge file and examine the message to see what went wrong.

[root]# cat job.250
{"action":"End","message":"Failure registering job End (250) with accounting manager -- Unable to invoke AM request - server rejected request with status code 740 - Failed charging 1.00 credits for instance 250 and created usage record 
25\nUser amy is not a valid member of Account biology","request":"<Request action=\"Charge\"><Object>UsageRecord</Object><Option name=\"AccountingMode\">strict-allocation</Option><Option name=\"StartTime\">1432070300</Option><Option 
name=\"Duration\">300</Option><Data><UsageRecord><Stage>End</Stage><Type>Job</Type><Instance>
250</Instance><User>amy</User><Group>staff</Group><Account>biology</Account><Class>batch</Class><QualityOfService>premium</QualityOfService>
<Machine>colony</Machine><Nodes>1</Nodes><Partition>colony</Partition><Processors consumptionRate=\"1.00\">12</Processors><StartTime>1432070300</StartTime><SubmitTime>1432070300</SubmitTime><EndTime>1432070600</EndTime>
<CompletionCode>0</CompletionCode></UsageRecord></Data></Request>"}

We can see that this charge failed because the user (amy) was not a member of the specified account (biology). In this case, the user was a member of the biology account when the job started, but had been moved to the account chemistry by the time the job ended, resulting in a charge failure.

If we were to reissue the charge without modification, it would fail again, as we can see by using the script with the --dry-run option.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 --dry-run
 

User amy is not a valid member of Account biology

We can reissue the charge after changing the request to use her new chemistry account.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 -a chemistry --dry-run

Successfully quoted 1.00 credits for instance 250

Since that looks like it will work correctly, we'll issue the corrected charge request and delete the charge file.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 -a chemistry --delete-on-success

Successfully charged 1.00 credits for instance 250 and created usage record 35

Related Topics 

  • 5.6 AMCFG Parameters and Flags
  • Per Class DISABLEAM attribute
  • 6.5.4 Charging for Reserved Resources
  • ENFORCEACCOUNTACCESS parameter

    • © 2016 Adaptive Computing