(Click to open topic with navigation)

You are here: 5 Managing Fairness - Throttling Policies, Fairshare, and Allocation Management > Accounting, Charging, and Allocation Management

5.5 Accounting, Charging, and Allocation Management

In this topic:

For a complete list of and additional information on the AMCFG parameters and flags, see 5.6 AMCFG Parameters and Flags.

5.5.1 Accounting Manager Overview

An accounting manager is a software system that enables tracking and charging for job resource usage. Moab Accounting Manager is a commercial charge-back accounting system that has built-in integration with Moab Workload Manager. Moab Accounting Manager can be used in a variety of accounting modes such as for usage tracking, notional charging or allocation enforcement.

When used for usage tracking only, the accounting manager simply records workload usage details. When configured additionally to perform charging, resource charge rates are used to impute a charge for each job. When configured to enforce resource allocation limits, jobs are charged against allocations and new jobs may be blocked from running if their account runs out of funds. See Accounting Mode and see Select an Appropriate Accounting Mode in the Moab Accounting Manager Administrator Guide for more details on supported accounting modes.

In a typical allocation enforcement use case, credits are allocated to accounts for designated time periods; establishing limits on the use of compute resources. The base currency credits can be defined in terms of system resource units (e.g. Processor-Seconds) or a real currency (e.g. U.S. dollars). Charge rates are established for the use of resources. Accounts are created and users are given access to the appropriate accounts. Deposits are made into funds associated with the account’s creating allocations. An allocation cycle can be established whereby funds are reset on a regular periodic basis (such as yearly, quarterly, or monthly) and where allocations are renewed for accepted accounts. Before a job is started, Moab Workload Manager will verify that the user has sufficient credits to run the job by attempting to place a hold against their funds (referred to as a lien). When a job completes, the user's funds will be debited via a charge, usage information will be recorded for the job, and the lien will be removed.

5.5.2 Accounting Mode

The accounting mode (specified via the AMCFG[] MODE parameter) modifies the way in which accounting-relevant job and reservation stages (e.g. create, start, end, etc.) are processed. See 5.5.7 Accounting Stages for more information on the behaviors of the different values of the accounting mode.

The following table describes the valid values for the accounting mode.

Value Description
strict‑allocation Use this mode if you wish to strictly enforce allocation limits. Under this mode, holds (called liens) will be placed against allocations in order to prevent multiple jobs from starting up on the same funds. Jobs and reservations may be prevented from running if the end-users do not have sufficient funds. This is the default.
fast‑allocation

Use this mode if you wish to debit allocations, but need higher throughput by eliminating the lien and quote operations of strict-allocation mode. Under this mode, jobs and reservations check a cached account balance, and may be prevented from running after the balance has become zero or negative.

The cached account balances in Moab can be viewed by running "mrmctl -q AccountBalanceCache AM".

If you are using fast-allocation, funds are assumed to have account-based constraints only. Moab will reject funds having no constraints or having non-account constraints. It is highly recommended that you enable ENFORCEACCOUNTACCESS to TRUE and AMCFG[] CREATECRED=TRUE with an appropriate refresh period (via AMCFG[] REFRESHPERIOD) so that Moab can prevent jobs from running under accounts that the user does not belong to (this is enforced via liens in the strict-allocation accounting mode). Also, the configured refresh period will apply to both credential updates and account balance updates. See Appendix A Moab Parameters for more information on the ENFORCEACCOUNTACCESS parameter.
notional‑charging Use this mode if you wish to calculate and record charges for workload usage, but not keep track of fund balances or allocation limits.
usage‑tracking Use this mode if you wish to record workload usage details, but not to calculate a charge nor keep track of fund balances or allocation limits.

5.5.3 Accounting Manager Interface Types

Moab Workload Manager supports two accounting manager interface types: MAM and Native.

5.5.3.A MAM

The MAM accounting manager interface type enables direct communication between Moab Workload Manager and Moab Accounting Manager. This often results in the fastest accounting performance. Use this interface type if you do not need to customize the interaction with the accounting manager.

To configure Moab to use the MAM accounting manager interface, run configure using the --with-am option.

Example 5-8:  

./configure -with-am=mam ...

Consequently, make install will add the essential configuration and connection entries into the moab.cfg and moab-private.cfg files.

The following are typical entries in the Moab configuration files for using the MAM interface:

Synchronize the secret key with Moab Accounting Manager by copying the value of the token.value parameter from the MAM_PREFIX/etc/mam-site.conf file which is randomly generated during the Moab Accounting Manager install process.

When using the MAM accounting manager interface, by default Moab will communicate directly with Moab Accounting Manager via the SSS wire protocol. However, it is possible to enable a hybrid model and override individual accounting actions by specifying the exec protocol and the path of a custom script to the appropriate AMCFG[] *URL parameters.

Moab Accounting Manager should be installed, started, and initialized. See Initial Setup in the Moab Accounting Manager Administrator Guide for examples of how to initialize MAM for your initial mode of operation.

5.5.3.B Native

The Native accounting manager interface type provides a customization layer between Moab Workload Manager and Moab Accounting Manager. This interface can be used where greater accounting customization is required. The native interface can also be customized to interact with third-party accounting manager systems. Moab passes job accounting details to scripts that handle the interaction with the external system.

To configure Moab to use the MAM accounting manager interface, run configure using the --with-am=native option.

Additionally, you may need to use the --with-am-dir configure option to specify the prefix directory for Moab Accounting Manager if MAM has been installed in a non-default location.

Example 5-9:  

./configure --with-am=native ...

Consequently, make install will add the essential accounting manager entries into moab.cfg and install the accounting-related scripts ($PREFIX/tools/mam/usage.*.mam.pl) in the correct locations..

Moab will default to using a set of stock scripts for the accounting stages. To view the scripts that are currently in use, run mdiag -R -v am: (even more information may be available in mdiag -R -v --xml). The following shows sample output from running the mdiag -R -v am: command.

AM[mam]  Type:  Native  State: 'Active'
 Timeout:                            15
 Accounting Mode:                    strict-allocation
 Job Charge Policy:                  All
 Reservation Charge Policy:          Select
 Create URL:                         exec:///opt/moab/tools/mam/usage.quote.mam.pl
 Start URL:                          exec:///opt/moab/tools/mam/usage.reserve.mam.pl
 Pause URL:                          exec:///opt/moab/tools/mam/usage.charge.mam.pl
 Resume URL:                         exec:///opt/moab/tools/mam/usage.reserve.mam.pl
 Update URL:                         exec:///opt/moab/tools/mam/usage.charge.mam.pl
 Continue URL:                       exec:///opt/moab/tools/mam/usage.reserve.mam.pl
 End URL:                            exec:///opt/moab/tools/mam/usage.charge.mam.pl
 Delete URL:                         exec:///opt/moab/tools/mam/lien.delete.mam.pl
 Query URL:                          exec:///opt/moab/tools/mam/account.query.mam.pl
 Retry Failed Charges:               TRUE

Moab will invoke the native accounting manager scripts by passing the job or reservation information via XML to the standard input of the script. You may override any of the default scripts with a custom script by specifying the appropriate AMCFG URL parameter in the moab server configuration file. See 5.6 AMCFG Parameters and Flags for CREATEURL, STARTURL, PAUSEURL, RESUMEURL, UPDATEURL, CONTINUEURL, ENDURL, DELETEURL, and QUERYURL values for more information.

The XML sent to the scripts is in the form of an SSS Request that is identical to the Request sent to MAM when you use the MAM Accounting Manager Interface type. For example, the XML sent to the usage.charge.mam.pl script in a final charge consists of an encapsulating Request element with an action attribute that has a value of "Charge"; an object element with a value of "UsageRecord"; one or more optional Option elements; and a Data element. The Data element has a single UsageRecord element with property elements describing the job or reservation properties. For example:

<Request action="Charge"><Object>UsageRecord</Object><Option name="Duration">1234</Option><Data><UsageRecord><Type>Job</Type><Instance>Moab.165</Instance><User>amy</User><Group>staff</Group><Account>chemistry</Account><Class>batch</Class><QualityOfService>high</QualityOfService><Machine>colony</Machine><Nodes>1</Nodes><NodeType>Fast</NodeType><NodeCharge>2.000000</NodeCharge><Partition>Torque</Partition><Processors consumptionRate="0.50">2</Processors><Memory>2048</Memory><Matlab>2</Matlab><StartTime>1398805354</StartTime><EndTime>1398805357</EndTime><CompletionCode>0</CompletionCode><OpSys>CentOS 6</Opsys><Temp>87.00</Temp></UsageRecord></Data></Request> *

In the sample XML above, Matlab is an example of a generic resource, Opsys is an example of a job variable, and Temp is an example of a generic metric.

A reservation charge, or quote or lien, is very similar. For example:

<Request action="Charge"><Object>UsageRecord</Object><Option name="Duration">7200</Option><Data><UsageRecord><Type>Reservation</Type><Instance>reservation.7</Instance><User>amy</User><Machine>colony</Machine><Nodes>1</Nodes><ProcessorsconsumptionRate="0.76">12</Processors><Duration>7200</Duration><StartTime>1398797430</StartTime><EndTime>1398804630</EndTime></UsageRecord></Data></Request>

The majority of the scripts use this same basic XML format; for instance, usage.quote.mam.pl, usage.reserve.mam.pl, and usage.charge.mam.pl.

The XML sent to the lien.delete.mam.pl script to clean up after a failure consists of an encapsulating Request element with an action attribute that has a value of "Delete"; an object element with the value of "Lien"; and a condition (Where) element indicating the lien instance to delete. For example:

<Request action="Delete"><Object>Lien</Object><Where name="Instance">Moab.127</Where></Request>

The script should return a return code (zero for success), data on standard out and messages on standard error. A failure in CREATEURL, STARTURL, RESUMEURL, or CONTINUEURL should result in the application of the CREATEFAILUREACTION, STARTFAILUREACTION, RESUMEFAILUREACTION, or CONTINUEFAILUREACTION respectively.

Moab Accounting Manager should be installed, started, and initialized. The simplest procedure is to install it on the same server as Moab Workload Manager so that the Moab Accounting Manager can share libraries and configuration files with the Moab Workload Manager and Moab Accounting Manager scripts. See Initial Setup in the Moab Accounting Manager Administrator Guide for examples of how to initialize MAM for your initial mode of operation.

5.5.4 Charging for Jobs

Moab can be configured to charge for the resources used in jobs.

Job tracking and charging via an accounting manager is enabled or disabled by the AMCFG[] JOBCHARGEPOLICY parameter. By default, if an accounting manager is defined in Moab, Moab will charge for all jobs, independent of the job’s exit status (derived from an internal default of AMCFG[] JOBCHARGEPOLICY=All). This is the policy that is most frequently used because of the unfortunate fact that a clever user can fake a job failure to avoid being charged for their job. Jobs that are discovered to have failed due to system issues can be proactively refunded by the system admin, or the user may be required to apply for a refund. Charges may be restricted to jobs that complete successfully by setting AMCFG[] JOBCHARGEPOLICY=Successful. To disable charging for jobs entirely, set AMCFG[] JOBCHARGEPOLICY=None.

When using the default accounting mode of strict-allocation, before Moab starts a job, it contacts the accounting manager and requests an allocation reservation (or lien) be placed on the associated account. The lien amount is equivalent to the total cost of resources that could be consumed by the job (based on the job's wallclock limit) and is used to prevent the possibility of allocation oversubscription. Moab then starts the job. When the job completes, Moab debits the allocation by the amount actually consumed by the job and then releases the lien.

These steps should be transparent to users. Only when an account has insufficient allocations to run a requested job will the presence of the accounting manager be noticed. The policies guiding what action should taken if a user is out of funds is dictated by the AMCFG[] STARTFAILUREACTION (and related) parameters. If desired, a fallback account may be specified for use when a job's primary account is out of allocations. This account, specified using the AMCFG parameter's FALLBACKACCOUNT attribute, is often associated with a low QoS privilege, priority, and cost and is often configured to run only when no other jobs are present.

The actual policies that determine what resources are charged for and in what amounts are specified by the charge rates defined in the accounting manager. Moab will pass the job properties (shown in 5.5.6 Accounting Properties Reported to the Accounting Manager) to the accounting manager. It is the task of the accounting manager to record and charge for the job according to site objectives.

5.5.5 Charging for Reservations

Moab can be configured to charge for the unused cycles in reservations. One of the hesitations with dedicating resources to a particular group is that if the resources are not used by that group, they go idle and are wasted. By configuring a reservation to be chargeable, sites can charge every idle cycle of the reservation to a particular account. When the reservation is in use, the consumed resources will be charged to the jobs using the resources. When the resources are idle, the resources will be charged to the reservation's charge account. See AMCFG[] RESERVATIONCHARGEPOLICY.

If RESERVATIONCHARGEPOLICY is set to Select (the default), charging may be enabled for select reservations by specifying the CHARGEACCOUNT and CHARGEUSER attributes for the reservation. For standing reservations, these are set via the SRCFG[X] CHARGEACCOUNT and CHARGEUSER parameters. For administrative reservations, these are set via the -S aaccount and auser options.

Example 5-10: Enabling charging in a standing reservation

SRCFG[foo] PERIOD=DAY DAYS=Mon,Tue,Wed,Thu,Fri DEPTH=1 USERLIST=amy CHARGEACCOUNT=chemistry CHARGEUSER=amy RESOURCES=PROCS:1 TASKCOUNT=2

Example 5-11: Enabling charging in an administrative reservation

mrsvctl -c -a USER=amy -S aaccount=chemistry -S auser=amy -R procs=1 -t 1 -d 7200

If RESERVATIONCHARGEPOLICY is set to All, idle cycles will be charged for all reservations unless disabled for individual reservations by specifying the reservation Charge attribute with a value of False. For standing reservations, this are set via the SRCFG CHARGE parameter. For administrative reservations, this is set via the -S charge options.

Example 5-12: Disabling charging in a standing reservation

SRCFG[foo] PERIOD=DAY DAYS=Mon,Tue,Wed,Thu,Fri DEPTH=1 USERLIST=amy CHARGE=False RESOURCES=PROCS:1 TASKCOUNT=2

Example 5-13: Disabling charging in an administrative reservation

mrsvctl -c -a USER=amy -S charge=False -R procs=1 -t 1 -d 7200

If RESERVATIONCHARGEPOLICY is set to None, idle cycles will not be charged for any reservations, regardless of what attributes are specified with the reservation.

When utilizing the accounting manager to track or charge for idle cycles in any reservations, the accounting manager must be configured to track and charge for reservation-relevant properties. See the examples for “Enabling Reservation Statistics” and “Charging for the unused cycles in reservations” in the Moab Accounting Manager Administrator Guide for steps on how to do this for the Moab Accounting Manager.

5.5.6 Accounting Properties Reported to the Accounting Manager

Moab can send the following information to the accounting manager via charging actions:

5.5.6.A For Jobs

Property name in the Accounting Manager Usage Record Description of property value recorded in the Accounting Manager Usage Record
Account Account name
BlockedProcessors*

Number of processors blocked from use to other jobs. This might be more than the allocated or requested processors if entire nodes are given exclusive access to the job (e.g. from a node access policy or node exclusivity flag).

Tracking or charging for BlockedProcessors should only be used with policies that allow only a single job to dedicate a node such as with a Node Access Policy of SINGEJOB or SINGLETASK, or using a QOS with the DEDICATED flag.

Using BlockedProcessors with any policy allowing more than one job to dedicate a node (such as a Node Access Policy of SINGLEUSER, SINGLECLASS, SINGLEACCOUNT or UNIQUEUSER) will yield inconsistent results and is not recommended or supported.
Charge If the AMCFG LOCALCOST flag is set, Moab will calculate and pass the Charge amount to MAM. If it is not, MAM will calculate the charge based on the transmitted job properties.
Class Class/queue name
Cores*

NUMA cores allocated to the job.

Cores will be reported to the accounting manager when using the -L NUMA Resource Request syntax and specifying place=socket, place=numanode or place=core.
CPUTime

CPU time. The value sent by Moab to the accounting manager is the cumulative CPU time for the job.

Using this value as the basis of charging is not compatible with periodic charging.
Duration Moab sends the wallclock time for the job charge(s) in seconds. This is aggregated in MAM as Duration.
EndTime Job end time

Exit Code*

(formerly CompletionCode*)

 Exit code
Features*

Allocated node features.

Node features are sent in the JSON object format {"<node_feature_or_combo_name>":<node_feature_or_combo_count>,...}.

Node features are not passed to the accounting manager unless configured to do so via the AMCFG[] IncludeFeatures parameter.

GPUs* Number of GPUs allocated to the job.
Group Group name
Instance Job ID
Licenses

License generic resources allocated to the job. License generic resources are sent in the JSON object format {"<license_generic_resource_name>":<license_generic_resource_value>,...}. The license generic resource value is the number of this license generic resource consumed by the job.

License generic resources are distinguished by those generic resources having the license trait. Generic resources can be marked as license generic resources in two primary ways.

  • Any generic resources seen on nodes reported via a native resource manager having the server configuration parameter RMCFG[] RESOURCETYPE=License will be marked as a license generic resource.
  • Generic resources defined in Moab may be identified as license types by using the Moab configuration parameter GRESCFG[] LICENSE=TRUE.

One may distinguish the generic resources having the license trait via the the command `mschedctl -l gres -v`.
Machine Cluster (RM) name
Memory Dedicated or utilized memory in megabytes
Metrics Generic metrics are sent in the JSON object format {"<generic_metric_name>":<generic_metric_value>,...} where the generic metric value is the average value of the generic metric across the nodes of the job and across time.
MICs* Number of MICs allocated to the job.
NodeCharge* Aggregate node charge rate. See NODECHARGEPOLICY and CHARGERATE for more information.
Nodes Node count
NodeType* Node type. See NODETYPE for more information.
NumaNodes* NUMA nodes allocated to the job. NumaNodes will be reported to the accounting manager when using the -L NUMA Resource Request syntax and specifying place=socket or place=numanode.
Partition Partition name
ProcessorEquivalents* Processor Equivalents
Processors Number of processors allocated to the job. This is normally equivalent to the requested processors.
QualityOfService QoS name
QueueDuration* Effective duration the job was in the idle state
RequestedDuration Requested wallclock limit
Resources

Generic resources are sent in the JSON object format {"<generic_resource_name>":<generic_resource_value>,...}. The generic resource value is the number of this generic resource consumed by the job.
Stage Accounting stage
Sockets* NUMA sockets allocated to the job. Sockets will be reported to the accounting manager when using the -L NUMA Resource Request syntax and specifying place=socket.
StartTime Job start time
SubmitTime Job submission time
Threads* NUMA threads allocated to the job. Threads will be reported to the accounting manager when using the -L NUMA Resource Request syntax and specifying place=socket, place=numanode, place=core or place=thread.
Type Set to "Job"
User User name
Variables Job variables are sent in the JSON object format {"<job_variable_name>":"<job_variable_value>",...}.

* For this property to be recorded in the MAM Usage Record, you must define a custom usage record attribute in MAM for it. See Customizing the Usage Record Object in the Moab Accounting Manager Administrator Guide for more information.

5.5.6.B For Reservations

Property name in MAM Usage Record Description of property value recorded in MAM Usage Record
Account Charge account
Duration Moab sends the wallclock time for the reservation in seconds. This is aggregated in MAM as Duration.
EndTime Reservation end time
IdleProcessorSeconds Processor seconds not blocked by jobs within the reservation. This is the metric typically used to charge for spare cycles not attributed to jobs within the reservation.
Instance Reservation ID
Machine Cluster (RM) name
Nodes Number of node allocated to the reservation
Partition Partition name
Processors Number of processors allocated to the reservation
ReservedProcessorSeconds Total processor seconds included within the reservation (to-date)
Stage Accounting stage
StartTime Reservation start time
Type Set to "Reservation"
User Charge user or reservation owner

5.5.7 Accounting Stages

The accounting manager performs various actions throughout different stages of a job or reservation lifetime. For a stock configuration (meaning you have not overridden the accounting actions with custom scripts), the following describes the stages and the respective actions that occur at these stages depending on the accounting mode:

* The cached account balances in Moab can be viewed by running "mrmctl -q AccountBalanceCache AM".

5.5.8 Accounting Events

You can add accounting events to the event log by specifying one or more of the following with RECORDEVENTLIST:

Event Description
AMCREATE Record accounting events trigger when an object is created; for example, when a balance check occurs at job submission.
AMDELETE Record accounting events triggered when an object's normal accounting lifecycle is interrupted; for example, when the lifecycle is interrupted to clean up reservations for a failed job start.
AMEND Record accounting events triggered when an object ends; for example, when a charge occurs at the end of a job.
AMPAUSE Record accounting events triggered when an object is paused; for example, when a partial charge occurs when a job is paused.
AMQUOTE Record accounting events triggered when an object requires a quote amount.
AMRESUME Record accounting events triggered when an object is resumed; for example, when a lien is made when a job is resumed.
AMSTART Record accounting events triggered when an object is started; for example, when a lien is made when a job starts.
AMUPDATE Record accounting events triggered when an object continues past a flush interval; for example, when a partial charge occurs and new lien is made for a job.

5.5.9 Blocking Versus Non-Blocking Accounting Actions

Moab uses a thread pool to perform non-blocking actions. Instead of blocking the scheduling thread, the request is added to a queue that is serviced by the accounting thread pool. Using the thread pool to perform non-blocking accounting actions can result in faster aggregate scheduling and better client response times, though individual actions can, in some cases, be shortly delayed. By default, Moab uses non-blocking calls for the final charge only. The default behavior for individual accounting actions (such as Create, Start, End) can be overridden via the associated parameter (CONTINUEISBLOCKING, CREATEISBLOCKING, DELETEISBLOCKING, ENDISBLOCKING, PAUSEISBLOCKING, RESUMEISBLOCKING, STARTISBLOCKING).

For best performance when using non-blocking accounting actions, it is recommended to specify an RM poll interval with a minimum poll time of zero (such as RMPOLLINTERVAL=0,30). Setting a non-zero minimum poll time can prevent Moab from responding quickly to accounting actions and can result in increased latency in job scheduling.

When using the fast-allocation accounting mode, if the charge action is set to be non-blocking (which is the default), Moab's account balance cache is not updated with the effects of the charge until the iteration after the charge is issued.

5.5.10 Retrying Failed Charges

If the AMCFG[] RETRYFAILEDCHARGES parameter is set to true (this is the default), job charges will be retried if they have failed due to a connection failure. When a job charge or usage record update (such as might occur when a job is suspended, at the periodic charge interval, or when a job completes) results in a connection failure between Moab and the accounting manager, then the charge request will be saved to a file in SPOOLDIR/am/retrying/. Once Moab detects that the connection with the accounting manager has been restored, the charge will be retried up to CHARGERETRYCOUNT times.

Charges that fail due to reasons other than a connection failure, or connection failures that surpass the CHARGERETRYCOUNT, will be saved to files in SPOOLDIR/am/failed/. Although these failures generally represent permanent failures, in some cases it may be possible to reissue some of these charges with a slight modification. For example, a user may have been moved from one account to another after the job started causing the final charge to fail. For such circumstances, a script has been provided (TOOLSDIR/mam/mam-charge-retry.pl) to facilitate the re-issuance of a failed usage charge from a failed charge retry file.

[root]# /software/moab-accounting/tools/mam/mam-charge-retry.pl --help

The mam-charge-retry.pl script mimics the mam-charge command in making a charge to MAM. The specified command-line options will override the original values contained in the failed charge file. The --dry-run option can be used to issue the retry as a quote rather than a charge in order to see if the charge would be successful. The --delete-on-success option can be used to delete the retry file after a successful charge. This script cannot be used to rerun a command when the accounting action uses a native script. In such cases, the modified request XML from the charge retry file can be passed as the standard input to the native script to reissue a charge.

5.5.10.A Using the Script

This section provides synopsis information and an example on using the TOOLSDIR/mam/mam-charge-retry.pl script

Synopisis

mam-charge-retry {[--filename] <retry_filename>} [-J <instance_name>] [-j <usage_record_id>] [-q <quote_id>] [-l <lien_id>] [-T <usage_record_type>] [-u <user_name>] [-g <group_name>] [-a <account_name>] [-o <organization_name>] [-c <class_name>] [-Q <quality_of_service>] [-m <machine_name>] [-N <nodes>] [-P <processors>] [-C <cpu_time>] [-M <memory>] [-D <disk>] [--stage <lifecycle_stage>] [-X, --extension <property>=<value>]... [-t <charge_duration>] [-s <charge_start_time>] [-e <charge_end_time>] [-d <charge_description>] [-z <charge_amount>]] [-f <fund_id>]] [--incremental] [-R <charge_rate_name>][{<charge_rate_value>]}]=<charge_rate_amount>],...]... [--hours] [--itemize] [--delete-on-success] [--dry-run] [--debug] [--site <site_name>]] [--help] [--man] [--quiet] [--verbose] [--version]

 

* For this property to be recorded in the MAM Usage Record, you must define a custom usage record attribute in MAM for it. See Customizing the Usage Record Object in the Moab Accounting Manager Administrator Guide for more information.

Reissuing a charge that has failed example

First we will list the files in the SPOOLDIR/am/failed directory to see if there are any "permanently" failed charges that we might want to reissue.

[root]# cd /opt/moab/spool/am/failed 
[root]# ls
job.250

We see there is a failed charge for job 250. It may be useful to check the charge file and examine the message to see what went wrong.

[root]# cat job.250
{"action":"End","message":"Failure registering job End (250) with accounting manager -- Unable to invoke AM request - server rejected request with status code 740 - Failed charging 1.00 credits for instance 250 and created usage record 
25\nUser amy is not a valid member of Account biology","request":"<Request action=\"Charge\"><Object>UsageRecord</Object><Option name=\"AccountingMode\">strict-allocation</Option><Option name=\"StartTime\">1432070300</Option><Option 
name=\"Duration\">300</Option><Data><UsageRecord><Stage>End</Stage><Type>Job</Type><Instance>
250</Instance><User>amy</User><Group>staff</Group><Account>biology</Account><Class>batch</Class><QualityOfService>premium</QualityOfService>
<Machine>colony</Machine><Nodes>1</Nodes><Partition>colony</Partition><Processors consumptionRate=\"1.00\">12</Processors><StartTime>1432070300</StartTime><SubmitTime>1432070300</SubmitTime><EndTime>1432070600</EndTime>
<CompletionCode>0</CompletionCode></UsageRecord></Data></Request>"}

We can see that this charge failed because the user (amy) was not a member of the specified account (biology). In this case, the user was a member of the biology account when the job started, but had been moved to the account chemistry by the time the job ended, resulting in a charge failure.

If we were to reissue the charge without modification, it would fail again, as we can see by using the script with the --dry-run option.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 --dry-run
 

User amy is not a valid member of Account biology

We can reissue the charge after changing the request to use her new chemistry account.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 -a chemistry --dry-run

Successfully quoted 1.00 credits for instance 250

Since that looks like it will work correctly, we'll issue the corrected charge request and delete the charge file.

[root]# /opt/moab/tools/mam/mam-charge-retry.pl job.250 -a chemistry --delete-on-success

Successfully charged 1.00 credits for instance 250 and created usage record 35

Related Topics 

  • 5.6 AMCFG Parameters and Flags
  • Per Class DISABLEAM attribute
  • 6.5 Reservation Policies
  • ENFORCEACCOUNTACCESS parameter

    • © 2016 Adaptive Computing