Appendix H: Interfacing with Moab (APIs)
Moab provides numerous interfaces allowing it to monitor and manage most services and resources. It also possesses flexible interfaces to allow it to interact with peer services and applications as both a broker and an information service. This appendix is designed to provide a general overview and links to more detailed interface documentation.
- H.1 Moab Query and Control APIs
- Allow external portals and services to obtain information about compute resources, workload, and usage statistics.
- H.2 Resource Management Interfaces
- Allow Moab to monitor, schedule, and control services and resources.
- H.3 Identity and Credential Management Interfaces
- Allow monitoring and active management of user configuration, credentials, policies, and usage information.
- H.4 Accounting and Event Interfaces
- Allow import/export of accounting and event information to external entities.
- H.5 Grid Services API
- Provide and use information, data, job, and resource management services in a distributed environment.
- H.6 Discovery/Directory Services
- H.7 Job Submission and Management Interface
- Query resource availability, submit, modify, and manage jobs, and query the status of active and completed jobs.
Moab interfaces to systems providing various services and using various protocols. This appendix is designed to assist users who want to enable Moab in new environments using one of the existing interfaces. It does not cover the steps required to create a new interface.
H.1 Query and Control APIs
The Moab Cluster and Grid Suites provide a (Moab) workload manager server that supports a broad array of client services. These services can either be directly accessed via Moab client commands or by way of a number of APIs. Which approach is best will depend on the particular use case.
H.1.1 Java API
The Moab-API is a Java library that allows Java developers to easily communicate with and interact with the Moab Workload Manager. The API handles the details of creating a secure connection, reporting information about workload and resources, and creating basic commands.
New users are encouraged to view the Moab Java API Quick Start Guide for details on interfacing with this API. The complete Javadocs are also available for review.
H.1.2 C API
The Moab C API provides access to all Moab information and services. This API consists of both easy to use high-level calls and powerful and flexible low-level calls.
High-Level C API Calls
| MCCInitialize(Host,Port,&C) | |
| Args: | Host: (char *) optional server hostname or NULL Port: (int) optional server port or 0 C: (void **) opaque interface handle |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call initializes the interface between the client and the server. It must be called once before attempting to issue any Moab client calls. |
| MCCJobCancel(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to cancel (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will cancel the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobCheckpoint(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to cancel (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will checkpoint the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobGetNumAllocRes(C,JID,Count,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to query (optional) Count: (int *) output value (number of allocated hosts) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will return the number of allocated hosts for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobGetRemainingTime(C,JID,Time,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to query (optional) Time: (long *) output value (remaining time in seconds) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will return the remaining wallclock duration/time for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. Note: This call can be made to cache data for performance reasons by setting the MCCCACHETIME #define to the desired cache duration (in seconds) at build time. |
| MCCJobListAllocRes(C,JID,OBuf,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to query (optional) OBuf: (char **) output response string (comma delimited host list) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will return a list of allocated hosts for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobMigrate(C,JID,Dest,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to migrate (optional) Dest: (char *) destination cluster or list of hosts (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will migrate the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobModify(C,JID,Attr,Val,Op,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) id of job to modify (optional - if not specified, modify job in current context) Attr: (char *) job attribute to modify (see mjobctl) Val:(char *) value to use when modifying specified attribute Op:(char *) operation to perform on attribute (one of set, unset, incr, or decr) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will modify the attributes of either the specified job (if JID is populated) or the job in the current context (if JID is empty). If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobQuery(C,JID,OBuf,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to query (optional - if not specified, report all jobs) OBuf: (char **) output response string (reported in XML) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will suspend the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobRequeue(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to requeue (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will requeue the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobResume(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to resume (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will resume the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobSignal(C,JID,Signo,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to resume (optional) Signo: (int) job signal to send to job EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will signal the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobStart(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to start/run (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will immediately run/execute the specified job ignoring policies and dependencies provided that adequate idle resources are available. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCJobSubmit(C,JDesc,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JDesc: (char *) XML job description (Scalable Systems Software Job Object Specification) JID: (char **) job id of new job (JID is populated on success only) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will submit the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. On success, memory is allocated for JID. JID is then populated with a new job's ID (which can then be used in calls such as MCCJobQuery and MCCJobCancel). The client is responsible for freeing the space allocated for JID. |
| MCCJobSuspend(C,JID,EMsg) | |
| Args: | C: (void **) address of interface handle JID: (char *) job id to suspend (optional) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will suspend the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCNodeQuery(C,NID,OBuf,EMsg) | |
| Args: | C: (void **) address of interface handle NID: (char *) node id to query (optional - if not specified, report all nodes) OBuf: (char **) output response string (reported in XML) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will suspend the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCNodeModify(C,NID,Attr,Val,Op,EMsg) | |
| Args: | C: (void **) address of interface handle NID: (char *) id of node to modify (optional - if not specified, modify job in current context) Attr: (char *) node attribute to modify (see mnodectl) Val:(char *) value to use when modifying specified attribute Op:(char *) operation to perform on attribute (one of set, unset, incr, or decr) EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will modify the attributes of either the specified node (if NID is populated) or the local node (if NID is empty). If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
| MCCFree(C) | |
| Args: | C: (void **) address of interface handle |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call will free allocated interface handle memory |
Example 1 - Interfacing to an Explicitly Specified Moab Server
#include <stdio.h>
#include <stdlib.h>
#include "mapi.h"
int main(
int ArgC,
char *ArgV[],
char *EnvP[])
{
void *C;
char EMsg[1024];
char *OBuf;
char JobID[32];
char *JobDescription = "<Job><Arguments>600</Arguments>
<Executable>/bin/sleep</Executable><InitialWorkingDirectory>/home/wightman</InitialWorkingDirectory>\
<NodeCount>1</NodeCount><GroupId>wightman</GroupId><UserId>wightman</UserId>\
<Requested></Requested><Processors>1</Processors></Job>";
if (MCCInitialize("sn-master03",14522,&C) < 0)
{
fprintf(stderr,"cannot initialize interface\n");
exit(-1);
}
if (MCCJobSubmit(&C,JobDescription,&OBuf,EMsg) < 0)
{
fprintf(stderr,"Error: '%s'\n",
EMsg);
exit(-1);
}
fprintf(stdout,"Results: Job '%s' successfully submitted\n",
OBuf);
snprintf(JobID,sizeof(JobID),"%s",OBuf);
free(OBuf);
OBuf = NULL;
if (MCCJobQuery(&C,JobID,&OBuf,EMsg) < 0)
{
fprintf(stderr,"Error: '%s'\n",
EMsg);
exit(-1);
}
fprintf(stdout,"Results: '%s'\n",
OBuf);
free(OBuf);
OBuf = NULL;
if (MCCJobCancel(&C,JobID,EMsg) < 0)
{
fprintf(stderr,"ERROR: '%s'\n",
EMsg);
exit(-1);
}
fprintf(stdout,"Job '%s' successfully canceled\n",
JobID);
MCCFree(NULL);
return(0);
} /* END main() */
Example 2 - Interface to Monitor Remaining Time w/in a Batch Job
#include <stdio.h>
#include <stdlib.h>
#include "mapi.h"
int main(
int ArgC,
char *ArgV[],
char *EnvP[])
{
mulong Time;
/* from inside a batch job, API will auto allocate the interface and
discover the appropriate jobid */
/* monitor time, clean-up and exit if less then 300 seconds remain */
while (1)
{
MCCJobGetRemainingTime(NULL,NULL,&Time,NULL);
if (Time < 300)
exit(0);
sleep(1);
}
/*NOTREACHED*/
exit(1);
}
gcc client.c -I/opt/moab-5.1.0/include/ -L/opt/moab-5.1.0/lib/ -lcmoab -lmoab -lpthread -lm -lmcom -lminit -o client
The Moab low-level C API allows direct access to all Moab client services. This API uses the MCCExecute routine and takes as an argument an XML request string. This interface provides a high-speed interface directly into the server and provides very broad and flexible cluster, workload, and resource control. This routine creates a direct socket connection to provides the following services:
- socket creation
- client authentication
- server discovery via built-in, configfile, and environment settings
- command request and response encryption
- marshalling and de-marshalling of requests and data
Requests are sent using native XML request strings and, depending on the command used, responses are returned in unformatted native XML or pretty-print human-readable format. The commands that do not return XML can be configured to return XML with the --format=xml option. The commands that do not natively return XML are:
- setspri
- setres
- releaseres
- runjob
- checkjob
- showconfig
- mdiag
- mjobctl
Use of Cluster Resources Consulting Services is recommended for organizations using this interface.
| MCCInitialize(Host,Port,&C) | |
| Args: | Host: (char *) optional server hostname or NULL Port: (int) optional server port or 0 C: (void **) opaque interface handle |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call initializes the interface between the client and the server. It should be called once before attempting to issue any Moab client calls if connecting to a server at a non-default location. If connecting to the default server, client commands will automatically initialize the interface handle. |
| MCCExecute(C,RCmd,RBuf,&OBuf,EMsg) | |
| Args: | C: (void **) address of interface handle RCmd: (char *) request client command RBuf: (char *) request command XML OBuf: (char **) output response string EMsg: (char *) optional error message |
| Return Code: | Returns 0 on success or -1 on failure |
| Description: | This call executes the requested command specified by the RBuf string using the client command specified by RCmd. Command output is placed in the OBuf buffer. This buffer is allocated and must be freed after use. The EMsg parameter is a 1024 character array and is optional. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. |
H.1.3 CLI (Command Line Interface) XML API
All Moab client commands can report results in XML format to allow the information to be easily integrated into peer services, portals, databases, and other applications. To request that a client command report its output in XML, specify the --format=xml flag as in the following example:
> showq --format=xml <Data> <Object>queue</Object> <cluster LocalActiveNodes="1" LocalAllocProcs="1" LocalIdleNodes="0" LocalIdleProcs="3" LocalUpNodes="1" LocalUpProcs="4" RemoteActiveNodes="0" RemoteAllocProcs="0" RemoteIdleNodes="0" RemoteIdleProcs="0" RemoteUpNodes="0" RemoteUpProcs="0" time="1128451812"></cluster> <queue count="1" option="active"> <job AWDuration="11672" EEDuration="1128451812" Group="[DEFAULT]" JobID="Moab.2" MasterHost="cw2" PAL="2" QOS="bug3" ReqAWDuration="54000" ReqNodes="1" ReqProcs="1" RsvStartTime="1128451812" RunPriority="0" StartPriority="1" StartTime="1128451812" StatPSDed="11886.580000" StatPSUtl="11886.580000" State="Running" SubmissionTime="1128451812" SuspendDuration="0" User="smith"></job> </queue> <queue count="1" option="eligible"> <job EEDuration="1128451812" Group="jacksond" JobID="customer.35" QOS="bug" ReqAWDuration="3600" ReqProcs="1" StartPriority="1" StartTime="0" State="Idle" SubmissionTime="1128451812" SuspendDuration="0" User="johnson"></job> <queue><queue count="0" option="blocked"></queue> </Data>
Common Query/Control Services
- jobs
- query status - mdiag -j (XML details)
- submit - msub (XML format)
- cancel - mjobctl -c
- nodes
- query status - mdiag -n (XML details)
- create resource reservation - mrsvctl -c
- destroy resource reservation - mrsvctl -r
H.2 Resource Management Interfaces
Moab can monitor, schedule, and control services and resources using multiple protocols. These protocols include the following:- HTTP
- LDAP
- SQL
- script/flat file
- Resource Manager Specific Interfaces - LSF, SGE, TORQUE, PBSPro, Loadleveler, and so forth
Using the resource manager interfaces, Moab can do the following:
- monitor resources (compute host, network, storage, and software license based resources)
- load configuration, architecture, and feature information
- load state, utilization, and workload information
- load policy and ownership information
- manage resources
- dynamically reconfigure and reprovision resource hardware (processors, memory, etc.)
- dynamically reconfigure and reprovision resource software (operating system, application software, filesystem mounts, etc.)
- dynamically reconfigure and reprovision resource security (VPN's, VLAN's, host security, etc.)
- monitor workload (batch jobs, interactive jobs, persistent services, dynamic services, distributed services)
- load state, resource requirement, and required environment information
- load user, group, and credential information
- load utilization, resource allocation, and policy information
- manage workload
- migrate jobs from one resource to another (intra-cluster and inter-cluster)
- modify jobs for translation and optimization purposes
- suspend, resume, checkpoint, restart, and cancel jobs
- query cluster policies and configuration
H.3 Identity and Credential Management Interfaces
Moab's identity and credential management interfaces allow Moab to exchange credential and user configuration, access, policy, and usage information.
H.4 Accounting Interfaces
Moab accounting interfaces allow Moab to export local utilization statistics, events, and accounting information to site specific scripts.
H.5 Grid Interfaces
Moab provides interfaces to allow interaction with various grid brokers and services. These interfaces allow Moab to provide services as well as utilize services.
Services Utilized
- Information Services (import and utilize information service data in making scheduling decisions)
- Job Migration
- Data Migration
- Credential Mapping
- Security and Delegation
See Moab Workload Manager for Grids for more information on utilized services.
Services Provided
- Information Services (provide resource, workload, and credential information)
- Job Migration
- Data Migration
- Credential Mapping
See Moab Workload Manager for Grids for more information on provided services.
H.6 Discovery/Directory Services
Moab can import and export key event information regarding workload, cluster resources, cluster and grid services, and other components of hardware and software infrastructure.
By default, these clients communicate with the scheduler using the U.S. Department of Energy (DOE) Scalable Systems Software socket and wire protocols. These protocols are largely HTML- and XML-based, using PKI, 3DES, MD5, challenge, and other security protocols and are documented within the SSS project pages.
As part of this initiative, the scheduler/client protocol has been extended to support multiple socket level protocol standards in communicating with its clients and peer services. These include SingleUseTCP, SSS-HALF, and HTTP. The client socket protocol can be specified by setting the MCSOCKETPROTOCOL parameter to SUTCP, SSS-HALF, or HTTP. Further protocols are being defined and standardized over time and backwards compatibility will be maintained. Documentation on the SSS-HALF implementation can be found within the DOE's SSS Project Notebooks.
H.7 Job Submission and Management Interface
Moab provides interfaces to enable the following services:
- Resource Availability Query
- Determine quantity, state, and configuration of configured resources (idle, busy, and down nodes)
- Determine quantity and configuration of all available resources (idle nodes)
- Determine resources available subject now and in the future for potential job
- Determine best target cluster destination for potential job
- Determine largest/longest job which could start immediately
- Determine estimated start time for potential job
- Determine earliest guaranteed start time for potential job
- Reserve Resources
- Reserve specific resources for desired time frame
- Submit Job (XML format)
- Submit job to specific cluster
- Submit job to global job queue
- Manage Job
- Hold job
- Adjust job priority
- Modify job executable, args, data requirements, job dependencies, duration, hostcount, or other attributes
- Suspend/resume job
- Checkpoint/requeue job
- Cancel job
- Migrate job
- Adjust job quality of service (QoS)
- Query Job
- Determine job state, utilization, or output results for idle, active, or completed job
- Determine estimated start time
- Determine guaranteed start time