(Click to open topic with navigation)

5.104 Managing Resources Directly with the Native Interface

5.104.1 Native Interface Overview

The Native interface allows a site to augment or even fully replace a resource manager for managing resources. In some situations, the full capabilities of the resource manager are not needed and a lower cost or lower overhead alternative is preferred. In other cases, the nature of the environment may make use of a resource manager impossible due to lack of support. Still, in other situations it is desirable to provide information about additional resource attributes, constraints, or state from alternate sources.

In any case, Moab provides the ability to directly query and manage resources along side of or without the use of a resource manager. This interface, called the NATIVE interface can also be used to launch, cancel, and otherwise manage jobs. This NATIVE interface offers several advantages including the following:

However, the NATIVE interface may also have some drawbacks.

At a high level, the native interface works by launching threaded calls to perform standard resource manager activities such as managing resources and jobs. The desired calls are configured within Moab and used whenever an action or updated information is required.

5.104.2 Configuring the Native Interface

Using the native interface consists of defining the interface type and location. As mentioned earlier, a single object may be fully defined by multiple interfaces simultaneously with each interface updating a particular aspect of the object.

5.104.2.A Configuring the Resource Manager

The Native resource manager must be configured using the RMCFG parameter. To specify the native interface, the TYPE attribute must be set to NATIVE.

RMCFG[local] TYPE=NATIVE
RMCFG[local] CLUSTERQUERYURL=exec:///tmp/query.sh

5.104.2.B Reporting Resources

To indicate the source of the resource information, the CLUSTERQUERYURL attribute of the RMCFG parameter should be specified. This attribute is specified as a URL where the protocols FILE, EXEC and SQL are allowed. If a protocol is not specified, the protocol EXEC is assumed.

Format Description
EXEC Execute the script specified by the URL path. Use the script stdout as data.
FILE Load the file specified by the URL path. Use the file contents as data.
SQL Load data directly from an SQL database using the FULL format described below.

Moab considers a NativeRM script to have failed if it returns with a non-zero exit code or if the CHILDSTDERRCHECK parameter is set and its appropriate conditions are met. In addition, the NativeRM script associated with a job submit URL will be considered as having failed if its standard output stream contains the text ERROR.

This simple example queries a file on the server for information about every node in the cluster. This differs from Moab remotely querying the status of each node individually. 

RMCFG[local]    TYPE=NATIVE
RMCFG[local]    CLUSTERQUERYURL=file:///tmp/query.txt

5.104.3 Generating Cluster Query Data

5.104.3.A Flat Cluster Query Data

If the EXEC or FILE protocol is specified in the CLUSTERQUERYURL attribute, the data should provide flat text strings indicating the state and attributes of the node. The format follows the Moab Resource Manager Language Data Format where attributes are delimited by white space rather than ';' (See W.1.1 Query Resources Data Format):

Describes any set of node attributes with format: <NAME><ATTR>=<VAL> [<ATTR>=<VAL>]...

<NAME> Name of node
<ATTR> Node attribute
<VAL> Value of node attribute 
n17 CPROC=4 AMEMORY=100980 STATE=idle

5.104.4 Interfacing to FLEXlm

Moab can interface with FLEXlm to provide scheduling based on License Management availability. Informing Moab of license dependencies can reduce the number of costly licenses required by your cluster by allowing Moab to intelligently schedule around license limitations.

Provided with Moab in the tools directory is a Perl script, license.mon.flexLM.pl. This script queries a FLEXlm license server and gathers data about available licenses. This script then formats this data for Moab to read through a native interface. This script can easily be used by any site to help facilitate FLEXlm integration--the only modification necessary to the script is setting the @FLEXlmCmd to specify the local command to query FLEXlm. To make this change, edit license.mon.flexLM.pl and, near the top of the file, look for the line:

my @FLEXlmCmd = ("SETME");

Set the @FLEXlmCmd to the appropriate value for your system to query a license server and license file (if applicable). If lmutil is not in the PATH variable, specify its full path. Using the lmutil -a argument will cause it to report all licenses. The -c option can be used to specify an optional license file.

To test this script, run it manually. If working correctly, it will produce output similar to the following:

> ./license.mon.flexLM.pl
GLOBAL UPDATETIME=1104688300 STATE=idle ARES=autoCAD:130,idl_mpeg:160 CRES=autoCAD:200,idl_mpeg:330

If the output looks incorrect, set the $LOGLEVEL variable inside of license.mon.flexLM.pl, run it again, and address the reported failure.

Once the license interface script is properly configured, the next step is to add a license native resource manager to Moab via the moab.cfg file:

RMCFG[FLEXlm]   TYPE=NATIVE RESOURCETYPE=LICENSE
RMCFG[FLEXlm]   CLUSTERQUERYURL=exec://$TOOLSDIR/flexlm/license.mon.flexLM.pl
...

Once this change is made, restart Moab. The command mdiag -R can be used to verify that the resource manager is properly configured and is in the state Active. Detailed information regarding configured and utilized licenses can be viewed by issuing the mdiag -n. Floating licenses (non-node-locked) will be reported as belonging to the GLOBAL node.

Due to the inherent conflict with the plus sign (+), the provided license manager script replaces occurrences of the plus sign in license names with the underscore symbol (_). This replacement requires that licenses with a plus sign in their names be requested with an underscore in place of any plus signs.

5.104.4.A Interfacing to Multiple License Managers Simultaneously

If multiple license managers are used within a cluster, Moab can interface to each of them to obtain the needed license information. In the case of FLEXlm, this can be done by making one copy of the license.mon.flexLM.pl script for each license manager and configuring each copy to point to a different license manager. Then, within Moab, create one native resource manager interface for each license manager and point it to the corresponding script as in the following example:

RMCFG[FLEXlm1]   TYPE=NATIVE RESOURCETYPE=LICENSE
RMCFG[FLEXlm1]   CLUSTERQUERYURL=exec://$TOOLSDIR/flexlm/license.mon.flexLM1.pl
RMCFG[FLEXlm2]   TYPE=NATIVE RESOURCETYPE=LICENSE
RMCFG[FLEXlm2]   CLUSTERQUERYURL=exec://$TOOLSDIR/flexlm/license.mon.flexLM2.pl
RMCFG[FLEXlm3]   TYPE=NATIVE RESOURCETYPE=LICENSE
RMCFG[FLEXlm3]   CLUSTERQUERYURL=exec://$TOOLSDIR/flexlm/license.mon.flexLM3.pl
...

For an overview of license management, including job submission syntax, see License Management.

It may be necessary to increase the default limit, MMAX_GRES. See Adjusting Default Limits for more implementation details.

5.104.5 Interfacing to Nagios

Moab can interface with Nagios to provide scheduling based on network hosts and services availability.

Nagios installation and configuration documentation can be found at Nagios.org.

Provided with Moab in the tools directory is a Perl script, node.query.nagios.pl. This script reads the Nagios status.dat file and gathers data about network hosts and services. This script then formats data for Moab to read through a native interface. This script can be used by any site to help facilitate Nagios integration. To customize the data that will be formatted for Moab, make the changes in this script.

You may need to customize the associated configuration file in the etc directory, config.nagios.pl. The statusFile line in this script tells Moab where the Nagios status.dat file is located. Make sure that the path name specified is correct for your site. Note that the interval which Nagios updates the Nagios status.dat file is specified in the Nagios nagios.cfg file. Refer to Nagios documentation for further details.

To make these changes, familiarize yourself with the format of the Nagios status.dat file and make the appropriate additions to the script to include the desired Moab RM language (formerly WIKI) Interface attributes in the Moab output.

To test this script, run it manually. If working correctly, it will produce output similar to the following:

> ./node.query.nagios.pl
gateway STATE=Running
localhost STATE=Running CPULOAD=1.22 ADISK=75332

Once the Nagios interface script is properly configured, the next step is to add a Nagios native resource manager to Moab via the moab.cfg file:

RMCFG[nagios] TYPE=NATIVE
RMCFG[nagios] CLUSTERQUERYURL=exec://$TOOLSDIR/node.query.nagios.pl
...

Once this change is made, restart Moab. The command mdiag -R can be used to verify that the resource manager is properly configured and is in the state Active. Detailed information regarding configured Nagios node information can be viewed by issuing the mdiag -n.

> mdiag -n -v
compute node summary
Name                    State   Procs      Memory         Disk          Swap      Speed   Opsys   Arch Par   Load Rsv Classes                        Network                        Features              
gateway               Running    0:0         0:0           0:0           0:0       1.00       -      - dav   0.00   0 -                              -                              -                   
  WARNING:  node 'gateway' is busy/running but not assigned to an active job
  WARNING:  node 'gateway' has no configured processors
localhost             Running    0:0         0:0       75343:75347       0:0       1.00       -      - dav   0.48   0 -                              -                              -                   
  WARNING:  node 'localhost' is busy/running but not assigned to an active job
  WARNING:  node 'localhost' has no configured processors
-----                     ---    3:8      1956:1956    75345:75349    5309:6273  
Total Nodes: 2  (Active: 2  Idle: 0  Down: 0)

5.104.6 Configuring Resource Types

Native Resource managers can also perform special tasks when they are given a specific resource type. These types are specified using the RESOURCETYPE attribute of the RMCFG parameter.

Type Description
COMPUTE Normal compute resources (no special handling)
FS File system resource manager (see Utilizing Multiple Resource Managers for an example)
LICENSE

Software license manager (see Interfacing to FLEXlm and License Management)

 

If LICENSE is specified, all generic resources reported by the resource manager will be marked as a license; causing them to be tracked by the accounting manager under the Licenses property. See Licenses for more information on the Licenses property.
NETWORK Network resource manager
PROV Provisioning resource manager. This is the RM Moab uses to modify the OS of a node (not a VM) and to power a node on or off.

5.104.7 Creating New Tools to Manage the Cluster

Using the scripts found in the $TOOLSDIR ($INSTDIR/tools) directory as a template, new tools can be quickly created to monitor or manage most any resource. Each tool should be associated with a particular resource manager service and specified using one of the following resource manager URL attributes.

CLUSTERQUERYURL
Description Queries resource state, configuration, and utilization information for compute nodes, networks, storage systems, software licenses, and other resources. For more details, see CLUSTERQUERYURL.
Output Node status and configuration for one or more nodes. See W.1.1 Query Resources Data Format.
Example 
RMCFG[v-stor] CLUSTERQUERYURL=exec://$HOME/storquery.pl

Moab will execute the storquery.pl script to obtain information about 'v-stor' resources.
JOBCANCELURL
Description Cancels a job.
Input <JOBID>
Example 
RMCFG[v-stor] JOBCANCELURL=exec://$HOME/cancel.pl

Moab will execute the cancel.pl script to cancel jobs.
JOBMODIFYURL
Description Modifies a job or application. For more details, see JOBMODIFYURL.
Input [-j <JOBEXPR>] [--s[et]|--u[nset]|--c[lear]|--i[ncrement]|--d[ecrement]] <ATTR>[=<VALUE>] [<ATTR>[=<VALUE>]]...
Example 
RMCFG[v-stor] JOBMODIFYURL=exec://$HOME/jobmodify.pl

Moab will execute the jobmodify.pl script to modify the specified job.
JOBREQUEUEURL
Description Requeues a job.
Input <JOBID>
Example 
RMCFG[v-stor] JOBREQUEUEURL=exec://$HOME/requeue.pl

Moab will execute the requeue.pl script to requeue jobs.
JOBRESUMEURL
Description Resumes a suspended job or application.
Input <JOBID>
Example 
RMCFG[v-stor] JOBRESUMEURL=exec://$HOME/jobresume.pl

Moab will execute the jobresume.pl script to resume suspended jobs.
JOBSTARTURL
Description Launches a job or application on a specified set of resources.
Input <JOBID><TASKLIST><USERNAME> [ARCH=<ARCH>] [OS=<OPSYS>] [IDATA=<STAGEINFILEPATH>[,<STAGEINFILEPATH>]...] [EXEC=<EXECUTABLEPATH>]
Example 
RMCFG[v-stor] JOBSTARTURL=exec://$HOME/jobstart.pl

Moab will execute the jobstart.pl script to execute jobs.
JOBSUBMITURL
Description Submits a job to the resource manager, but it does not execute the job. The job executes when the JOBSTARTURL is called.
Input [ACCOUNT=<ACCOUNT>] [ERROR=<ERROR>] [GATTR=<GATTR>] [GNAME=<GNAME>] [GRES=<GRES>:<Value>[,<GRES>:<Value>]*] [HOSTLIST=<HOSTLIST>] [INPUT=<INPUT>] [IWD=<IWD>] [NAME=<NAME>] [OUTPUT=<OUTPUT>] [RCLASS=<RCLASS>] [REQUEST=<REQUEST>] [RFEATURES=<RFEATURES>] [RMFLAGS=<RMFLAGS>] [SHELL=<SHELL>] [TASKLIST=<TASKLIST>] [TASKS=<TASKS>] [TEMPLATE=<TEMPLATE>] [UNAME=<UNAME>] [VARIABLE=<VARIABLE>] [WCLIMIT=<WCLIMIT>] [ARGS=<Value>[ <Value>]*]

ARGS must be the last submitted attribute because there can be multiple space-separated values for ARGS.
Example 
RMCFG[v-stor] JOBSUBMITURL=exec://$HOME/jobsubmit.pl

Moab submits the job to the jobsubmit.pl script for future job execution.
JOBSUSPENDURL
Description Suspends in memory an active job or application.
Input <JOBID>
Example 
RMCFG[v-stor] JOBSUSPENDURL=exec://$HOME/jobsuspend.pl

Moab will execute the jobsuspend.pl script to suspend active jobs.
NODEMODIFYURL
Description

Provide method to dynamically modify/provision compute resources including operating system, applications, queues, node features, power states, etc.

Moab blocks scheduling when invoking the script defined by NODEMODIFYURL , If your NODEMODIFYURL script is a long running process (i.e. > 30 seconds or, if defined, longer than the RMCFG TIMEOUT setting), then you must background the process and return quickly.
Input <NODEID>[,<NODEID>] [--force] {--set <ATTR>=<VAL>|--clear <ATTR>}
ATTR is one of the node attributes listed in W.1.1 Query Resources Data Format
Example 
RMCFG[warewulf] NODEMODIFYURL=exec://$HOME/provision.pl

Moab will reprovision compute nodes using the provision.plscript.
NODEPOWERURL
Description Allows Moab to issue IPMI power commands.
Input <NODEID>[,<NODEID>] ON | OFF
Example 
RMCFG[node17rm] NODEPOWERURL=exec://$TOOLSDIR/ipmi.power.pl

Moab will issue a power command contained in the ipmi.power.plscript.
SYSTEMMODIFYURL
Description Provide method to dynamically modify aspects of the compute environment which are directly associated with cluster resources. For more details, see SYSTEMMODIFYURL.
SYSTEMQUERYURL
Description Provide method to dynamically query aspects of the compute environment which are directly associated with cluster resources. For more details, see SYSTEMQUERYURL.
Input default <ATTR>
ATTR is one of images
Output <STRING>
Example 
RMCFG[warewulf] SYSTEMQUERYURL=exec://$HOME/checkimage.pl

Moab will load the list of images available from warewulf using the checkimage.pl script.
WORKLOADQUERYURL
Description: Provide method to dynamically query the system workload (jobs, services, etc.) of the compute environment which are associated with managed resources.

Job/workload information should be reported back from the URL (script, file, web service, etc.) using the Moab Resource Manager Language Data Format (formerly WIKI).

For more details, see WORKLOADQUERYURL.
Output: <STRING>
Example: 
RMCFG[xt] WORKLOADQUERYURL=exec://$HOME/job.query.xt3.pl

Moab will load job/workload information by executing the job.query.xt3.pl script.

Related Topics 

© 2017 Adaptive Computing