(Quick Reference)

1 Introduction

1 Introduction

1.1 Moab® Web Services Overview

Moab Web Services (MWS) is a component of Adaptive Computing Suites that enables programmatic interaction with Moab Workload Manager via a RESTful interface. MWS allows you to create and interact with jobs, nodes, virtual machines, reservations, and more! MWS is the preferred method for those wishing to create custom user interfaces for Moab and is the primary method by which Moab Viewpoint communicates with Moab.

MWS communicates with the Moab Workload Manager (MWM) server using the same wire protocol as the Moab command-line interface. By publishing a standard interface into Moab's intelligence, MWS significantly reduces the amount of work required to integrate MWM into your solution.

This documentation is intended for developers performing such integrations. If you are a Moab administrator, and for conceptual information about MWM, see the Moab Administrator's Guide.

1.2 Installation Guide

These instructions describe how to deploy the Moab® Web Services (MWS) to a Tomcat server.

1.2.1 Requirements

The minimum requirements to run Moab Web Services are as follows:
  • Dual core processor
  • At least 4 GB of RAM

The following software is required to run Moab Web Services:

  • Moab® Workload Manager 7.0
  • Oracle®/Sun® Java® 6 Runtime Environment
  • Apache Tomcat™ 6
  • MongoDB® 2.0.1 or later

1.2.2 Quickstart Guide

Moab Web Services (MWS) must be deployed on the same server as the Moab Workload Manager (MWM).
  • Install and configure the Moab Workload Manager as desired.
  • Generate a secret key to be used for communication between MWS and MWS. Below is a suggested script which can be used to generate the key.

#!/usr/bin/perl
use MIME::Base64;
open(FILE, "/dev/urandom") or die "$!";
read(FILE, $buf, 18) or die "$!";
print encode_base64($buf);
  • Set the secret key for MWM by creating the .moab.key file.

rm -f /opt/moab/.moab.key /opt/moab/etc/.moab.key
# DO NOT USE THIS KEY. This is just an example. Use your generated key here.
echo 33e1e4e86a5efd116893da8efe > /opt/moab/.moab.key
chown root /opt/moab/.moab.key
chmod 400 /opt/moab/.moab.key
ln /opt/moab/.moab.key /opt/moab/etc/.moab.key
  • Start MWM.
  • Install MongoDB version 2.0.1 or later on the MWM server using packages or manual installation.
    • Packages (recommended) are available from the Mongo-provided repositories for CentOS and Fedora or Ubuntu and Debian.
    • After installing the packages, start MongoDB and arrange for it to start automatically during server startup.
      • Here is an example for CentOS and Fedora:
        • /sbin/chkconfig mongod on
        • /etc/init.d/mongod start
      • For Ubuntu and Debian, see the Configuration section of this page for examples.
    • For manual installation (not recommended), follow these instructions. Be sure to start the Mongo server (mongod) after installation and arrange for it to start automatically during server startup (by writing and registering an /etc/init.d script, for example).
  • Install and enable the 64-bit version of Oracle/Sun Java 6. Example:

sh jre-6u<BUILD>-linux-x64-rpm.bin
rm -f /usr/bin/java
ln -s /etc/alternatives/java /usr/bin/java
/usr/sbin/alternatives --install /usr/bin/java java /usr/java/latest/bin/java 1
/usr/sbin/alternatives --set java /usr/java/latest/bin/java
    • The Java installation may be verified by running the following command:

java -version
(It should mention "Java(TM) SE Runtime Environment" and a build number like build 1.6.0_31-b04.)
  • Choose or create a directory to hold the MWS configuration and other information. For these instructions, we will use /opt/mws. This is also called the MWS home directory.
  • Verify that the home directory contains the following sub-directories and that the Tomcat user has write access to hooks and plugins:
    • etc
    • hooks - Required only if utilizing Hooks.
    • plugins - Required only if utilizing MWS Plugins.
  • Extract the contents of the MWS download tarball into a temporary directory and move into the extracted directory.

mkdir mws-install
cd mws-install
cp /tmp/mws-<VERSION>.tar.gz .
tar xvzf mws-<VERSION>.tar.gz
cd mws-<VERSION>
  • Copy the sample configuration file mws-config.groovy located in the extracted MWS directory to the MWS home etc sub-directory. The contents of this file are similar to the following example:

Be sure to modify the moab.secretKey setting to match the secure key generated and set for MWM above.
/opt/mws/etc/mws-config.groovy
// DO NOT USE THIS KEY. This is just an example. Use your generated key here.
moab.secretKey = "33e1e4e86a5efd116893da8efe"
moab.server = "localhost"
moab.port = 42559

// Change these to be whatever you like, but note that username and // password must be different from each other. If they are the same, // MWS will not start. auth.defaultUser.username = "admin" auth.defaultUser.password = "secret"

// These are the default settings for converting generic (untyped) pending // actions to the more specific action types based on the template name. This // mapping is given as a reference and should not be changed or uncommented. /* mws.pendingActions.mappings = [ "genericVM-setup":"VMCREATE", "genericVM-migrate":"VMMIGRATE", "genericVM-destroy":"VMDESTROY", "OSStorage-setup":"VMSTORAGE", "extraStorage-setup":"VMSTORAGE", "genericPM-setup":"OSPROVISION", ] */

// Below is an example of how to set a custom pending action type based // on the template name. // mws.pendingActions.mappings["customVMTemplate-action"] = "vmcreate"

// By default, generic pending actions are returned (along with typed // pending actions). To prevent this, uncomment the line below. // mws.pendingActions.displayGeneric = false

log4j = { appenders { rollingFile name: 'stacktrace', file: '/var/log/mws/stacktrace.log', maxFileSize: '1GB' rollingFile name: 'rootLog', file: '/var/log/mws/mws.log', // NOTE: This threshold may be changed to log more information (ie DEBUG) threshold: org.apache.log4j.Level.ERROR, maxFileSize: '1GB', maxBackupIndex:10, //Retain only the 10 most recent log files (saves file space) layout:pattern(conversionPattern: '%d %p %c %m%n') //Configure output format of log entries }

// NOTE: This definition is a catch-all for any logger not defined below root { error 'rootLog' }

// Individual logger configurations debug 'com.ace.mws', 'grails.app.bootstrap.BootStrap', 'grails.app.controller.com.ace.mws', 'grails.app.domain.com.ace.mws', 'grails.app.filters.com.ace.mws', 'grails.app.service.com.ace.mws', 'grails.app.tagLib.com.ace.mws', 'grails.app.gapiParser', 'grails.app.gapiRequest', 'grails.app.gapiSerializer', 'grails.app.nativeTranslator', 'plugins' // MWS plugins

info 'com.ace.moab.gapi.Connection', 'com.ace.moab.gapi.parsers', 'grails.app.service.grails.plugins.reloadconfig', 'com.ace.moab.gapi.serializers'

off 'org.codehaus.groovy.grails.web.errors' }

  • Define MWS_HOME in your Tomcat CATALINA_OPTS to be the top-level directory created or chosen. Example: CATALINA_OPTS="-DMWS_HOME=/opt/mws"

Where to store CATALINA_OPTS depends on various factors including operating system and sysadmin preference. Here are some suggestions:
  • CentOS™ 5: /etc/tomcat6/tomcat6.conf
  • Red Hat® Enterprise Linux 5: /etc/tomcat6/tomcat6.conf
  • SUSE® Linux Enterprise Server 11: /etc/tomcat6/tomcat6.conf
  • Ubuntu® 10.04: /usr/share/tomcat6/bin/setenv.sh

MWS looks for mws-config.groovy in these locations, in this order:
  • MWS_HOME/etc/mws-config.groovy
  • MWS_HOME/mws-config.groovy
  • /opt/mws/etc/mws-config.groovy
  • /opt/mws/mws-config.groovy
  • MOABHOMEDIR/etc/mws-config.groovy
  • MOABHOMEDIR/mws-config.groovy
  • /opt/moab/etc/mws-config.groovy
  • /opt/moab/mws-config.groovy

  • Run these commands as the root user:

mkdir /var/log/mws
chown -R tomcat /var/log/mws
The chown command line might need to be different from the above. The goal is to grant ownership of /var/log/mws to the ID of the user that is running Tomcat.
  • Start Tomcat and then deploy mws.war using whatever deployment method you like. A typical approach is to copy the war file to Tomcat's webapps directory.
  • If all went well, MWS will now be running at http://localhost:8080/mws/. If you visit this page with a browser, you'll see some sample queries and a few other actions.
  • If problems are encountered or the application does not seem to be running, see the steps below in the Troubleshooting section.

1.2.3 Troubleshooting Installation

The recommended course of determining errors occurring in MWS is to look in the following files:
  1. The MWS log file. This is located as configured in mws-config.groovy or in a temporary directory.
  2. The application container's log files. For Tomcat, this is the catalina.out log, typically in $CATALINA_HOME/logs/catalina.out.

These files will help to find the cause of errors. The following is a list of typical errors that may occur and their fixes:

MongoDB Errors

If the application fails to start and gives error messages such as these:

Error creating bean with name 'mongoDatastore'
can't say something; nested exception is com.mongodb.MongoException
MongoDB is most likely not running, or the host and port are misconfigured. Start MongoDB or reconfigure MWS and restart MWS.

Java Out of Memory Errors

Increase the size of the Java Heap by modifying CATALINA_OPTS. Example:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx1024m"
Xms is the minimum size of the heap, and Xmx is the maximum size of the heap. If you still have Out of Memory errors, increase Xmx.

32-bit Systems/PermGen Size

The default PermGen size is too small for 32-bit systems. Include the following option in CATALINA_OPTS:

-XX:MaxPermSize=128m

This will make the CATALINA_OPTS look like the following:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx1024m -XX:MaxPermSize=128m"

1.3 Configuration

This section describes where Moab Web Services searches for its configuration files. It also shows some examples of how to configure logging.

To see a full reference to all configuration and logging parameters available in MWS, see the Configuration page under Moab Web Services in the Quick Reference menu.

Home Directory

The MWS home directory contains all configuration as well as other files that serve features of MWS such as hooks and plugins. This is typically set by using the MWS_HOME property as explained in the Quickstart Guide. If MWS_HOME is not set as a Java property or as an environment variable for the current application container (i.e. Tomcat), /opt/mws will be used as the default MWS_HOME. If no configuration files are found in MWS_HOME, MOABHOMEDIR will be used. If this property also does not exist, the home directory will default to /opt/moab.

MWS_HOME or MOABHOMEDIR can be set either as a Java property or as an environment variable. See the Quickstart Guide for suggestions on how to set MWS_HOME.

The home directory consists of several sub-directories:

  • etc - Used for storing configuration files.
  • hooks - Used for storing hook files. This is not required if hooks are not being used.
  • plugins - Used for storing plugin types. This is not required if custom plugin types are not being used.

The hooks and plugins directories should be writable by the application container's user, such as the tomcat user.

Configuration File Locations

MWS searches the following directories for configuration files in the order shown below. As soon as a configuration file is found in one of these directories, that file is loaded and searching stops. If a log4j.properties file exists in the same directory, it will be loaded as well.

  • MWS_HOME/etc
  • MWS_HOME
  • /opt/mws/etc
  • /opt/mws
  • MOABHOMEDIR/etc
  • MOABHOMEDIR
  • /opt/moab/etc
  • /opt/moab

  • In each directory, MWS looks first for mws-config.groovy and then for mws-config.properties. If it finds mws-config.groovy, it does not look for mws-config.properties.
  • mws-config.groovy uses a style that is similar to a Java properties file with some extensions from Groovy.
  • mws-config.properties is a regular Java properties file.

Logging Configuration Using mws-config.groovy

Shown below is an example that logs all error messages and fatal messages to /var/log/mws/mws.log. It also logs all stack traces to /var/log/mws/stacktrace.log.

Minimal Logging Configuration
log4j = {
	appenders {
		rollingFile name: 'stacktrace',
			file: '/var/log/mws/stacktrace.log',
			maxFileSize: '1GB'
		rollingFile name: 'rootLog',
			file: '/var/log/mws/mws.log',
			threshold: org.apache.log4j.Level.ERROR,
			maxFileSize: '1GB'
	}
	root {
		debug 'rootLog'
	}
}

Alternatively, you may configure a console appender instead of a rolling file as shown below.

Console Logging Configuration
log4j = {
	appenders {
		rollingFile name: 'stacktrace',
			file: '/var/log/mws/stacktrace.log',
			maxFileSize: '1GB'
		console name: 'consoleLog',
			threshold: org.apache.log4j.Level.ERROR
	}
	root {
		debug 'consoleLog'
	}
}

  • For the examples above, you must make sure that /var/log/mws exists and is writable by the application server.
  • You may configure logging using either mws-config.groovy or a regular log4j.properties file. The log4j.properties file must be in the same directory as the mws-config.groovy file.
  • If you do not define any log4j configuration, MWS will write its log files to java.io.tmpdir. For Tomcat, java.io.tmpdir is generally set to $CATALINA_BASE/temp.

For all possible configuration options, see the Configuration section in the reference guide.

1.4 Security

When running MWS in production environments, security is a major concern. This section focuses on securing the three kinds of connections with MWS:
  1. The connection between MWS and Moab Workload Manager (MWM)
  2. The connection between MWS and MongoDB
  3. The connections between clients and MWS

Connection with MWM

MWS communicates with MWM via the Moab Wire Protocol, which uses a direct connection between the two applications. The communication over this connection uses a shared secret key, which is discussed in the Quickstart Guide. However, the communication is not encrypted and is therefore susceptible to eavesdropping and replay attacks. For this reason, MWS is supported only when running on the same machine as MWM. This assures that any connections between the two applications occur internally on the server and are not exposed to external users.

Connection with MongoDB

By default, the connection between MWS and MongoDB is not authenticated. To enable authentication between them, see the instructions below.

  • MWS Configuration: see the Configuration reference guide for information on the grails.mongo properties to set in mws-config.groovy.
  • MongoDB Configuration: see the MongoDB Security and Authentication guide. Generally, the following steps are required:
    • Add an administrative user to MongoDB in the admin database.
    • Start MongoDB with authentication activated (using the --auth command-line option for example).
    • Log in as the administrative user to the admin database.
    • Add a user for MWS to use with full read and write access to the database specified in the configuration file (mws by default).
    • Change the proper configuration file properties with the created username and password.
    • Restart MWS by restarting the servlet container (Tomcat).

If authentication is activated on MongoDB, but the user was not properly created or configured with MWS, MWS will not start. See the log file(s) for additional information in this case.

Client Connections to MWS

All connections to MWS, except those requesting the documentation or the main page, must be authenticated properly. MWS uses a single-trusted-user authentication model, meaning a single user exists that has access to all aspects of MWS. The username and password for this user are configured with the auth.defaultUser properties in the configuration file. See the Configuration reference guide for more information.

When using the MWS user interface in a browser, the user will be prompted for username and password. For information on how to authenticate requests when not using a browser, see the API Security section in the user guide.

The username and password in the Basic Authentication header are encoded but not encrypted. Therefore, it is strongly recommended that MWS be run behind a proxy (like Apache) with SSL enabled. Another approach would be to enable SSL on the servlet container on which MWS is deployed.

1.5 Version and Build Information

To get detailed version information about MWS, use one of the following three methods:

Browser

Using a browser, visit the MWS home page (for example, http://localhost:8080/mws/). At the bottom of the page is the MWS version information. See the screenshot below:

REST Request

Using a REST client or other HTTP client software, send a GET request to the rest/diag/about resource. Here is an example:

curl -u username:password http://localhost:8080/mws/rest/diag/about

This resource is also described under Diagnostics.

MANIFEST.MF File

If MWS fails to start, version and build information can be found in the META-INF/MANIFEST.MF file inside the MWS WAR file. The version properties begin with Implementation. Below is an excerpt of a MANIFEST.MF file:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 20.4-b02 (Sun Microsystems Inc.)
Bundle-ManifestVersion: 2
Bundle-Name: mws
Bundle-SymbolicName: mws
Bundle-Version: 7.0.0

...

Implementation-Build: 4 Implementation-Build-Date: 2012-02-07_17-01-39 Implementation-Revision: 9e109b9a4289800a2c985082d7595d759807aca9

Name: Grails Application Implementation-Title: mws Implementation-Version: 7.0.0 Grails-Version: 1.3.7