Table of Contents

1 Introduction

1.1 Moab® Web Services Overview

MWS communicates with the Moab Workload Manager (MWM) server using the same wire protocol as the Moab command-line interface. By publishing a standard interface into Moab's intelligence, MWS significantly reduces the amount of work required to integrate MWM into your solution.

This documentation is intended for developers performing such integrations. If you are a Moab administrator, and for conceptual information about MWM, see the Moab Administrator's Guide.

1.2 Installation Guide

1.2.1 Requirements

Hardware Requirements

• 64-bit dual-core processor
• At least 4 GB of RAM

Software Requirements

• Moab® Workload Manager (version must match exactly the version of MWS)
• Oracle® Java® 6 Runtime Environment
• Apache Tomcat™ 6
• MongoDB® 2.0.x, where x is 2 or greater

Oracle Java 6 Runtime Environment is the only supported Java environment.

All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.

1.2.2 Quickstart Guide

1) Install MongoDB version 2.0.x, where x is 2 or greater.

MWS does not yet support MongoDB 2.2.x. Be sure to install the 2.0.x packages. As of this writing, the RPM package names are mongo20-10gen-2.0.7-mongodb_1.x86_64.rpm and mongo20-10gen-server-2.0.7-mongodb_1.x86_64.rpm. The Ubuntu package name is mongodb20-10gen_2.0.7_amd64.deb.

• Install MongoDB on RedHat Enterprise, CentOS, or Fedora Linux
• Install MongoDB on Debian or Ubuntu Linux

2) Start MongoDB.

# CentOS 6 example
chkconfig mongod on
service mongod start

The instructions provided above for installing MongoDB describe a base installation only. See the MongoDB section of the security page.

3) Install and configure Moab Workload Manager (MWM).

• You must deploy Moab Web Services (MWS) on the same server as Moab Workload Manager (MWM).
• The version of MWS must match exactly the version of MWM. For example, MWS 7.1.1 works only with MWM 7.1.1.

4) Generate a secret key to be used for communication between MWM and MWS.

# All these steps are required. Do not skip any steps.
service moab stop
dd if=/dev/urandom count=18 bs=1 2>/dev/null | base64 > /opt/moab/etc/.moab.key
chown root /opt/moab/etc/.moab.key
chmod 400 /opt/moab/etc/.moab.key
ln -f /opt/moab/etc/.moab.key /opt/moab/.moab.key
service moab start

5) Install Apache Tomcat 6.

# CentOS 6 example
yum install tomcat6

6) Install the 64-bit version of the Oracle Java SE 6 JRE.

Oracle Java 6 Runtime Environment is the only supported Java environment.

All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.

# CentOS 6 example
sh jre-6u37-linux-x64-rpm.bin
rm -f /usr/bin/java
ln -s /etc/alternatives/java /usr/bin/java
alternatives --install /usr/bin/java java /usr/java/jre1.6.0_37/bin/java 500
alternatives --set java /usr/java/jre1.6.0_37/bin/java

The alternatives command is called update-alternatives on some Linux distributions.

• You can verify the Java installation by running java -version
• The output should look similar to this:

java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

7) Create the MWS home directory and its subdirectories etc, hooks, plugins, and log.

The default location for the MWS home directory is /opt/mws. These instructions assume the default location.

• Give the Tomcat user read access to these directories and write access to the plugins and log directories.
• Here is a sample script for these steps:

mkdir -p /opt/mws/etc /opt/mws/hooks /opt/mws/plugins /opt/mws/log
chown -R tomcat /opt/mws # Depending on your OS, the Tomcat username might be tomcat6.
chmod -R 555 /opt/mws
chmod u+w /opt/mws/plugins /opt/mws/log

8) Extract the contents of the MWS tarball into a temporary directory.

mkdir /tmp/mws-install
cd /tmp/mws-install
tar xvzf $HOME/Downloads/mws-<VERSION>.tar.gz
cd /tmp/mws-install/mws-<VERSION>

9) Set up the MWS configuration file.

• In the extracted MWS directory is a sample configuration file: mws-config.groovy. Copy this file to /opt/mws/etc.
• Give the Tomcat user read access to /opt/mws/etc/mws-config.groovy.
• In the /opt/mws/etc/mws-config.groovy file, change these settings:
• moab.secretKey: needs to match the MWM secret key you generated earlier (contained in /opt/moab/etc/.moab.key)
• auth.defaultUser.username: any value you like, or leave as is
• auth.defaultUser.password: any value you like, but choose a good password

vi /opt/mws/etc/mws-config.groovy
…
moab.secretKey = "<ENTER-KEY-HERE>"
moab.server = "localhost"
moab.port = 42559
// Change these to be whatever you like.
auth.defaultUser.username = "admin"
auth.defaultUser.password = "adminpw"

If you do not change auth.defaultUser.password, then your MWS is not secure, since anyone reading these instructions can log into your MWS. Here are some tips for choosing a good password.

10) Set the following parameters in your Tomcat CATALINA_OPTS.

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"

Where you choose to store CATALINA_OPTS depends on various factors, including operating system and sysadmin preference. Here are some suggestions:

• CentOS™ 5 and 6: /etc/sysconfig/tomcat6
• Red Hat® Enterprise Linux 5 and 6: /etc/tomcat6/tomcat6.conf
• SUSE® Linux Enterprise Server 11: /etc/tomcat6/tomcat6.conf
• Ubuntu® 10.04: /etc/default/tomcat6

11) Start Tomcat and deploy mws.war.

# CentOS 6 example
chkconfig tomcat6 on
service tomcat6 stop
cp /tmp/mws-install/mws-<VERSION>/mws.war /var/lib/tomcat6/webapps
service tomcat6 start

12) Visit http://localhost:8080/mws/ in a web browser to verify that MWS is running.

13) Log into MWS to verify that the MWS credentials are working.

If you encounter problems, or if MWS does not seem to be running, see the steps below in the Troubleshooting section.

1.3 Troubleshooting

1. The MWS log file. By default this is /opt/mws/log/mws.log.
2. The Tomcat catalina.out file, usually in /var/log/tomcat6 or $CATALINA_HOME/logs.

If you remove the log4j configuration from mws-config.groovy, MWS will write its log files to java.io.tmpdir. For Tomcat, java.io.tmpdir is generally set to $CATALINA_BASE/temp or CATALINA_TMPDIR.

Here is a list of some errors and their fixes:

MongoDB Errors

If the application fails to start and gives error messages such as these:

Error creating bean with name 'mongoDatastore'
can't say something; nested exception is com.mongodb.MongoException

Out of semaphores to get db connection

The default number of MongoDB connections allowed per host is 10. To increase this pool size, add grails.mongo.options.connectionsPerHost to your mws-config.groovy. Example:

grails.mongo.options.connectionsPerHost = 50

See also the Configuration page under Moab Web Services in the Quick Reference menu.

java.lang.OutOfMemoryError: Java heap space

Increase the size of the heap using JVM options -Xms and -Xmx. Here are the suggested values from the Quickstart Guide:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"

• -Xms: Set initial Java heap size.
• -Xmx: Set maximum Java heap size.

java.lang.OutOfMemoryError: PermGen space

Increase the size of the permanent generation using JVM option -XX:MaxPermSize. Here are the suggested values from the Quickstart Guide:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"

SEVERE: Context [/mws] startup failed due to previous errors

If catalina.out contains this error, look in /opt/mws/log/mws.log and /opt/mws/log/stacktrace.log for more details on the error.

Moab Reached Maximum Number of Concurrent Client Connections

When this error message is encountered, simply add a new line to the moab.cfg file:

CLIENTMAXCONNECTIONS 256

This will change the Moab configuration when Moab is restarted. Run the following command to immediately use the new setting:

changeparam CLIENTMAXCONNECTIONS 256

The number 256 above may be substituted for the desired maximum number of MWM client connections.

1.4 Configuration

To see a full reference to all configuration and logging parameters available in MWS, see the Configuration page under Moab Web Services in the Quick Reference menu.

Home Directory

The MWS home directory contains all configuration as well as other files that serve features of MWS such as hooks and plugins. This is typically set by using the MWS_HOME property as explained in the Quickstart Guide. If MWS_HOME is not set as a Java property or as an environment variable for the current application container (i.e. Tomcat), /opt/mws will be used as the default MWS_HOME. If no configuration files are found in MWS_HOME, MOABHOMEDIR will be used. If this property also does not exist, the home directory will default to /opt/moab.

MWS_HOME or MOABHOMEDIR can be set either as a Java property or as an environment variable. See the Quickstart Guide for suggestions on how to set MWS_HOME.

The home directory consists of several sub-directories:

• etc - Used for storing configuration files.
• hooks - Used for storing hook files. This is not required if hooks are not being used.
• plugins - Used for storing plugin types. This is not required if custom plugin types are not being used.

The hooks and plugins directories should be writable by the application container's user, such as the tomcat user.

Configuration File Locations

MWS searches the following directories for configuration files in the order shown below. As soon as a configuration file is found in one of these directories, that file is loaded and searching stops. If a log4j.properties file exists in the same directory, it will be loaded as well.

• MWS_HOME/etc
• MWS_HOME
• /opt/mws/etc
• /opt/mws
• MOABHOMEDIR/etc
• MOABHOMEDIR
• /opt/moab/etc
• /opt/moab

• In each directory, MWS looks first for mws-config.groovy and then for mws-config.properties. If it finds mws-config.groovy, it does not look for mws-config.properties.
• mws-config.groovy uses a style that is similar to a Java properties file with some extensions from Groovy.
• mws-config.properties is a regular Java properties file.

Logging Configuration Using mws-config.groovy

Shown below is an example that logs all error messages and fatal messages to /opt/mws/log/mws.log. It also logs all stack traces to /opt/mws/log/stacktrace.log.

log4j = {
	appenders {
		rollingFile name: 'stacktrace',
			file: '/opt/mws/log/stacktrace.log',
			maxFileSize: '1GB'
		rollingFile name: 'rootLog',
			file: '/opt/mws/log/mws.log',
			threshold: org.apache.log4j.Level.ERROR,
			maxFileSize: '1GB'
	}
	root {
		debug 'rootLog'
	}
}

Alternatively, you may configure a console appender instead of a rolling file as shown below.

log4j = {
	appenders {
		rollingFile name: 'stacktrace',
			file: '/opt/mws/log/stacktrace.log',
			maxFileSize: '1GB'
		console name: 'consoleLog',
			threshold: org.apache.log4j.Level.ERROR
	}
	root {
		debug 'consoleLog'
	}
}

• For the examples above, you must make sure that /opt/mws/log exists and is writable by the application server.
• You may configure logging using either mws-config.groovy or a regular log4j.properties file. The log4j.properties file must be in the same directory as the mws-config.groovy file.
• If you do not define any log4j configuration, MWS will write its log files to java.io.tmpdir. For Tomcat, java.io.tmpdir is generally set to $CATALINA_BASE/temp or CATALINA_TMPDIR.

For all possible configuration options, see the Configuration section in the reference guide.

1.5 Security

1. The connection between MWS and Moab Workload Manager (MWM)
2. The connection between MWS and MongoDB
3. The connections between clients and MWS

Connection with MWM

MWS communicates with MWM via the Moab Wire Protocol, which uses a direct connection between the two applications. The communication over this connection uses a shared secret key, which is discussed in the Quickstart Guide. However, the communication is not encrypted and is therefore susceptible to eavesdropping and replay attacks. For this reason, MWS is supported only when running on the same machine as MWM. This assures that any connections between the two applications occur internally on the server and are not exposed to external users.

Connection with MongoDB

By default, the connection between MWS and MongoDB is not authenticated. To enable authentication between them, see the instructions below.

• MWS Configuration: see the Configuration reference guide for information on the grails.mongo properties to set in mws-config.groovy.
• MongoDB Configuration: see the MongoDB Security and Authentication guide. Generally, the following steps are required:
• Add an administrative user to MongoDB in the admin database.
• Start MongoDB with authentication activated (using the --auth command-line option for example).
• Log in as the administrative user to the admin database.
• Add a user for MWS to use with full read and write access to the database specified in the configuration file (mws by default).
• Change the proper configuration file properties with the created username and password.
• Restart MWS by restarting the servlet container (Tomcat).

If authentication is activated on MongoDB, but the user was not properly created or configured with MWS, MWS will not start. See the log file(s) for additional information in this case.

Client Connections to MWS

All connections to MWS, except those requesting the documentation or the main page, must be authenticated properly. MWS uses a single-trusted-user authentication model, meaning a single user exists that has access to all aspects of MWS. The username and password for this user are configured with the auth.defaultUser properties in the configuration file. See the Configuration reference guide for more information.

When using the MWS user interface in a browser, the user will be prompted for username and password. For information on how to authenticate requests when not using a browser, see the API Security section in the user guide.

The username and password in the Basic Authentication header are encoded but not encrypted. Therefore, it is strongly recommended that MWS be run behind a proxy (like Apache) with SSL enabled. The instructions below provide an example of how to do this.

Encrypting Client Connections using Apache and SSL

This section shows how to encrypt client connections to MWS using Apache and SSL. These instructions have been tested on CentOS™ 6.2 with the "Web Server" software set installed. The same ideas are applicable to other operating systems, but the details might be different. As shown in the diagram below, these instructions assume that Tomcat and Apache are running on the same server.

• Create a self-signed certificate. See http://www.openssl.org/docs/HOWTO/certificates.txt for more details if desired.

Instead of creating a self-signed certificate, you can buy a certificate from a certificate vendor. If you do, then the vendor will provide instructions on how to configure Apache with your certificate.

• Run these commands:

cd /etc/pki/tls/certs
cp -p make-dummy-cert make-dummy-cert.bak
cp -p localhost.crt localhost.crt.bak

• Edit make-dummy-cert and replace the answers() function with code similar to this:

answers() {
		echo US
		echo Utah
		echo Provo
		echo Adaptive Computing Enterprises, Inc.
		echo Engineering
		echo test1.adaptivecomputing.com
		echo
}

• Run this command:

./make-dummy-cert localhost.crt

• Configure Apache to use the new certificate and to redirect MWS requests to Tomcat. To do so, edit /etc/httpd/conf.d/ssl.conf.
• Comment out this line:

SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

• Add these lines near the end, just above </VirtualHost>:

ProxyPass /mws http://127.0.0.1:8080/mws retry=5
ProxyPassReverse /mws http://127.0.0.1:8080/mws

• Configure Apache to use SSL for all MWS requests.
• Add these lines to the end of /etc/httpd/conf/httpd.conf:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (/mws.*) https://%{HTTP_HOST}%{REQUEST_URI}

• Give Apache permission to connect to Tomcat.

setsebool -P httpd_can_network_connect 1

• Turn on Apache.

chkconfig httpd on
service httpd start

• Using system-config-firewall-tui, enable "Secure WWW (HTTPS)" and "WWW (HTTP)" as trusted services.

1.6 Version and Build Information

Browser

Using a browser, visit the MWS home page (for example, http://localhost:8080/mws/). At the bottom of the page is the MWS version information. See the screenshot below:

REST Request

Using a REST client or other HTTP client software, send a GET request to the rest/diag/about resource. Here is an example:

curl -u username:password http://localhost:8080/mws/rest/diag/about

This resource is also described under Diagnostics.

MANIFEST.MF File

If MWS fails to start, version and build information can be found in the META-INF/MANIFEST.MF file inside the MWS WAR file. The version properties begin with Implementation. Below is an excerpt of a MANIFEST.MF file:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 20.4-b02 (Sun Microsystems Inc.)
Bundle-ManifestVersion: 2
Bundle-Name: mws
Bundle-SymbolicName: mws
Bundle-Version: 7.0.0
...
Implementation-Build: 4
Implementation-Build-Date: 2012-02-07_17-01-39
Implementation-Revision: 9e109b9a4289800a2c985082d7595d759807aca9
Name: Grails Application
Implementation-Title: mws
Implementation-Version: 7.0.0
Grails-Version: 1.3.7