1 Introduction
1.1 Moab® Web Services Overview
Moab Web Services (MWS) is a component of Adaptive Computing Suites that enables programmatic interaction with Moab Workload Manager via a RESTful interface. MWS allows you to create and interact with Moab objects and properties such as jobs, nodes, virtual machines, and reservations. MWS is the preferred method for those wishing to create custom user interfaces for Moab and is the primary method by which Moab Viewpoint communicates with Moab.MWS communicates with the Moab Workload Manager (MWM) server using the same wire protocol as the Moab command-line interface. By publishing a standard interface into Moab's intelligence, MWS significantly reduces the amount of work required to integrate MWM into your solution.This documentation is intended for developers performing such integrations. If you are a Moab administrator, and for conceptual information about MWM, see the Moab Administrator's Guide.1.2 Installation Guide
These instructions describe how to deploy the Moab® Web Services (MWS) to a Tomcat server.1.2.1 Requirements
Hardware Requirements
- Dual core processor
- At least 4 GB of RAM
Software Requirements
- Moab® Workload Manager 7.0
- Oracle® Java® 6 Runtime Environment
- Apache Tomcat™ 6
- MongoDB® 2.0.1 or later
Oracle Java 6 Runtime Environment is the only supported Java environment.All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.
1.2.2 Quickstart Guide
You must deploy Moab Web Services (MWS) on the same server as Moab Workload Manager (MWM).
- Make sure that you have first installed and configured Moab Workload Manager as desired.
- Generate a secret key to be used for communication between MWM and MWS.
Before you create the key file, make sure Moab Workload Manager is not
running. Otherwise, all Moab client commands will stop working. Example:
service moab stop
- Here is a script you can use to generate the key:
dd if=/dev/urandom count=18 bs=1 2>/dev/null | base64 > /opt/moab/etc/.moab.key chown root /opt/moab/etc/.moab.key chmod 400 /opt/moab/etc/.moab.key ln -f /opt/moab/etc/.moab.key /opt/moab/.moab.key
- Start MWM. Example:
service moab start
- Install MongoDB version 2.0.1 or later on the MWM server using packages (recommended) or manual installation.
- Packages (recommended) are available from the Mongo-provided repositories for CentOS and Fedora or Ubuntu and Debian.
- After installing the packages, start MongoDB and arrange for it to start automatically during server startup.
- Here is an example for CentOS and Fedora:
/sbin/chkconfig mongod on
/etc/init.d/mongod start
- For Ubuntu and Debian, see the Configuration section of this page for examples.
- For manual installation (not recommended), follow these instructions. Be sure to start the Mongo server (
mongod
) after installation and arrange for it to start automatically during server startup (by writing and registering an/etc/init.d
script, for example). - Download, install, and enable the 64-bit version of the Oracle Java SE 6 JRE.
Oracle Java 6 Runtime Environment is the only supported Java environment.All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.
- Here is an example for a CentOS system:
sh jre-6u33-linux-x64-rpm.bin rm -f /usr/bin/java ln -s /etc/alternatives/java /usr/bin/java /usr/sbin/alternatives --install /usr/bin/java java /usr/java/jre1.6.0_33/bin/java 500 /usr/sbin/alternatives --set java /usr/java/jre1.6.0_33/bin/java
Thealternatives
command is calledupdate-alternatives
on some Linux distributions.
- You can verify the Java installation by running the following command:
java -version
- The output should look similar to this:
java version "1.6.0_33" Java(TM) SE Runtime Environment (build 1.6.0_33-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
- Create the MWS home directory and its subdirectories
etc
,hooks
,plugins
, andlog
.
The default location for the MWS home directory is /opt/mws
. These instructions assume the default location.
- Give the Tomcat user read access to these directories and write access to the
plugins
andlog
directories. - Here is a sample script for these steps:
mkdir -p /opt/mws/etc /opt/mws/hooks /opt/mws/plugins /opt/mws/log chown -R tomcat /opt/mws # Depending on your OS, the Tomcat username might be tomcat6. chmod -R 555 /opt/mws chmod u+w /opt/mws/plugins /opt/mws/log
- Extract the contents of the MWS tarball into a temporary directory. Example:
mkdir /tmp/mws-install cd /tmp/mws-install tar xvzf $HOME/Downloads/mws-<VERSION>.tar.gz cd /tmp/mws-install/mws-<VERSION>
- Set up the MWS configuration file.
- In the extracted MWS directory is a sample configuration file:
mws-config.groovy
. Copy this file to/opt/mws/etc
. - Give the Tomcat user read access to
/opt/mws/etc/mws-config.groovy
. - In the
/opt/mws/etc/mws-config.groovy
file, change these settings: moab.secretKey
: needs to match the MWM secret key you generated earlier (contained in/opt/moab/etc/.moab.key
)auth.defaultUser.username
: any value you like, or leave as isauth.defaultUser.password
: any value you like, but choose a good password
vi /opt/mws/etc/mws-config.groovy… moab.secretKey = "<ENTER-KEY-HERE>" moab.server = "localhost" moab.port = 42559// Change these to be whatever you like. auth.defaultUser.username = "admin" auth.defaultUser.password = "adminpw"
If you do not change auth.defaultUser.password
, then your MWS is not secure, since anyone reading these instructions can log into your MWS.
Here are some tips for choosing a good password.
- Set the following parameters in your Tomcat CATALINA_OPTS:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
Where you choose to storeCATALINA_OPTS
depends on various factors, including operating system and sysadmin preference. Here are some suggestions:
- CentOS™ 5 and 6:
/etc/sysconfig/tomcat6
- Red Hat® Enterprise Linux 5 and 6:
/etc/sysconfig/tomcat6
- SUSE® Linux Enterprise Server 11:
/etc/tomcat6/tomcat6.conf
- Ubuntu® 10.04:
/etc/default/tomcat6
- Start Tomcat and deploy
mws.war
. Example:
chkconfig tomcat6 on
service tomcat6 start
cp /tmp/mws-install/mws-<VERSION>/mws.war /var/lib/tomcat6/webapps
- Visit http://localhost:8080/mws/ in a web browser to verify that MWS is running. You will see some sample queries and a few other actions.
- Log into MWS to verify that the MWS credentials are working. (The credentials are the values of
auth.defaultUser.username
andauth.defaultUser.password
that you set above.)
If you encounter problems, or if MWS does not seem to be running, see the steps below in the Troubleshooting section.
1.2.3 Troubleshooting Installation
If something goes wrong with MWS, look in the following files:- The MWS log file. By default this is
/opt/mws/log/mws.log
. - The Tomcat
catalina.out
file, usually in/var/log/tomcat6
or$CATALINA_HOME/logs
.
If you remove theHere is a list of some errors and their fixes:log4j
configuration frommws-config.groovy
, MWS will write its log files tojava.io.tmpdir
. For Tomcat,java.io.tmpdir
is generally set to$CATALINA_BASE/temp
orCATALINA_TMPDIR
.
MongoDB Errors
If the application fails to start and gives error messages such as these:Error creating bean with name 'mongoDatastore' can't say something; nested exception is com.mongodb.MongoException
java.lang.OutOfMemoryError: Java heap space
Increase the size of the heap using JVM options-Xms
and -Xmx
. Here are the suggested values from the Quickstart Guide:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
-Xms
: Set initial Java heap size.-Xmx
: Set maximum Java heap size.
java.lang.OutOfMemoryError: PermGen space
Increase the size of the permanent generation using JVM option-XX:MaxPermSize
. Here are the suggested values from the Quickstart Guide:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
SEVERE: Context [/mws] startup failed due to previous errors
Ifcatalina.out
contains this error, look in /opt/mws/log/mws.log
and /opt/mws/log/stacktrace.log
for more details on the error.Moab Reached Maximum Number of Concurrent Client Connections
When this error message is encountered, simply add a new line to themoab.cfg
file:CLIENTMAXCONNECTIONS 256
changeparam CLIENTMAXCONNECTIONS 256
The number 256
above may be substituted for the desired maximum number of MWM client connections.
1.3 Configuration
This section describes where Moab Web Services searches for its configuration files. It also shows some examples of how to configure logging.To see a full reference to all configuration and logging parameters available in MWS, see the Configuration page under Moab Web Services in the Quick Reference menu.
Home Directory
The MWS home directory contains all configuration as well as other files that serve features of MWS such as hooks and plugins. This is typically set by using theMWS_HOME
property as
explained in the Quickstart Guide. If MWS_HOME
is not set
as a Java property or as an environment variable for the current application
container (i.e. Tomcat), /opt/mws
will be used as the default MWS_HOME
. If no configuration files
are found in MWS_HOME
, MOABHOMEDIR
will be used. If this property also does
not exist, the home directory will default to /opt/moab
.The home directory consists of several sub-directories:MWS_HOME
orMOABHOMEDIR
can be set either as a Java property or as an environment variable. See the Quickstart Guide for suggestions on how to setMWS_HOME
.
etc
- Used for storing configuration files.hooks
- Used for storing hook files. This is not required if hooks are not being used.plugins
- Used for storing plugin types. This is not required if custom plugin types are not being used.
Thehooks
andplugins
directories should be writable by the application container's user, such as thetomcat
user.
Configuration File Locations
MWS searches the following directories for configuration files in the order shown below. As soon as a configuration file is found in one of these directories, that file is loaded and searching stops. If alog4j.properties
file exists in the same directory, it will be loaded
as well.
MWS_HOME/etc
MWS_HOME
/opt/mws/etc
/opt/mws
MOABHOMEDIR/etc
MOABHOMEDIR
/opt/moab/etc
/opt/moab
- In each directory, MWS looks first for
mws-config.groovy
and then formws-config.properties
. If it findsmws-config.groovy
, it does not look formws-config.properties
.mws-config.groovy
uses a style that is similar to a Java properties file with some extensions from Groovy.mws-config.properties
is a regular Java properties file.
Logging Configuration Using mws-config.groovy
Shown below is an example that logs all error messages and fatal
messages to /opt/mws/log/mws.log
. It also logs all stack traces to
/opt/mws/log/stacktrace.log
.Minimal Logging Configuration
log4j = { appenders { rollingFile name: 'stacktrace', file: '/opt/mws/log/stacktrace.log', maxFileSize: '1GB' rollingFile name: 'rootLog', file: '/opt/mws/log/mws.log', threshold: org.apache.log4j.Level.ERROR, maxFileSize: '1GB' } root { debug 'rootLog' } }
Console Logging Configuration
log4j = { appenders { rollingFile name: 'stacktrace', file: '/opt/mws/log/stacktrace.log', maxFileSize: '1GB' console name: 'consoleLog', threshold: org.apache.log4j.Level.ERROR } root { debug 'consoleLog' } }
For all possible configuration options, see the Configuration section in the reference guide.
- For the examples above, you must make sure that
/opt/mws/log
exists and is writable by the application server.- You may configure logging using either
mws-config.groovy
or a regularlog4j.properties
file. Thelog4j.properties
file must be in the same directory as themws-config.groovy
file.- If you do not define any
log4j
configuration, MWS will write its log files tojava.io.tmpdir
. For Tomcat,java.io.tmpdir
is generally set to$CATALINA_BASE/temp
orCATALINA_TMPDIR
.
1.4 Security
When running MWS in production environments, security is a major concern. This section focuses on securing the three kinds of connections with MWS:- The connection between MWS and Moab Workload Manager (MWM)
- The connection between MWS and MongoDB
- The connections between clients and MWS
Connection with MWM
MWS communicates with MWM via the Moab Wire Protocol, which uses a direct connection between the two applications. The communication over this connection uses a shared secret key, which is discussed in the Quickstart Guide. However, the communication is not encrypted and is therefore susceptible to eavesdropping and replay attacks. For this reason, MWS is supported only when running on the same machine as MWM. This assures that any connections between the two applications occur internally on the server and are not exposed to external users.Connection with MongoDB
By default, the connection between MWS and MongoDB is not authenticated. To enable authentication between them, see the instructions below.- MWS Configuration: see the Configuration reference guide for information on the
grails.mongo
properties to set inmws-config.groovy
. - MongoDB Configuration: see the MongoDB Security and Authentication guide. Generally, the following steps are required:
- Add an administrative user to MongoDB in the
admin
database. - Start MongoDB with authentication activated (using the
--auth
command-line option for example). - Log in as the administrative user to the
admin
database. - Add a user for MWS to use with full read and write access to the database specified in the configuration file (
mws
by default). - Change the proper configuration file properties with the created username and password.
- Restart MWS by restarting the servlet container (Tomcat).
Client Connections to MWS
All connections to MWS, except those requesting the documentation or the main page, must be authenticated properly. MWS uses a single-trusted-user authentication model, meaning a single user exists that has access to all aspects of MWS. The username and password for this user are configured with theauth.defaultUser
properties in the configuration file. See the
Configuration reference guide for more information.When using the MWS user interface in a browser, the user will be prompted
for username and password. For information on how to authenticate requests
when not using a browser, see the API Security
section in the user guide.The username and password in the Basic Authentication header are encoded but not encrypted. Therefore, it is strongly recommended that MWS be run behind a proxy (like Apache) with SSL enabled. The instructions below provide an example of how to do this.
Encrypting Client Connections using Apache and SSL
This section shows how to encrypt client connections to MWS using Apache and SSL. These instructions have been tested on CentOS™ 6.2 with the "Web Server" software set installed. The same ideas are applicable to other operating systems, but the details might be different. As shown in the diagram below, these instructions assume that Tomcat and Apache are running on the same server.- Create a self-signed certificate. See http://www.openssl.org/docs/HOWTO/certificates.txt for more details if desired.
Instead of creating a self-signed certificate, you can buy a certificate from a certificate vendor. If you do, then the vendor will provide instructions on how to configure Apache with your certificate.
- Run these commands:
cd /etc/pki/tls/certs cp -p make-dummy-cert make-dummy-cert.bak cp -p localhost.crt localhost.crt.bak
- Edit
make-dummy-cert
and replace theanswers()
function with code similar to this:
answers() { echo US echo Utah echo Provo echo Adaptive Computing Enterprises, Inc. echo Engineering echo test1.adaptivecomputing.com echo }
- Run this command:
./make-dummy-cert localhost.crt
- Configure Apache to use the new certificate and to redirect MWS requests to Tomcat. To do so, edit
/etc/httpd/conf.d/ssl.conf
. - Comment out this line:
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
- Add these lines near the end, just above
</VirtualHost>
:
ProxyPass /mws http://127.0.0.1:8080/mws retry=5 ProxyPassReverse /mws http://127.0.0.1:8080/mws
- Configure Apache to use SSL for all MWS requests.
- Add these lines to the end of
/etc/httpd/conf/httpd.conf
:
RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (/mws.*) https://%{HTTP_HOST}%{REQUEST_URI}
- Give Apache permission to connect to Tomcat.
setsebool -P httpd_can_network_connect 1
- Turn on Apache.
chkconfig httpd on service httpd start
- Using
system-config-firewall-tui
, enable "Secure WWW (HTTPS)" and "WWW (HTTP)" as trusted services.
1.5 Version and Build Information
To get detailed version information about MWS, use one of the following three methods:Browser
Using a browser, visit the MWS home page (for example, http://localhost:8080/mws/). At the bottom of the page is the MWS version information. See the screenshot below:REST Request
Using a REST client or other HTTP client software, send a GET request to therest/diag/about
resource. Here is an example:curl -u username:password http://localhost:8080/mws/rest/diag/about
MANIFEST.MF File
If MWS fails to start, version and build information can be found in theMETA-INF/MANIFEST.MF
file inside the MWS WAR file. The version
properties begin with Implementation
. Below is an excerpt of a
MANIFEST.MF
file:Manifest-Version: 1.0 Ant-Version: Apache Ant 1.7.1 Created-By: 20.4-b02 (Sun Microsystems Inc.) Bundle-ManifestVersion: 2 Bundle-Name: mws Bundle-SymbolicName: mws Bundle-Version: 7.0.0...Implementation-Build: 4 Implementation-Build-Date: 2012-02-07_17-01-39 Implementation-Revision: 9e109b9a4289800a2c985082d7595d759807aca9Name: Grails Application Implementation-Title: mws Implementation-Version: 7.0.0 Grails-Version: 1.3.7