1 Introduction
1.1 Moab® Web Services Overview
Moab Web Services (MWS) is a component of Adaptive Computing Suites that enables programmatic interaction with Moab Workload Manager via a RESTful interface. MWS allows you to create and interact with Moab objects and properties such as jobs, nodes, virtual machines, and reservations. MWS is the preferred method for those wishing to create custom user interfaces for Moab and is the primary method by which Moab Viewpoint communicates with Moab.MWS communicates with the Moab Workload Manager (MWM) server using the same wire protocol as the Moab command-line interface. By publishing a standard interface into Moab's intelligence, MWS significantly reduces the amount of work required to integrate MWM into your solution.This documentation is intended for developers performing such integrations. If you are a Moab administrator, and for conceptual information about MWM, see the Moab Administrator's Guide.1.2 Installation Guide
These instructions describe how to install Moab® Web Services (MWS).1.2.1 Requirements
Hardware Requirements
- 64-bit dual-core processor
- At least 4 GB of RAM
Software Requirements
- Moab® Workload Manager (version must match exactly the version of MWS)
- Oracle® Java® 6 Runtime Environment
- Apache Tomcat™ 6
- MongoDB® 2.0.x, where x is 2 or greater
Oracle Java 6 Runtime Environment is the only supported Java environment.All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.
1.2.2 Quickstart Guide
1) Install MongoDB version 2.0.x, where x is 2 or greater.
MWS does not yet support MongoDB 2.2.x. Be sure to install the 2.0.x packages. As of this writing, the RPM package names aremongo20-10gen-2.0.7-mongodb_1.x86_64.rpm
andmongo20-10gen-server-2.0.7-mongodb_1.x86_64.rpm
. The Ubuntu package name ismongodb20-10gen_2.0.7_amd64.deb
.
2) Start MongoDB.
# CentOS 6 examplechkconfig mongod on service mongod start
The instructions provided above for installing MongoDB describe a base installation only. See the MongoDB section of the security page.
3) Install and configure Moab Workload Manager (MWM).
- You must deploy Moab Web Services (MWS) on the same server as Moab Workload Manager (MWM).
- The version of MWS must match exactly the version of MWM. For example, MWS 7.1.1 works only with MWM 7.1.1.
4) Generate a secret key to be used for communication between MWM and MWS.
# All these steps are required. Do not skip any steps.service moab stop dd if=/dev/urandom count=18 bs=1 2>/dev/null | base64 > /opt/moab/etc/.moab.key chown root /opt/moab/etc/.moab.key chmod 400 /opt/moab/etc/.moab.key ln -f /opt/moab/etc/.moab.key /opt/moab/.moab.key service moab start
5) Install Apache Tomcat 6.
# CentOS 6 exampleyum install tomcat6
6) Install the 64-bit version of the Oracle Java SE 6 JRE.
Oracle Java 6 Runtime Environment is the only supported Java environment.All other versions of Java, including Oracle Java 7, OpenJDK/IcedTea, GNU Compiler for Java, and so on, cannot run Moab Web Services.
# CentOS 6 examplesh jre-6u37-linux-x64-rpm.bin rm -f /usr/bin/java ln -s /etc/alternatives/java /usr/bin/java alternatives --install /usr/bin/java java /usr/java/jre1.6.0_37/bin/java 500 alternatives --set java /usr/java/jre1.6.0_37/bin/java
Thealternatives
command is calledupdate-alternatives
on some Linux distributions.
- You can verify the Java installation by running
java -version
- The output should look similar to this:
java version "1.6.0_37" Java(TM) SE Runtime Environment (build 1.6.0_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)
7) Create the MWS home directory and its subdirectories etc
, hooks
, plugins
, and log
.
The default location for the MWS home directory is /opt/mws
. These instructions assume the default location.
- Give the Tomcat user read access to these directories and write access to the
plugins
andlog
directories. - Here is a sample script for these steps:
mkdir -p /opt/mws/etc /opt/mws/hooks /opt/mws/plugins /opt/mws/log chown -R tomcat /opt/mws # Depending on your OS, the Tomcat username might be tomcat6. chmod -R 555 /opt/mws chmod u+w /opt/mws/plugins /opt/mws/log
8) Extract the contents of the MWS tarball into a temporary directory.
mkdir /tmp/mws-install cd /tmp/mws-install tar xvzf $HOME/Downloads/mws-<VERSION>.tar.gz cd /tmp/mws-install/mws-<VERSION>
9) Set up the MWS configuration file.
- In the extracted MWS directory are two sample configuration files:
mws-config-cloud.groovy
andmws-config-hpc.groovy
. mws-config-cloud.groovy
provides sample configuration for the Moab Cloud Suite.mws-config-hpc.groovy
provides sample configuration for the Moab HPC Suites.- Choose the correct file for your suite, rename it to
mws-config.groovy
, and copy it to/opt/mws/etc
. - Give the Tomcat user read access to
/opt/mws/etc/mws-config.groovy
. - In the
/opt/mws/etc/mws-config.groovy
file, change these settings: moab.secretKey
: needs to match the MWM secret key you generated earlier (contained in/opt/moab/etc/.moab.key
)auth.defaultUser.username
: any value you like, or leave as isauth.defaultUser.password
: any value you like, but choose a good password
vi /opt/mws/etc/mws-config.groovy… moab.secretKey = "<ENTER-KEY-HERE>" moab.server = "localhost" moab.port = 42559// Change these to be whatever you like. auth.defaultUser.username = "admin" auth.defaultUser.password = "adminpw"
If you do not change auth.defaultUser.password
, then your MWS is not secure, since anyone reading these instructions can log into your MWS.
Here are some tips for choosing a good password.
10) Add the following line to the end of /etc/tomcat6/tomcat6.conf
:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
Some Linux distributions use/etc/default/tomcat6
instead of/etc/tomcat6/tomcat6.conf
.
11) Start Tomcat and deploy mws.war
.
# CentOS 6 examplechkconfig tomcat6 on
service tomcat6 stop
cp /tmp/mws-install/mws-<VERSION>/mws.war /var/lib/tomcat6/webapps
service tomcat6 start
12) Visit http://localhost:8080/mws/ in a web browser to verify that MWS is running.
You will see some sample queries and a few other actions.13) Log into MWS to verify that the MWS credentials are working.
The credentials are the values ofauth.defaultUser.username
and auth.defaultUser.password
that you set above.
If you encounter problems, or if MWS does not seem to be running, see the steps below in the Troubleshooting section.
1.3 Troubleshooting
If something goes wrong with MWS, look in the following files:- The MWS log file. By default this is
/opt/mws/log/mws.log
. - The Tomcat
catalina.out
file, usually in/var/log/tomcat6
or$CATALINA_HOME/logs
.
If you remove theHere is a list of some errors and their fixes:log4j
configuration frommws-config.groovy
, MWS will write its log files tojava.io.tmpdir
. For Tomcat,java.io.tmpdir
is generally set to$CATALINA_BASE/temp
orCATALINA_TMPDIR
.
MongoDB: Errors during MWS startup
If the application fails to start and gives error messages such as these:Error creating bean with name 'mongoDatastore' can't say something; nested exception is com.mongodb.MongoException
MongoDB: Out of semaphores to get db connection
To resolve this error, adjust the values ofconnectionsPerHost
or threadsAllowedToBlockForConnectionMultiplier
by adding them to
mws-config.groovy
. Example:
grails.mongo.options.connectionsPerHost = 60 grails.mongo.options.threadsAllowedToBlockForConnectionMultiplier = 10
- The Configuration page under Moab Web Services in the Quick Reference menu, which briefly discusses a few MongoDB driver options.
- The MongoOptions documentation, which contains full details on all MongoDB driver options.
- You must restart Tomcat after adding, removing, or changing
grails.mongo.options
parameters.- As shipped,
mws-config.groovy
does not contain anygrails.mongo.options
parameters. To adjust their values, you need to add them tomws-config.groovy
.- The default value of
connectionsPerHost
is normally 10, but MWS sets it internally to 50.- The default value of
threadsAllowedToBlockForConnectionMultiplier
is 5.- Any of the options listed in MongoOptions can be specified in
mws-config.groovy
. Just use the prefixgrails.mongo.options
as shown above.
MongoDB: Connection wait timeout after 120000 ms
See the section "MongoDB: Out of semaphores to get db connection" above.java.lang.OutOfMemoryError: Java heap space
Increase the size of the heap using JVM options-Xms
and -Xmx
. Here are the suggested values from the Quickstart Guide:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
-Xms
: Set initial Java heap size.-Xmx
: Set maximum Java heap size.
java.lang.OutOfMemoryError: PermGen space
Increase the size of the permanent generation using JVM option-XX:MaxPermSize
. Here are the suggested values from the Quickstart Guide:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
SEVERE: Context [/mws] startup failed due to previous errors
Ifcatalina.out
contains this error, look in /opt/mws/log/mws.log
and /opt/mws/log/stacktrace.log
for more details on the error.Moab Reached Maximum Number of Concurrent Client Connections
When this error message is encountered, simply add a new line to themoab.cfg
file:CLIENTMAXCONNECTIONS 256
changeparam CLIENTMAXCONNECTIONS 256
The number 256
above may be substituted for the desired maximum number of MWM client connections.
1.4 Configuration
This section describes the location of the Moab Web Services configuration files. It also shows some examples of how to configure logging.To see a full reference to all configuration and logging parameters available in MWS, see the Configuration page under Moab Web Services in the Quick Reference menu.
Home Directory
The MWS home directory contains configuration files, log files, and files that serve features of MWS such as hooks and plugins. You should set the location of the MWS home directory using theMWS_HOME
property
as shown in the Quickstart Guide. If you do not set
MWS_HOME
as a Java property or as an environment variable, then MWS
will use /opt/mws
as the default MWS_HOME
.Configuration Files
The primary configuration file isMWS_HOME/etc/mws-config.groovy
.
If this file is missing or contains errors, MWS will not start.
If MWS_HOME/etc/log4j.properties
exists, MWS will load it as well.Logging Configuration Using mws-config.groovy
Shown below is an example that logs all error messages and fatal
messages to /opt/mws/log/mws.log
. It also logs all stack traces to
/opt/mws/log/stacktrace.log
. Note that this example is not configured
to log events.Minimal Logging Configuration
log4j = { appenders { rollingFile name: 'stacktrace', file: '/opt/mws/log/stacktrace.log', maxFileSize: '1GB' rollingFile name: 'rootLog', file: '/opt/mws/log/mws.log', threshold: org.apache.log4j.Level.ERROR, maxFileSize: '1GB' } root { debug 'rootLog' } }
Console Logging Configuration
log4j = { appenders { rollingFile name: 'stacktrace', file: '/opt/mws/log/stacktrace.log', maxFileSize: '1GB' console name: 'consoleLog', threshold: org.apache.log4j.Level.ERROR } root { debug 'consoleLog' } }
- You may configure logging using either
MWS_HOME/etc/mws-config.groovy
orMWS_HOME/etc/log4j.properties
.- If you do not define any
log4j
configuration, MWS will write its log files tojava.io.tmpdir
. For Tomcat,java.io.tmpdir
is generally set to$CATALINA_BASE/temp
orCATALINA_TMPDIR
.
1.5 Security
When running MWS in production environments, security is a major concern. This section focuses on securing the three kinds of connections with MWS:- The connection between MWS and Moab Workload Manager (MWM)
- The connection between MWS and MongoDB
- The connections between clients and MWS
Connection with MWM
MWS communicates with MWM via the Moab Wire Protocol, which uses a direct connection between the two applications. The communication over this connection uses a shared secret key, which is discussed in the Quickstart Guide. However, the communication is not encrypted and is therefore susceptible to eavesdropping and replay attacks. For this reason, MWS is supported only when running on the same machine as MWM. This assures that any connections between the two applications occur internally on the server and are not exposed to external users.Connection with MongoDB
By default, the connection between MWS and MongoDB is not authenticated. To enable authentication between them, see the instructions below.- MWS Configuration: see the Configuration reference guide for information on the
grails.mongo
properties to set inmws-config.groovy
. - MongoDB Configuration: see the MongoDB Security and Authentication guide. Generally, the following steps are required:
- Add an administrative user to MongoDB in the
admin
database. - Start MongoDB with authentication activated (using the
--auth
command-line option for example). - Log in as the administrative user to the
admin
database. - Add a user for MWS to use with full read and write access to the database specified in the configuration file (
mws
by default). - Change the proper configuration file properties with the created username and password.
- Restart MWS by restarting the servlet container (Tomcat).
Client Connections to MWS
All connections to MWS, except those requesting the documentation or the main page, must be authenticated properly. MWS uses a single-trusted-user authentication model, meaning a single user exists that has access to all aspects of MWS. The username and password for this user are configured with theauth.defaultUser
properties in the configuration file. See the
Configuration reference guide for more information.When using the MWS user interface in a browser, the user will be prompted
for username and password. For information on how to authenticate requests
when not using a browser, see the Authentication
section in the user guide.The username and password in the Basic Authentication header are encoded but not encrypted. Therefore, it is strongly recommended that MWS be run behind a proxy (like Apache) with SSL enabled. The instructions below provide an example of how to do this.
Encrypting Client Connections using Apache and SSL
This section shows how to encrypt client connections to MWS using Apache and SSL. These instructions have been tested on CentOS™ 6.2 with the "Web Server" software set installed. The same ideas are applicable to other operating systems, but the details might be different. As shown in the diagram below, these instructions assume that Tomcat and Apache are running on the same server.- Create a self-signed certificate. See http://www.openssl.org/docs/HOWTO/certificates.txt for more details if desired.
Instead of creating a self-signed certificate, you can buy a certificate from a certificate vendor. If you do, then the vendor will provide instructions on how to configure Apache with your certificate.
- Run these commands:
cd /etc/pki/tls/certs cp -p make-dummy-cert make-dummy-cert.bak cp -p localhost.crt localhost.crt.bak
- Edit
make-dummy-cert
and replace theanswers()
function with code similar to this:
answers() { echo US echo Utah echo Provo echo Adaptive Computing Enterprises, Inc. echo Engineering echo test1.adaptivecomputing.com echo }
- Run this command:
./make-dummy-cert localhost.crt
- Configure Apache to use the new certificate and to redirect MWS requests to Tomcat. To do so, edit
/etc/httpd/conf.d/ssl.conf
. - Comment out this line:
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
- Add these lines near the end, just above
</VirtualHost>
:
ProxyPass /mws http://127.0.0.1:8080/mws retry=5 ProxyPassReverse /mws http://127.0.0.1:8080/mws
- Configure Apache to use SSL for all MWS requests.
- Add these lines to the end of
/etc/httpd/conf/httpd.conf
:
RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (/mws.*) https://%{HTTP_HOST}%{REQUEST_URI}
- Give Apache permission to connect to Tomcat.
setsebool -P httpd_can_network_connect 1
- Turn on Apache.
chkconfig httpd on service httpd start
- Using
system-config-firewall-tui
, enable "Secure WWW (HTTPS)" and "WWW (HTTP)" as trusted services.
1.6 Version and Build Information
To get detailed version information about MWS, use one of the following three methods:Browser
Using a browser, visit the MWS home page (for example, http://localhost:8080/mws/). At the bottom of the page is the MWS version information. See the screenshot below:REST Request
Using a REST client or other HTTP client software, send a GET request to therest/diag/about
resource. Here is an example:curl -u username:password http://localhost:8080/mws/rest/diag/about
MANIFEST.MF File
If MWS fails to start, version and build information can be found in theMETA-INF/MANIFEST.MF
file inside the MWS WAR file. The version
properties begin with Implementation
. Below is an excerpt of a
MANIFEST.MF
file:Implementation-Build: 26 Implementation-Build-Date: 2012-06-19_14-18-59 Implementation-Revision: 376079a5e5f552f2fe25e6070fd2e84c646a98fdName: Grails Application Implementation-Title: mws Implementation-Version: 7.1.0-rc2 Grails-Version: 2.0.3