Troubleshooting

# service moab start
Starting moab: ERROR:    cannot determine local hostname - node is misconfigured
                                    [FAILED]

If you encounter this error when starting Moab, check the /opt/moab/etc/moab.cfg file to make sure a valid host is configured. For example:

...
SCHEDCFG[Moab]                      SERVER=<moab-hostname>:42559
...

Also check /etc/hosts to be sure the host name resolves, at least with localhost:

...
127.0.0.1   <moab-hostname> localhost localhost.localdomain localhost4 localhost4.localdomain4
...

# service moab start
Starting moab: Moab will now exit due to license file not found
Please contact Adaptive Computing ([email protected]) to get a license for your system
                                    [FAILED]

If you encounter this error when starting Moab, make sure your Moab license file is named moab.lic and is located in the /opt/moab/etc/ directory.

Also make sure the license is not expired. The expiration date is listed in the license file. For example:

# cat /opt/moab/etc/moab.lic
...
# Expires after Tue Dec 31 10:43:46 2013
...

If you encounter issues in VMWare vSphere during VM provisioning with the "Create NAS datastore" portion of the process, click on the View details... link in the Recent Tasks section.

Click to enlarge

In this case, the hypervisor was not able to resolve the hostname of the shared storage server ("storage3.ac"). Work with your network administrator to make sure the hypervisors managed by xCAT can resolve storage server(s) within the local network or via a gateway router.

Please see "Troubleshooting and System Maintenance" in the Moab Workload ManagerAdministrator Guide.

Viewpoint super user (moab-admin) password

The default username and password for Viewpoint are moab-admin and changeme! (respectively). For details on how to change the Viewpoint super user password, see "Changing the super user password" in the Viewpoint Management and User Guide.

If the super user password (the moab-admin user) is lost or forgotten, the following instructions will reset the password to the default (moab-admin/changeme!).

These instructions will also remove Viewpoint’s MWS and LDAP configuration, which means you must re-enter the information after restart.

The MWS credentials and the LDAP information can be found in the MWS configuration file (/opt/mws/etc/mws-config.groovy).

1. Stop the tomcat6 service.

[root]# service tomcat6 stop

2. Remove the h2 database files in the /opt/viewpoint/data/h2/ folder.

[root]# rm /opt/viewpoint/data/h2/*

3. Start the tomcat6 service.

[root]# service tomcat6 start

4. Log in to Viewpoint, and do the following:
1. Set the Moab Web Services connection. (For more information, see "Setting the MWS configuration" in the Viewpoint Management and User Guide.)
2. Authenticate the LDAP connection by specifying the bind user password. (For more information, see "Setting the LDAP configuration" in the Viewpoint Management and User Guide.)
3. Change the super user password. (For more information, see "Changing the super user password" in the Viewpoint Management and User Guide.)
4. Check the MWS Diagnostics to verify that the MWS connections are active. (For more information, see "Fields: MWS Diagnostics" in the Viewpoint Management and User Guide.)

MWS super user username and password

The default username and password for MWS are moab-admin and changeme! (respectively).

To change the username and/or the password for the MWS super user:

1. Stop the tomcat6 and moab services.

[root]# service moab stop
[root]# service tomcat6 stop

2. Change the respective values in the following files:
   • /opt/mws/etc/mws-config.groovy:

   auth.defaultUser.username = "moab-admin"
   auth.defaultUser.password = "changeme!"

   •  /opt/moab/etc/moab-private.cfg:

   CLIENTCFG[RM:mws] USERNAME=moab-admin PASSWORD=changeme!

   • /opt/moab/etc/cloud.cfg:

   CONFIG[default]      MWS_USERNAME=moab-admin
   CONFIG[default]      MWS_PASSWORD=changeme!

   • /opt/moab/tools/msm/xcat/msm.cfg:

   APPCFG[xcat]         _MWSUSERNAME=moab-admin
   APPCFG[xcat]         _MWSPASSWORD=changeme!

3. Start the tomcat6 service.

[root]# service tomcat6 start

4. Start the moab service.

[root]# service moab start

5. On the MWS Configuration page in Viewpoint, update the MWS username and/or password. (For more information, see "Setting the MWS configuration" in the ViewpointManagement and User Guide.)

MongoDB passwords

To change the passwords for MongoDB:

1. Stop the tomcat6 and moab services.

[root]# service moab stop
[root]# service tomcat6 stop

2. Change the passwords for the MongoDB accounts (i.e., moab_user and/or mws_user). For instructions, see the MongoDB documentation.
3. Edit the password values in the following files:
   • /opt/moab/etc/moab-private.cfg:

   MONGOUSER            moab_user
   MONGOPASSWORD        secret2

   •  /opt/mws/etc/mws-config.groovy:

   // MongoDB configuration.
   grails.mongo.username = "mws_user"
   grails.mongo.password = "secret3"

4. Start the tomcat6 service.

[root]# service tomcat6 start

5. Start the moab service.

[root]# service moab start

If the application fails to start and gives error messages such as these:

Error creating bean with name 'mongoDatastore'
can't say something; nested exception is com.mongodb.MongoException

ERROR   grails.app.services.com.ace.mws.ErrorService    0
   Error encountered while attempting to authenticate account or query database; the MongoDB server is not available. Please verify connection to server '/127.0.0.1:27017' and that MongoDB is running.

MongoDB is most likely not running, or the MongoDB host and port are misconfigured.

In this case, there are a few things to verify:

• (Not relevant if MongoDB is installed on a separate server) Is MongoDB installed?

Run the following commands to assess whether MongoDB is installed on the current server.

$ mongo
-bash: mongo: command not found

To remedy, install MongoDB, start the mongod service and then restart the tomcat6 service. See Installing MongoDB for more information on how to install and configure MongoDB.

• (Only relevant if MongoDB is installed on a separate server) Is MWS configured to connect to the remote MongoDB server?

Run the following commands to assess whether MongoDB is installed on the current server.

[root]# cat /opt/mws/etc/mws-config.groovy | grep 'grails.mongo'
// grails.mongo.username = "mws_user"
// grails.mongo.password = "<ENTER-KEY-HERE>"
// grails.mongo.host = "127.0.0.1"
// grails.mongo.port = 27017

Make sure that the grails.mongo.* options are configured in /opt/mws/etc/mws-config.groovy for the remote MongoDB server and then restart the tomcat6 service.

[root]# service tomcat6 restart

• Is MWS configured to authenticate with MongoDB, and is MongoDB configured to enforce authentication?

Run the following commands to assess the relevant MWS and MongoDB configurations.

[root]# cat /opt/mws/etc/mws-config.groovy | grep 'grails.mongo'
// grails.mongo.username = "mws_user"
// grails.mongo.password = "<ENTER-KEY-HERE>"

[root]# cat /etc/mongod.conf | grep 'auth'
#noauth = true
auth = true

The configuration above is problematic because the grails.mongo credentials are commented out in the /opt/mws/etc/mws-config.groovy file while MongoDB is configured to enforce authentication ("auth = true"). Similar connection issues will exist if the grails.mongo parameters do not match the credentials configured for the "mws_user" on both the mws and moab databases in MongoDB.

(For upgrade scenarios only) If the application fails to start and gives the following message in /opt/mws/etc/log/mws.log:

java.lang.Exception: The db-migrate.js script has not yet been run. Please see the upgrade section of the installation guide for instructions.

Then the db-migrate.js script must be run to update the schema of the mws database in MongoDB.

To resolve this error, adjust the values of connectionsPerHost or threadsAllowedToBlockForConnectionMultiplier by adding them to mws-config.groovy. Example:

grails.mongo.options.connectionsPerHost = 60
grails.mongo.options.threadsAllowedToBlockForConnectionMultiplier = 10

For more information on these options, refer to these documents:

• "Configuration" (in the Moab Web Services Reference Guide), which briefly discusses a few MongoDB driver options.
• The MongoOptions documentation, which contains full details on all MongoDB driver options.

You must restart Tomcat after adding, removing, or changing grails.mongo.options parameters.

As shipped, mws-config.groovy does not contain any grails.mongo.options parameters. To adjust their values, you need to add them to mws-config.groovy.

The default value of connectionsPerHost is normally 10, but MWS sets it internally to 50.

The default value of threadsAllowedToBlockForConnectionMultiplier is 5.

Any of the options listed in MongoOptions can be specified in mws-config.groovy. Just use the prefix grails.mongo.options as shown above.

See MongoDB: Out of semaphores to get db connection above.

Increase the size of the heap using JVM options -Xms and -Xmx. Here are the suggested values:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m-Xmx3g -XX:MaxPermSize=384m"

• -Xms: Set initial Java heap size.
• -Xmx: Set maximum Java heap size.

Increase the size of the permanent generation using JVM option -XX:MaxPermSize. Here are the suggested values:

CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"

If catalina.out contains this error, look in /opt/mws/log/mws.log and /opt/mws/log/stacktrace.log for more details on the error.

Also ensure that the /opt/mws/etc/mws-config.groovy file can be read by the Tomcat user. The permissions should appear as follows:

$ ls -al /opt/mws/etc/mws-config.groovy
-r-------- 1 tomcat tomcat 4056 Dec  4 12:07 mws-config.groovy

When this error message is encountered, simply add a new line to the moab.cfg file:

CLIENTMAXCONNECTIONS 256

This will change the Moab configuration when Moab is restarted. Run the following command to immediately use the new setting:

[root]# changeparam CLIENTMAXCONNECTIONS 256

The number 256 above may be substituted for the desired maximum number of Moab client connections.