6.59
This topic details some common problems and general solutions for Moab Web Services.
If something goes wrong with MWS, look in the following files:
- The MWS log file. By default this is /opt/mws/log/mws.log.
- The Tomcat catalina.out file, usually in /var/log/tomcat or $CATALINA_HOME/logs.
If you remove the log4j configuration from /opt/mws/etc/mws-config.groovy, MWS writes its log files to java.io.tmpdir. For Tomcat, java.io.tmpdir is generally set to $CATALINA_BASE/temp or CATALINA_TMPDIR.
In this topic:
- 6.59.1 MongoDB: Errors during MWS startup
- 6.59.2 MongoDB: Out of semaphores to get db connection
- 6.59.3 MongoDB: Connection wait timeout after 120000 ms
- 6.59.4 java.lang.OutOfMemoryError: Java heap space
- 6.59.5 java.lang.OutOfMemoryError: PermGen space
- 6.59.6 SEVERE: Context [/mws] startup failed due to previous errors
- 6.59.7 MoabReached Maximum Number of Concurrent Client Connections
6.59.1 MongoDB: Errors during MWS startup
If the application fails to start and gives error messages such as these:
Error creating bean with name 'mongoDatastore' can't say something; nested exception is com.mongodb.MongoException
ERROR grails.app.services.com.ace.mws.ErrorService 0 Error encountered while attempting to authenticate account or query database; the MongoDB server is not available. Please verify connection to server '/127.0.0.1:27017' and that MongoDB is running.
MongoDB is most likely not running, or the MongoDB host and port are misconfigured.
In this case, there are a few things to verify:
- (Not relevant if MongoDB is installed on a different host) Is MongoDB installed?
- (Only relevant if MongoDB is installed on a different host) Is MWS configured to connect to the remote MongoDB host?
- Is MWS configured to authenticate with MongoDB, and is MongoDB configured to enforce authentication?
Run the following commands to assess whether MongoDB is installed on the current host.
$ mongo -bash: mongo: command not found
To remedy, install MongoDB, start the mongod service and then restart the tomcat service. See 6.13.2.C Install MongoDB (Manual Installation) or 6.34.2.B Install MongoDB (RPM Installation) for more information on how to install and configure MongoDB.
Run the following commands to assess whether MongoDB is installed on the current host.
[root]# cat /opt/mws/etc/mws-config.groovy | grep 'grails.mongo' // grails.mongo.username = "mws_user" // grails.mongo.password = "<ENTER-KEY-HERE>" // grails.mongo.host = "127.0.0.1" // grails.mongo.port = 27017
Make sure that the grails.mongo.* options are configured in /opt/mws/etc/mws-config.groovy for the remote MongoDB server and then restart the tomcat service.
[root]# service tomcat restart
[root]# systemctl restart tomcat.service
[root]# systemctl restart tomcat.service
Run the following commands to assess the relevant MWS and MongoDB configurations.
[root]# cat /opt/mws/etc/mws-config.groovy | grep 'grails.mongo' // grails.mongo.username = "mws_user" // grails.mongo.password = "<ENTER-KEY-HERE>" [root]# cat /etc/mongod.conf | grep 'auth' #noauth = true auth = true
The configuration above is problematic because the grails.mongo credentials are commented out in the /opt/mws/etc/mws-config.groovy file while MongoDB is configured to enforce authentication ("auth = true"). Similar connection issues will exist if the grails.mongo parameters do not match the credentials configured for the "mws_user" on both the mws and moab databases in MongoDB.
(For upgrade scenarios only) If the application fails to start and gives the following message in /opt/mws/etc/log/mws.log:
java.lang.Exception: The db-migrate.js script has not yet been run. Please see the upgrade section of the installation guide for instructions.
Then the db-migrate.js script must be run to update the schema of the mws database in MongoDB.
6.59.2 MongoDB: Out of semaphores to get db connection
To resolve this error, adjust the values of connectionsPerHost or threadsAllowedToBlockForConnectionMultiplier by adding them to /opt/mws/etc/mws-config.groovy. For example:
grails.mongo.options.connectionsPerHost = 60 grails.mongo.options.threadsAllowedToBlockForConnectionMultiplier = 10
For more information on these options, refer to these documents:
- Configuring Moab Web Services in the Moab Web Services Administrator Guide, which briefly discusses a few MongoDB driver options.
- The MongoOptions documentation, which contains full details on all MongoDB driver options.
You must restart Tomcat after adding, removing, or changing grails.mongo.options parameters.
As shipped, /opt/mws/etc/mws-config.groovy does not contain any grails.mongo.options parameters. To adjust their values, you need to add them to /opt/mws/etc/mws-config.groovy.
The default value of connectionsPerHost is normally 10, but MWS sets it internally to 50.
The default value of threadsAllowedToBlockForConnectionMultiplier is 5.
Any of the options listed in MongoOptions can be specified in /opt/mws/etc/mws-config.groovy. Just use the prefix grails.mongo.options as shown above.
6.59.3 MongoDB: Connection wait timeout after 120000 ms
See MongoDB: Out of semaphores to get db connection above.
6.59.4 java.lang.OutOfMemoryError: Java heap space
Increase the size of the heap using JVM options -Xms and -Xmx. Here are the suggested values:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
- -Xms: Set initial Java heap size.
- -Xmx: Set maximum Java heap size.
6.59.5 java.lang.OutOfMemoryError: PermGen space
(Recommended) Upgrade to Java. Java 8 has completely removed PermGen space and the MaxPermSize option is ignored.
For Java version prior to 8, you can increase the size of the permanent generation using JVM option -XX:MaxPermSize. Here are the suggested values:
CATALINA_OPTS="-DMWS_HOME=/opt/mws -Xms256m -Xmx3g -XX:MaxPermSize=384m"
6.59.6 SEVERE: Context [/mws] startup failed due to previous errors
If catalina.out contains this error, look in /opt/mws/log/mws.log and /opt/mws/log/stacktrace.log for more details on the error.
Also ensure that the /opt/mws/etc/mws-config.groovy file can be read by the Tomcat user. The permissions should appear as follows:
$ ls -al /opt/mws/etc/mws-config.groovy -r-------- 1 tomcat tomcat 4056 Dec 4 12:07 mws-config.groovy
6.59.7 MoabReached Maximum Number of Concurrent Client Connections
When this error message is encountered, simply add a new line to the moab.cfg file:
CLIENTMAXCONNECTIONS 256
This will change the Moab configuration when Moab is restarted. Run the following command to immediately use the new setting:
[root]# changeparam CLIENTMAXCONNECTIONS 256
The number 256 above may be substituted for the desired maximum number of Moab client connections.