5.5 Nitro Web Services Issues

This topic details some common problems and general solutions for Nitro Web Services.

In this topic:

• 5.5.1 Logging
• 5.5.2 Debugging
• 5.5.3 Viewpoint does not show job status updates

5.5.1 Logging

Logs are located in /opt/nitro-web-services/logs/*.log.

Logging is set to INFO (just below DEBUG) by default. Each service has it's own log_level setting. See /opt/nitro-web-services/etc/*.cfg for details.

If you change the log_level, you must restart the respective service.

5.5.2 Debugging

Try running the service from the command line.

The following procedure is an example of debugging the nitro-web-services service.

1. Stop the nitro-web-services service and leave the nitro-zmq-job-status-adapter service running.

   [root]# service nitro-web-services stop

2. Run the nitro-web-service service from the command line.
   • Use an ampersand (nitro-web-services &) if you want to run the service in the background.
   • Fully qualify the path (i.e. /usr/bin/nitro-web-services or /bin/nitro-web-services) if nitro-web-services isn't found.

   [root]# nitro-web-services

3. Exercise the service from a client/UI.

   Check for stacktraces in STDOUT/STDERR.

4. If you need to debug further, contact your Adaptive Computing account manager.

5.5.3 Viewpoint does not show job status updates

If you are not getting job status updates after launching your Nitro job, it is likely to be an inter-node communication problem (reachability, firewall, etc.) or an authentication/authorization issue (incorrect username/password).

This section provides a diagram and depicted steps to diagnose the problem. The order in which you follow these steps isn't significant, but is recommended.

Use the following diagram as a reference to the steps that follow.

Click to enlarge

Use the following steps to diagnose the problem. The order in which you follow these steps isn't required, but is recommended.

1. Check "Job Status Update (ZMQ)".

   1. Job status updates are published to the ZMQ message bus by the Nitro Coordinator (node1, step 7 in the diagram). Each node (node1, node2, ..., node(N)) can play the role of the Nitro coordinator, therefore, each node must define the "nws-connector-address" in their respective Nitro configuration file (/opt/nitro/etc/nitro.cfg).

      Next steps:

      • Modify the /opt/nitro/etc/nitro.cfg as needed.

        If you are using a shared file system, you will only have to make the modification once; otherwise, make the update on each compute node.

   2. <nitro-web-services-hostname> must be reachable from each Nitro coordinator and the designated ZMQ port (47100) must match the "msg_port" defined in /opt/nitro-web-services/etc/zmq_job_status_adapter.cfg (defaults to 47100 when not explicitly defined).

      ...
      # Viewpoint connection allows Nitro to communicate job status information
      # to viewpoint.  This option indicates name and port of the remote server
      # in the form: <host>:<port>
      nws-connector-address <nitro-web-services-hostname>:47100
      ...

      Next steps:

      • If your system uses firewalls, verify the necessary ports are open. See 1.1.1 Open Necessary Ports for more information.

        You may also be able to use telnet, netstat, etc. to check if <nitro-web-services-hostname> is reachable and the configured ZMQ port is open.

2. Check the ZMQ Job Status Adapter log for information. Check "log_level" in /opt/nitro-web-services/etc/zmq_job_status_adapter.cfg on the Nitro Web Services host (for example, the Moab Head Node). When "log_level" is not defined, the default is "INFO". The only log level more verbose is "DEBUG". Restart the zmq-job-status-adapter service if you change any configuration options.

   Tail the the ZMQ Job Status Adapter log (/opt/nitro-web-services/logs/zmq_job_status_adapter.log) while running a Nitro job.

   [Moab Head Node]# tail -f /opt/nitro-web-services/logs/zmq_job_status_adapter.log

   Next steps:

   • If you see any information pertaining to your job, then the Nitro Coordinators are successfully communicating with the ZMQ Job Status Adapter via ZMQ.
   • If you do not see any job information, check step 1 in this procedure.
3. Check whether the ZMQ Job Status Adapter can authenticate to Nitro Web Services.

   The following are the ZMQ Job Status Adapter configuration settings (/opt/nitro-web-services/etc/zmq_job_status_adapter.cfg).

   

   After initial installation, the defaults are depicted as comments in the configuration file.

   # DNS/IP and port where REST API (i.e. Nitro Web Services) is hosted
   #http_protocol = https
   #rest_api_host = localhost
   #rest_api_port = 9443
   #username = nitro-writeonly-user
   #password = ChangeMe4!

   Try authenticating to Nitro Web Services from the ZMQ Job Status Adapter host (Moab Head Node).

   [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-writeonly-user", "password": "ChangeMe4!"}' \
    https://localhost:9443/auth

   Next steps:

   • If you get an HTTP status code other than 200 or 401, make sure the Nitro Web Services service is up and running.
   • If you get an HTTP status code of 200, go to step 4.

   • If you get an HTTP status code of 401, the configured "username" (nitro-writeonly-user) is unable to authenticate.

     Do the following:

     1. Check the value of "ws_writeonly_username" and "ws_writeonly_password" in /opt/nitro-web-services/etc/nitro.cfg.
     2. Set "username" and "password" in /opt/nitro-web-services/etc/zmq_job_status_adapter.cfg so that they match.
     3. Restart the ZMQ Job Status Adapter service.
     4. Retry the above curl command with the updated "username" and "password".

   • If you still don't get an HTTP status code of 200, try resetting the nitro-writeonly-user's password in MongoDB.

     # Any user can update its own password.
     # The nitro-admin user can update any user's password.
     
     # Obtain a nitro-key (session token) by authenticating as either the nitro-admin or the nitro-writeonly-user
     # Option 1: nitro-admin
     [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-admin", "password": "ChangeMe2!"}' https://localhost:9443/auth
     # Option 2: nitro-writeonly-user
     [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-writeonly-user", "password": "ChangeMe4!"}' https://localhost:9443/auth
     
     # Example nitro-admin authentication response:
     > {"status": 200, "data": {"nitro-key": "3e0fb95e9a0e44ae91daef4deb500dcc67a3714880e851d781512a49", "user": {"username": "nitro-admin", "last_updated": "2016-08-19 16:46:17.395000", "name": "Nitro Admin", "created": "2016-08-19 16:46:17.395000", "auth": {"job": ["read", "write", "delete"], "user": ["read", "write", "delete"]}}}}
       
     # Use the nitro-key from the authentication response to change nitro-writeonly-user's password
     [root@MoabHeadNode]# curl --insecure -X PUT --header "nitro-key: 3e0fb95e9a0e44ae91daef4deb500dcc67a3714880e851d781512a49" --data '{"password": "Astr0ngPa$$!"}'  https://localhost:9443/user/nitro-writeonly-user

   • Once you have reset the password, do the following:
     1. Update the ZMQ Job Status Adapter's configuration.
     2. Restart the service.
     3. Update the curl command to use the new password.
     4. Rerun the curl command.

4. Check if Viewpoint can authenticate to Nitro Web Services. Follow the instructions in the 1.1.4 Configure Viewpoint for Nitro Web Services.

   If the TEST button indicates failure, then try the following curl command from the Nitro Web Services host, using the 'ws_readonly_username' and 'ws_readonly_password' defined in /opt/nitro-web-services/etc/nitro.cfg.

   [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-readonly-user", "password": "ChangeMe3!"}' \
    https://localhost:9443/auth

   Next steps:

   • If you get an HTTP status code other than 200 or 401, make sure the Nitro Web Services service is up and running.
   • If you get an HTTP status code of 200, and the username and password used in the curl command match the Nitro Services Configuration in Viewpoint > Configuration, the Viewpoint server is unable to communicate with the Nitro Web Services host. Login to the Viewpoint host and check if the Nitro Web Services host and port (i.e. 9443) is reachable (i.e. ping the host and/or use telnet or netcat to test port 9443). You might need to check firewall settings.

   • If you get an HTTP status code of 401, in Viewpoint, the configured "username" (i.e. nitro-readonly-user) is unable to authenticate.

     Do the following:

     1. Check the value of "ws_readonly_username" and "ws_readonly_password" in /opt/nitro-web-services/etc/nitro.cfg
     2. Set "username" and "password" in the Viewpoint "Nitro Services Configuration" so that they match.
     3. Retry the above curl command with the updated "username" and "password".

     4. If you still don't get an HTTP status code of 200, try resetting the nitro-readonly-user's password in MongoDB.

        # Any user can update his/her/it's own password.
        # The nitro-admin user can update any user's password.
         
        # Obtain a nitro-key (session token) by authenticating as either the nitro-admin or the nitro-readonly-user
        # Option 1: nitro-admin
        [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-admin", "password": "ChangeMe2!"}' https://localhost:9443/auth
        # Option 2: nitro-readonly-user
        [root@MoabHeadNode]# curl --insecure --data '{"username": "nitro-readonly-user", "password": "ChangeMe4!"}' https://localhost:9443/auth
         
        # Example nitro-admin authentication response:
        > {"status": 200, "data": {"nitro-key": "3e0fb95e9a0e44ae91daef4deb500dcc67a3714880e851d781512a49", "user": {"username": "nitro-admin", "last_updated": "2016-08-19 16:46:17.395000", "name": "Nitro Admin", "created": "2016-08-19 16:46:17.395000", "auth": {"job": ["read", "write", "delete"], "user": ["read", "write", "delete"]}}}}
         
        # Use the nitro-key from the authentication response to change nitro-readonly-user's password
        [root@MoabHeadNode]# curl --insecure -X PUT --header "nitro-key: 3e0fb95e9a0e44ae91daef4deb500dcc67a3714880e851d781512a49" --data '{"password": "Astr0ngPa$$!"}'  https://localhost:9443/user/nitro-readonly-user

© 2016 Adaptive Computing