5.616 Compute Node Health Check

Torque provides the ability to perform health checks on each compute node. If these checks fail, a failure message can be associated with the node and routed to the scheduler. Schedulers (such as Moab) can forward this information to administrators by way of scheduler triggers, make it available through scheduler diagnostic commands, and automatically mark the node down until the issue is resolved. See the RMMSGIGNORE parameter in Moab Parameters in the Moab Workload Manager Administrator Guide for more information.

Additionally, Michael Jennings at LBNL has authored an open-source bash node health check script project. It offers an easy way to perform some of the most common node health checking tasks, such as verifying network and filesystem functionality. More information is available on the project's page.

For more information about node health checks, see these topics:

Configuring MOMs to Launch a Health Check
Creating the Health Check Script
Adjusting Node State Based on the Health Check Output
Example Health Check Script