TORQUE log files

11.3 TORQUE log files

pbs_server and pbs_mom log files

The pbs_server keeps a daily log of all activity in the TORQUE_HOME/server_logs directory. The pbs_mom also keeps a daily log of all activity in the TORQUE_HOME/mom_logs/ directory. These logs contain information on communication between server and MOM as well as information on jobs as they enter the queue and as they are dispatched, run, and terminated. These logs can be very helpful in determining general job failures. For MOM logs, the verbosity of the logging can be adjusted by setting the $loglevel parameter in the mom_priv/config file. For server logs, the verbosity of the logging can be adjusted by setting the server log_level attribute in qmgr.

For both pbs_mom and pbs_server daemons, the log verbosity level can also be adjusted by setting the environment variable PBSLOGLEVEL to a value between 0 and 7. Further, to dynamically change the log level of a running daemon, use the SIGUSR1 and SIGUSR2 signals to increase and decrease the active loglevel by one. Signals are sent to a process using the kill command.

For example, kill -USR1 `pgrep pbs_mom` would raise the log level up by one.

The current loglevel for pbs_mom can be displayed with the command momctl -d3.

trqauthd log files

As of TORQUE 4.1.3, trqauthd logs its events in the $TORQUE_HOME/client_logs directory. It names the log files in the format <YYYYMMDD>, creating a new log daily as events occur.

The trqauthd records the following events along with the date and time of the occurrence:

When trqauthd successfully starts. It logs the event with the IP address and port.
When a user successfully authenticates with trqauthd.
When a user fails to authenticate with trqauthd.
When trqauthd encounters any unexpected errors.

Example 11-1: trqauthd logging sample

2012-10-05 15:05:51.8404 Log opened

2012-10-05 15:05:51.8405 TORQUE authd daemon started and listening on IP:port 101.0.1.0:12345

2012-10-10 14:48:05.5688 User hfrye at IP:port abc:12345 logged in