TORQUE log files

11.3 TORQUE log files

The pbs_server keeps a daily log of all activity in the TORQUE_HOME/server_logs directory. The pbs_mom also keeps a daily log of all activity in the TORQUE_HOME/mom_logs/ directory. These logs contain information on communication between server and MOM as well as information on jobs as they enter the queue and as they are dispatched, run, and terminated. These logs can be very helpful in determining general job failures. For MOM logs, the verbosity of the logging can be adjusted by setting the $loglevel parameter in the mom_priv/config file. For server logs, the verbosity of the logging can be adjusted by setting the server log_level attribute in qmgr.

For both pbs_mom and pbs_server daemons, the log verbosity level can also be adjusted by setting the environment variable PBSLOGLEVEL to a value between 0 and 7. Further, to dynamically change the log level of a running daemon, use the SIGUSR1 and SIGUSR2 signals to increase and decrease the active loglevel by one. Signals are sent to a process using the kill command.

For example, kill -USR1 `pgrep pbs_mom` would raise the log level up by one.

The current loglevel for pbs_mom can be displayed with the command momctl -d3.