Appendix B: Server Parameters

acl_hosts

acl_host_enable

acl_logic_or

allow_node_submit

allow_proxy_user

auto_node_np

automatic_requeue_exit_code

checkpoint_defaults

clone_batch_delay

clone_batch_size

copy_on_rerun

cray_enabled

$cuda_visible_devices

default_queue

disable_server_id_check

display_job_server_suffix

interactive_jobs_can_roam

job_exclusive_on_use

job_force_cancel_time

job_log_file_max_size

job_log_file_roll_depth

job_log_keep_days

job_nanny

job_stat_rate

job_start_timeout

job_sync_timeout

keep_completed

lock_file

lock_file_update_time

lock_file_check_time

log_events

log_file_max_size

log_file_roll_depth

log_keep_days

log_level

mail_body_fmt

mail_domain

mail_from

mail_subject_fmt

managers

max_job_array_size

max_slot_limit

max_threads

max_user_queuable

min_threads

moab_array_compatible

mom_job_sync

next_job_number

node_check_rate

node_pack

node_ping_rate

no_mail_force

np_default

operators

pass_cpuclock

poll_jobs

query_other_jobs

record_job_info

record_job_script

resources_available

scheduling

submit_hosts

tcp_timeout

thread_idle_seconds

Specifies a list of hosts which can have access to pbs_server when acl_host_enable is set to TRUE. This does not enable a node to submit jobs. To enable a node to submit jobs use submit_hosts.

Hosts which are in the $TORQUE_HOME/server_priv/nodesfile do not need to be added to this list.

Qmgr: set queue batch acl_hosts="hostA,hostB"
Qmgr: set queue batch acl_hosts+=hostC
Qmgr: set server acl_hosts="hostA,hostB"
Qmgr: set server acl_hosts+=hostC

In version 2.5 and later, the wildcard (*) character can appear anywhere in the host name, and ranges are supported; these specifications also work for managers and operators.

Qmgr: set server acl_hosts = "galaxy*.tom.org"
Qmgr: set server acl_hosts += "galaxy[0-50].tom.org"

When set to TRUE, allows all hosts that are in the $TORQUE_HOME/server_priv/nodes file (MOM nodes) to submit jobs to pbs_server.

Specifies for a queue the default checkpoint values for a job that does not have checkpointing specified. The checkpoint_defaults parameter only takes effect on execution queues.

When set to TRUE, Moab HPC Suite will copy the output and error files over to the user-specified directory when the grerun command is executed (i.e. a job preemption). Output and error files are only created when a job is in running state before the preemption occurs.

pbs_server and pbs_mom need to be on the same version.

When you change the value, you must perform a pbs_server restart for the change to effect.

When set to TRUE, makes it so the user for the job doesn't have to exist on the server. The user must still exist on all the compute nodes or the job will fail when it tries to execute.

If you have disable_server_id_check set to TRUE, a user could request a group to which they do not belong. Setting VALIDATEGROUP to TRUE in the torque.cfg file prevents such a scenario (see Appendix K: "torque.cfg" Configuration File).

When set to TRUE, TORQUE will display both the job ID and the host name. When set to FALSE, only the job ID will be displayed.

If set to FALSE, the environment variable NO_SERVER_SUFFIX must be set to TRUE for pbs_track to work as expected.

If the mother superior has not sent an update by the specified time, at the specified time pbs_server requests an update on job status from the mother superior.

---

If you ran torque.setup on TORQUE installation, the default is 300.

The amount of time a job will be kept in the queue after it has entered the completed state. keep_completed must be set for job dependencies to work.

For more information, see Keeping Completed Jobs.

Specifies the name and location of the lock file used to determine which high availability server should be active.

If a full path is specified, it is used verbatim by TORQUE. If a relative path is specified, TORQUE will prefix it with torque/server_priv.

By default, all events are logged. However, you can customize things so that only certain events show up in the log file. These are the bitmaps for the different kinds of logs:

#define PBSEVENT_ERROR 0x0001 /* internal errors */
#define PBSEVENT_SYSTEM 0x0002 /* system (server) events */
#define PBSEVENT_ADMIN 0x0004 /* admin events */
#define PBSEVENT_JOB 0x0008 /* job related events */
#define PBSEVENT_JOB_USAGE 0x0010 /* End of Job accounting */
#define PBSEVENT_SECURITY 0x0020 /* security violation events */
#define PBSEVENT_SCHED 0x0040 /* scheduler events */
#define PBSEVENT_DEBUG 0x0080 /* common debug messages */
#define PBSEVENT_DEBUG2 0x0100 /* less needed debug messages */
#define PBSEVENT_FORCE 0x8000 /* set to force a message */

If you want to log only error, system, and job information, use qmgr to set log_events to 11:

Override the default format for the body of outgoing mail messages. A number of printf-like format specifiers and escape sequences can be used:

\n new line
\t tab
\\ backslash
\' single quote
\" double quote
%d details concerning the message
%h PBS host name
%i PBS job identifier
%j PBS job name
%m long reason for message
%r short reason for message
%% a single %

Override the default format for the subject of outgoing mail messages. A number of printf-like format specifiers and escape sequences can be used:

\n new line
\t tab
\\ backslash
\' single quote
\" double quote
%d details concerning the message
%h PBS host name
%i PBS job identifier
%j PBS job name
%m long reason for message
%r short reason for message
%% a single %

This is the maximum number of jobs that can run concurrently in any job array. Slot limits can be applied at submission time with qsub, or it can be modified with qalter.

No array can request a slot limit greater than 10. Any array that does not request a slot limit receives a slot limit of 10. Using the example above, slot requests greater than 10 are rejected with the message: "Requested slot limit is too large, limit is 10."

When set, max_user_queuable places a system-wide limit on the amount of jobs that an individual user can queue.

When set to TRUE, specifies that the pbs_server will synchronize its view of the job queue and resource allocation with compute nodes as they come online. If a job exists on a compute node, it will be automatically cleaned up and purged. (Enabled by default in TORQUE 2.2.0 and higher.)

Jobs that are no longer reported by the mother superior are automatically purged by pbs_server. Jobs that pbs_server instructs the MOM to cancel have their processes killed in addition to being deleted (instead of leaving them running as in versions of TORQUE prior to 4.1.1).

Specifies the ID number of the next job. If you set your job number too low and TORQUE repeats a job number that it has already used, the job will fail. Before setting next_job_number to a number lower than any number that TORQUE has already used, you must clear out your .e and .o files.

If you use Moab Workload Manager and have configured it to synchronize job IDs with TORQUE), then Moab will generate the job ID and next_job_number will have no effect on the job ID. See Resource Manager Configuration for more information.

If set to TRUE, the pbs_server daemon passes the option and its value to the pbs_mom daemons for direct implementation by the daemons, making the CPU frequency adjustable as part of a resource request by a job submission.

If set to FALSE, the pbs_server daemon creates and passes a PBS_CPUCLOCK job environment variable to the pbs_mom daemons that contains the value of the cpuclock attribute used as part of a resource request by a job submission. The CPU frequencies on the MOMs are not adjusted. The environment variable is for use by prologue and epilogue scripts, enabling administrators to log and research when users are making cpuclock requests, as well as researchers and developers to perform CPU clock frequency changes using a method outside of that employed by the TORQUE pbs_mom daemons.

If set to TRUE, pbs_server will poll job info from MOMs over time and will not block on handling requests which require this job information.

If set to FALSE, no polling will occur and if requested job information is stale, pbs_server may block while it attempts to update this information. For large systems, this value should be set to TRUE.

If set to TRUE, this adds the contents of the script executed by a job to the log.

For record_job_script to take effect, record_job_info must be set to TRUE.

Hosts in this list are able to submit jobs. This applies to any node whether within the cluster or outside of the cluster.

If acl_host_enable is set to TRUE and the host is not in the $TORQUE/server_priv/nodes file, then the host must also be in the acl_hosts list..

Specifies the timeout for idle TCP connections. If no communication is received by the server on the connection after the timeout, the server closes the connection. There is an exception for connections made to the server on port 15001 (default); timeout events are ignored on the server for such connections established by a client utility or scheduler. Responsibility rests with the client to close the connection first (See Appendix F: Large Cluster Considerations for additional information.).

If you use Moab Workload Manager, prevent communication errors by giving tcp_timeout at least twice the value of the Moab RMPOLLINTERVAL.