TORQUE Resource Manager
Home Up Previous Next

Appendix C: Node Manager (MOM) Configuration

Under TORQUE, MOM configuration is accomplished using the mom_priv/config file located in the PBS directory on each execution server.

C.1 Parameters

tbody>
arch
Format: <STRING>
Description: specifies the architecture of the local machine. This information is used by the scheduler only.
Example: arch ia64
   
$alias_server_name
Format: <STRING>
Description: (Applicable in version 2.5.0 and later.) Allows the MOM to accept an additional pbs_server host name as a trusted address.

This feature was added to overcome a problem with UDP and RPP where alias IP addresses are used on a server. With alias IP addresses a UDP packet can be sent to the alias address but the UDP reply packet will come back on the primary IP address. RPP matches addresses from its connection table to incoming packets. If the addresses do not match an entry in the RPP table, the packet is dropped. This feature allows an additional address for the server to be added to the table so legitimate packets are not dropped.
Example: $alias_server_name node01
   
$clienthost
Format: <STRING>
Description: specifies the machine running pbs_server

Note This parameter is deprecated, use pbsserver

Example: $clienthost node01.teracluster.org
   
$check_poll_time
Format: <STRING>
Description: amount of time between checking running jobs, polling jobs, and trying to resend obituariess for jobs that haven't sent successfully. Default is 45 seconds.
Example: $check_poll_time 90
   
$configversion
Format: <STRING>
Description: specifies the version of the config file data
Example: $configversion 113
   
$cputmult
Format: <FLOAT>
Description: cpu time multiplier.

Note If set to 0.0, MOM level cputime enforcement is disabled.

Example: $cputmult 2.2
   
$exec_with_exec
Format: <BOOLEAN>
Description: pbs_mom uses the exec command to start the job script rather than the TORQUE default method, which is to pass the script's contents as the input to the shell. This means that if you trap signals in the job script, they will be trapped for the job. Using the default method, you would need to configure the shell to also trap the signals. Default is FALSE.
Example: $exec_with_exec true
   
$ideal_load
Format: <FLOAT>
Description: ideal processor load
Example: $ideal_load 4.0
   
$igncput
Format: <BOOLEAN>
Description: ignores limit violations pertaining to cpu time. Default is false.
Example: $igncput true
   
$ignmem
Format: <BOOLEAN>
Description: ignores limit violations pertaining to physical memory. Default is false.
Example: $ignmem true
   
$ignvmem
Format: <BOOLEAN>
Description: ignores limit violations pertaining to virtual memory. Default is false.
Example: $ignvmem true
   
$ignwalltime
Format: <BOOLEAN>
Description: ignore walltime (do not enable mom based walltime limit enforcement)
Example: $ignwalltime true
   
$job_output_file_umask
Format: <STRING>
Description: uses the specified umask when creating job output and error files. Values can be specified in base 8, 10, or 16; leading 0 implies octal and leading 0x or 0X hexadecimal. A value of "userdefault" will use the user's default umask. This parameter is in version 2.3.0 and later.
Example: $job_output_file_umask 027
   
$job_starter
Format: <STRING>
Description: specifies the fully qualified pathname of the job starter. If this parameter is specified, instead of executing the job command and job arguments directly, the MOM will execute the job starter, passing the job command and job arguments to it as its arguments. The job starter can be used to launch jobs within a desired environment.
Example:

$job_starter /var/torque/mom_priv/job_starter.sh
> cat /var/torque/mom_priv/job_starter.sh
#!/bin/bash
export FOOHOME=/home/foo
ulimit -n 314
$*
   
$log_directory
Format: <STRING>
Description: changes the log directory. Default is TORQUE_HOME/mom_logs/. TORQUE_HOME default is /var/spool/torque/ but can be changed in the ./configure script. The value is a string and should be the full path to the desired mom log directory.
Example: $log_directory /opt/torque/mom_logs/
   
$log_file_suffix
Format: <STRING>
Description: optional suffix to append to log file names. If %h is the suffix, pbs_mom appends the hostname for where
the log files are stored if it knows it, otherwise it will append the hostname where the mom is running.
Example: $log_file_suffix %h = 20100223.mybox
$log_file_suffix foo = 20100223.foo
   
$logevent
Format: <STRING>
Description: specifies a bitmap for event types to log
Example: $logevent 255
   
$loglevel
Format: <INTEGER>
Description: specifies the verbosity of logging with higher numbers specifying more verbose logging. Values may range between 0 and 7.
Example: $loglevel 4
   
$log_file_max_size
Format: <INTEGER>
Description: Soft limit for log file size in kilobytes. Checked every 5 minutes. If the log file is found to be greater than or equal to log_file_max_size the current log file will be moved from X to X.1 and a new empty file will be opened.
Example: $log_file_max_size = 100
   
$log_file_roll_depth
Format: <INTEGER>
Description: specifies how many times a log fill will be rolled before it is deleted.
Example: $log_file_roll_depth = 7
   
$log_keep_days
Format: <INTEGER>
Description: Specifies how many days to keep log files. pbs_mom deletes log files older than the specified number of days. If not specified, pbs_mom won't delete log files based on their age.
Example: $log_keep_days 10
   
$max_load
Format: <FLOAT>
Description: maximum processor load
Example: $max_load 4.0
   
$memory_pressure_duration
Format: <INTEGER>
Description: (Applicable in version 3.0 and later.) Memory pressure duration sets a limit to the number of times the value of memory_pressure_threshold can be exceeded before a process is terminated. This can only be used with $memory_pressure_threshold.
Example: $memory_pressure_duration 5
   
$memory_pressure_threshold
Format: <INTEGER>
Description: (Applicable in version 3.0 and later.) The memory_pressure of a cpuset provides a simple per-cpuset running average of the rate that the processes in a cpuset are attempting to free up in-use memory on the nodes of the cpuset to satisfy additional memory requests. The memory_pressure_threshold is an integer number used to compare against the reclaim rate provided by the memory_pressure file. If the threshold is exceeded and memory_pressure_duration is set, then the process terminates after exceeding the threshold by the number of times set in memory_pressure_duration. If memory_pressure duration is not set, then a warning is logged and the process continues. Memory_pressure_threshold is only valid with memory_pressure enabled in the root cpuset. To enable, log in as the super user and execute the command echo 1 >> /dev/cpuset/memory_pressure_enabled. See the cpuset man page for more information concerning memory pressure.
Example: $memory_pressure_threshold 1000
   
$node_check_script
Format: <STRING>
Description: specifies the fully qualified pathname of the health check script to run. (see Health Check for more information)
Example: $node_check_script /opt/batch_tools/nodecheck.pl
   
$node_check_interval
Format: <INTEGER>
Description:

specifies the number of MOM intervals between subsequent executions of the specified health check. This value default to 1 indicating the check is run every mom interval (see Health Check for more information).

$node_check_interval has two special strings that can be set:

  • jobstart - makes the node health script run when a job is started.
  • jobend - makes the node health script run after each job has completed on a node.
Example: $node_check_interval 5
   
$nodefile_suffix
Format: <STRING>
Description: Specifies the suffix to append to a host names to denote the data channel network adapter in a multihomed compute node.
Example: $nodefile_suffix i

With the suffix of 'i' and the control channel adapter with the name node01, the data channel would have a hostname of node01i.
   
$nospool_dir_list
Format: <STRING>
Description:

If this is configured, the job's output is spooled in the working directory of the job or the specified output directory.

Specify the list in full paths, delimited by commas. If the job's working directory (or specified output directory) is in one of the paths in the list (or a subdirectory of one of the paths in the list), the job is spooled directly to the output location. $nospool_dir_list * is accepted.

The user that submits the job must have write permission on the folder where the job is written, and read permission on the folder where the file is spooled.

Alternatively, you can use the $spool_as_final_name parameter to force the job to spool directly to the final output.

Note This should generally be used only when the job can run on the same machine as where the output file goes, or if there is a shared filesystem. If not, this parameter can slow down the system or fail to create the output file.
Example: $nospool_dir_list /home/mike/jobs/,/var/tmp/spool/
   
opsys
Format: <STRING>
Description: specifies the operating system of the local machine. This information is used by the scheduler only.
Example: opsys RHEL3
   
$pbsclient
Format: <STRING>
Description: specifies machines which the mom daemon will trust to run resource manager commands via momctl. This may include machines where monitors, schedulers, or admins require the use of this command.)
Example: $pbsclient node01.teracluster.org
   
$pbsserver
Format: <STRING>
Description: specifies the machine running pbs_server

Note This parameter replaces the deprecated parameter clienthost.

Example: $pbsserver node01.teracluster.org
   
$prologalarm
Format: <INTEGER>
Description: Specifies maximum duration (in seconds) which the mom will wait for the job prologue or job epilogue to complete. This parameter defaults to 300 seconds (5 minutes).
Example: $prologalarm 60
   
$rcpcmd
Format: <STRING>
Description: specifies the full path and optional additional command line args to use to perform remote copies
Example: mom_priv/config: 
$rcpcmd /usr/local/bin/scp -i /etc/sshauth.dat
   
$remote_reconfig
Format: <STRING>
Description: Enables the ability to remotely reconfigure pbs_mom with a new config file. Default is disabled. This parameter accepts various forms of true, yes, and 1. For more information on how to reconfigure MOMs, see momctl -r.
Example: $remote_reconfig true
   
$reduce_prolog_checks
Format: <STRING>
Description: If enabled, TORQUE will only check if the file is a regular file and is executable, instead of the normal checks listed on the prologue and epilogue page. Default is false.
Example: $reduce_prolog_checks true
   
$restricted
Format: <STRING>
Description: Specifies hosts which can be trusted to access mom services as non-root. By default, no hosts are trusted to access mom services as non-root.
Example: $restricted *.teracluster.org
   
$rpp_throttle
Format: <INTEGER>
Description: This integer is in microseconds and causes a sleep after every RPP packet is sent. It is for systems that experience job failures because of incomplete data.
Example: $rpp_throttle 100 (will cause a 100 microsecond sleep)
   
size[fs=<FS>]
Format: N/A
Description: Specifies that the available and configured disk space in the <FS> filesystem is to be reported to the pbs_server and scheduler.

Note To request disk space on a per job basis, specify the file resource as in 'qsub -l nodes=1,file=1000kb'.

Note Unlike most mom config options, the size parameter is not preceded by a '$' character.

Example: size[fs=/localscratch]

the available and configured disk space in the /localscratch filesystem will be reported.
   
$source_login_batch
Format: <STRING>
Description: Specifies whether or not mom will source the /etc/profile, etc. type files for batch jobs. Parameter accepts various forms of true, false, yes, no, 1 and 0. Default is True. This parameter is in version 2.3.1 and later.
Example: $source_login_batch False

mom will bypass the sourcing of /etc/profile, etc. type files.
   
$source_login_interactive
Format: <STRING>
Description: Specifies whether or not mom will source the /etc/profile, etc. type files for interactive jobs. Parameter accepts various forms of true, false, yes, no, 1 and 0. Default is True. This parameter is in version 2.3.1 and later.
Example: $source_login_interactive False

mom will bypass the sourcing of /etc/profile, etc. type files.
   
$spool_as_final_name
Format: <BOOLEAN>
Description: This will spool the job under the final name that the output and error files will receive, instead of having an intermediate file and then copying the result to the final file when the job has completed. This allows users easier access to the file if they want to watch the jobs output as it runs.
Example: 
$spool_as_final_name true
   
$status_update_time
Format: <INTEGER>
Description: Specifies the number of seconds between subsequent mom-to-server update reports. Default is 45 seconds.
Example: status_update_time: 
$status_update_time 120
mom will send server update reports every 120 seconds.
   
$thread_unlink_calls
Format: <BOOLEAN>
Description: Threads calls to unlink when deleting a job. Default is false. If it is set to TRUE, pbs_mom will use a thread to delete the job's files.
Example: thread_unlink_calls: 
$thread_unlink_calls true
   
$timeout
Format: <INTEGER>
Description: Specifies the number of seconds before mom-to-mom messages will timeout if RPP is disabled. Default is 60 seconds.
Example: $timeout 120

mom-to-mom communication will allow up to 120 seconds before timing out.
   
$tmpdir
Format: <STRING>
Description: Specifies a directory to create job-specific scratch space (see Creating Per-Job Temporary Directories
Example: $tmpdir /localscratch
   
$usecp
Format: <HOST>:<SRCDIR> <DSTDIR>
Description: Specifies which directories should be staged (see TORQUE Data Management)
Example: $usecp *.fte.com:/data /usr/local/data
   
$use_smt
Format: <BOOLEAN>
Description: Indicates that the user would like to use SMT. If set, each logical core inside of a physical core will be used as a normal core for cpusets.

Note If SMT is used, you will need to set the np attribute so that each logical processor is counted.

Example: $use_smt false
   
varattr
Format: <INTEGER> <STRING>
Description:

Provides a way to keep track of dynamic attributes on nodes.

<INTEGER> is how many seconds should go by between calls to the script to update the dynamic values. If set to -1, the script is read only one time.

<STRING> is the script path. This script should check for whatever dynamic attributes are desired, and then output lines in this format:

    name=value

Include any arguments after the script's full path.

These features are visible in the output of pbsnodes -a:

varattr=Matlab=7.1;Octave=1.0.
Example: varattr 25 /usr/local/scripts/nodeProperties.pl arg1 arg2 arg3
   
$wallmult
Format: <FLOAT>
Description: Sets a factor to adjust walltime usage by multiplying a default job time to a common reference system. It modifies real walltime on a per-MOM basis (MOM configuration parameters). The factor is used for walltime calculations and limits in the same way that cputmult is used for cpu time.

Note If set to 0.0, MOM level walltime enforcement is disabled.

Example: $wallmult 2.2
   

C.2 Node Features and Generic Consumable Resource Specification

Node features (a.k.a. node properties) are opaque labels which can be applied to a node. They are not consumable and cannot be associated with a value. (use generic resources described below for these purposes). Node features are configured within the global nodes file on the pbs_server head node and are not specified on a per node basis. This file can be used to specify an arbitrary number of node features.

Additionally, per node consumable generic resources may be specified using the format '<ATTR> <VAL>' with no leading dollar ('$') character. When specified, this information is routed to the scheduler and can be used in scheduling decisions. For example, to indicate that a given host has two tape drives and one node-locked matlab license available for batch jobs, the following could be specified:

mom_priv/config: 
$clienthost 241.13.153.7

tape    2
matlab  1

Dynamic Consumable Generic Resources

Dynamic consumable resource information can be routed in by specifying a value preceded by a exclamation point (i.e., '!') as in the example below. If the resource value is configured in this manner, the specified file will be periodically executed to load the effective resource value. (See section 2.5.3 of the 'PBS Administrator Guide' for more information)

mom_priv/config: 
$clienthost 241.13.153.7

tape    !/opt/rm/gettapecount.pl
matlab  !/opt/tools/getlicensecount.pl

C.3 Command-line Arguments

Below is a table of pbs_mom command-line startup flags.

Flag Description
a <integer> Alarm time in seconds.
c <file> Config file path.
C <directory> Checkpoint path.
d <directory> Home directory.
L <file> Logfile.
M <integer> MOM port to listen on.
p Perform 'poll' based job recovery on restart (jobs persist until associated processes terminate).
P On restart, deletes all jobs that were running on MOM (Available in 2.4.X and later).
q On restart, requeues all jobs that were running on MOM (Available in 2.4.X and later).
r On restart, kills all processes associated with jobs that were running on MOM, and then requeues the jobs.
R <integer> MOM 'RM' port to listen on.
S <integer> pbs_server port to connect to.
v Display version information and exit.
x Disable use of privileged port.
? Show usage information and exit.
*For more details on these command-line options, see the pbs_mom man page.

See Also

Home Up Previous Next