5.4 Job Scripts
This topic provides information about the different job scripts, including customization options (where applicable).
In this topic
The nitro_job.sh script is located in the /opt/nitro/etc/ directory.
Typically the nitro_job.sh script is customized by the system administrator and executed by the nitrosub command. This job script is used for static jobs, if resources are not in a range. For dynamic jobs, it sets up the initial resource request (minimum resource value in the range).
Alternatively, the system administrators can modify the nitro_job.sh script and then have authorized users copy and customize the script for their task file. This script is then executed using the resource manager's job submission command (for example, Torque's qsub). This functionality is similar to the Nitro functionality prior to version 2.1.
The nitro_job.sh script:
- Defines path to your task file (NITRO_TASK_FILE)
- Defines the directory to which Nitro writes log files (NITROJOBDIR). This directory can be used to store output files from your tasks.
- Executes the launch_nitro.sh script (last line in the script)
In the nitro_job.sh script, you can also customize the launch_nitro.sh script.
- NITROJOBID – Job ID used by Nitro. If not provided, this ID is based on the resource manager's job ID.
Unless you are restarting a job that partially completed and was canceled, you don't need to set this environment variable. If you specify this environment variable in the job's submission, it will override the resource manager job ID and Nitro will use the value you supplied.
If your job scheduler and resource manager use different numbering systems, the job ID that Nitro will use is the one that it gets from the resource manager. You may want to submit the job directly to the resource manager in this case to avoid confusion. Check with your system administrator to find out if your job scheduler's and resource manager's job ids are synchronized.
- Command line options – Any command line options you want passed to the launch_nitro.sh script must be contained in the NITRO_OPTIONS, NITRO_COORD_OPTIONS, or NITRO_WORKER_OPTIONS environment variables. See 5.2 Command Line Flags, or Options, and Positional Parameters or 5.3 Environment Variables for more information.
The worker_job.sh script is located in the /opt/nitro/etc/ directory.
The worker_job.sh script is executed only by the nitrosub command. This job script is used for the dynamic portion of dynamic jobs (resources after the initial request up to the maximum value).
- Defines the job ID for the coordinator set up by the nitro_job.sh script for the first part of the dynamic job submission (NITROJOBID).
- Defines the directory to which Nitro writes log files (NITROJOBDIR). This directory can be used to store output files from your tasks.
- Executes the launch_worker.sh script (last line in the script).
5.4.3 Moab/Torque Customization Commands and Options
If using Moab/Torque, be aware of the following:
- Nitro will consume all available OS cores on the nodes on which it runs. However, if the jobs are using the cgroups functionality (provided by the resource manager), then the coordinator and worker will only see as many cores as are in the cgroup.
- If Nitro is running exclusively on nodes, you may want to configure your system to run a single job per node (for example, in Moab, set NODEACCESSPOLICY to SINGLEJOB in the moab.cfg file or during job submission).
- If you will run other workload on the system that may run multiple jobs per node, you will need to provide a script or instruct users on how to specify their Nitro jobs to run with a single task per node configuration. In Moab and Torque the jobs should be submitted (using either msub or qsub) with the "-l nodes=<node count>:tpn=1" command line option.
Example: Setting Environment Variables at Job Submission
The "-v" option in Moab's msub command sets environment variables before calling the user job script. The user job script inherits these environment variables.
The user job script can add to or overwrite any environment variable defined in msub's "-v" option.
- User job script (/home/jdoe/user_job_script.sh)
This script must exist on the node where you execute the msub command. In this example, this user job script exists in jdoe's home directory.
exec /opt/nitro/bin/launch_nitro.sh
- msub example
msub -lnodes=5 -ltpn=1 -lwalltime=600 -v "NITRO_TASK_FILE=/home/jdoe/nitro/monte_sim.txt,NITRO_OPTIONS=--job-dir /home/jdoe/monte01,NITRO_COORD_OPTIONS=--run-local-worker" /home/jdoe/user_job_script.sh
- The task file monte_sim.txt contains the list of task command lines that you want to execute.
- The "‑‑job‑dir" sets the path to which Nitro will write the job and task logs.
- The last parameter is the path of the user job script.
Example: Setting Environment Variables In the User Job Script
-
User job script (/home/jdoe/user_job_script.sh)
This script must exist on the node where you execute the msub command. In this example, this user job script exists in jdoe's home directory.
NITRO_TASK_FILE=/home/jdoe/nitro/monte_sim.txt NITRO_OPTIONS="--job-dir /home/jdoe/monte01" NITRO_COORD_OPTIONS=--run-local-worker exec /opt/nitro/bin/launch_nitro.sh
- msub example
msub -lnodes=5 -ltpn=1 -lwalltime=600 /home/jdoe/user_job_script.sh
- The task file monte_sim.txt contains the list of task command lines that you want to execute.
- The "‑‑job‑dir" sets the path to which Nitro will write the job and task logs.
- The last parameter is the path of the user job script.
NITRO_TASK_FILE, NITRO_OPTIONS and NITRO_COORD_OPTIONS are set in the user job script.
Related Topics