You are here: Using Nitro > Submit a Nitro Job

4.2 Submit a Nitro Job to a Scheduler

This topic provides information about submitting a Nitro job and executing Nitro. The user creates the user job script and the system administrator sets up and configures the Nitro launch script, which the user job script executes to run Nitro.

Adaptive Computing's objective for Nitro is two-fold; first, allow the user to use the same job script for a Nitro job regardless of the scheduler used by a system to schedule jobs and, second, allow the administrator to customize Nitro for the system based on the scheduler and/or resource manager the system uses. This means the user job script can be scheduler-agnostic while the Nitro launch script (executed by the user job script) must be scheduler- or resource manager-specific.

In this topic:

4.2.1 User Job Script
4.2.2 Nitro Launch Script
4.2.3 Customize the Nitro Launch Script
4.2.4 Launch Script Environment Variables
4.2.5 Moab/Torque Customization Commands and Options

4.2.1 User Job Script

Typically, the system administrator provides a sample Nitro job script to users to customize as needed.

A sample user job script is located in the Nitro installation's "samples" subdirectory.

Users can set launch script environment variables at the user job script-level to affect Nitro's behavior. See 4.2.4 Launch Script Environment Variables for a list of available variables.

The user job script also contains the information to execute the Nitro launch script (/opt/nitro/bin/launch_nitro.sh).

4.2.2 Nitro Launch Script

Adaptive Computing provides a configured Nitro launch script for different schedulers or resource managers. You can also build a new script. See 5.2 Resource Manager Launch Scripts.

During the installation of Nitro, the system administrator copies the Nitro launch script ( /opt/nitro/scripts directory) to the directory (/opt/nitro/bin) from where the user job script executes it.

4.2.3 Customize the Nitro Launch Script

The system administrator may customize the Nitro launch script to suit the needs of the system. The Nitro launch script may examine and modify, or simply pass through, the command line options specified by the environment variables set by the user job script. The command line options are then passed via the environment variables to the Nitro workers and coordinator that are started by the Nitro launch script.

4.2.4 Launch Script Environment Variables

The Nitro launch script looks for several environment variables. Users can also customize these variables in the user job script.

NITRO_TASK_FILE
NITROJOBID
NITRO_OPTIONS
NITRO_COORD_OPTIONS
NITRO_WORKER_OPTIONS.

NITRO_TASK_FILE

The NITRO_TASK_FILE environment variable must contain the path to the task file, and the task file must be accessible to the coordinator host that will run the job.

Users should put the task file in a shared location accessible by any host in the system.

NITROJOBID

The NITROJOBID environment variable is created automatically by the launch script, based on the resource manager's job ID, if the user job script does not provide one.

Unless you are restarting a job that partially completed and was canceled, you don't need to set this environment variable. If you specify this environment variable in the job's submission, it will override the resource manager job ID and Nitro will use the value you supplied.

If your job scheduler and resource manager use different numbering systems, the job ID that Nitro will use is the one that it gets from the resource manager. You may want to submit the job directly to the resource manager in this case to avoid confusion. Check with your system administrator to find out if your job scheduler's and resource manager's job ids are synchronized.

NITRO_OPTIONS, NITRO_COORD_OPTIONS, or _NITRO_WORKER_OPTIONS

Any command line options the user job script needs to pass to Nitro must be contained in either the NITRO_OPTIONS, NITRO_COORD_OPTIONS, or NITRO_WORKER_OPTIONS environment variables.

4.2.5 Moab/Torque Customization Commands and Options

If using Moab/Torque, be aware of the following:

Nitro will consume all available OS cores on the hosts on which it runs.
- If Nitro is running exclusively on hosts, you may want to configure your system to run a single job per host (for example, in Moab, set NODEACCESSPOLICY to SINGLEJOB in the moab.cfg file).
- If you will run other workload on the system that may run multiple jobs per host, you will need to provide a script or instruct users on how to specify their Nitro jobs to run with a single task per host configuration. In Moab and Torque the jobs should be submitted (using either msub or qsub) with the "-l nodes=<node count>:tpn=1" command line option.

4.2.5.A Examples

The sample launch_nitro.sh job scripts provided in /opt/nitro/scripts/<resource manager>/ all expect environment variables. You can set environment variables for the Nitro launch script at job submission or define them in the user job script.

Example: Setting Environment Variables at Job Submission

The "-v" option in Moab's msub command sets environment variables before calling the user job script. The user job script inherits these environment variables.

The user job script can add to or overwrite any environment variable defined in msub's "-v" option.

User job script (/home/jdoe/user_job_script.sh)
This script must exist on the host where you execute the msub command. In this example, this user job script exists in jdoe's home directory.
```
exec /opt/nitro/bin/launch_nitro.sh
```
msub example
```
msub -lnodes=5 -ltpn=1 -lwalltime=600 -v "NITRO_TASK_FILE=/home/jdoe/nitro/monte_sim.txt,NITRO_OPTIONS=--job-dir /home/jdoe/monte01,NITRO_COORD_OPTIONS=--run-local-worker" /home/jdoe/user_job_script.sh
```
- The task file monte_sim.txt contains the list of task command lines that you want to execute.
- The "‑‑job‑dir" sets the path to which Nitro will write the job and task logs.
- The last parameter is the path of the user job script.

Example: Setting Environment Variables In the User Job Script

User job script (/home/jdoe/user_job_script.sh)

This script must exist on the host where you execute the msub command. In this example, this user job script exists in jdoe's home directory.
```
NITRO_TASK_FILE=/home/jdoe/nitro/monte_sim.txt
NITRO_OPTIONS="--job-dir /home/jdoe/monte01"
NITRO_COORD_OPTIONS=--run-local-worker
exec /opt/nitro/bin/launch_nitro.sh
```
msub example
```
msub -lnodes=5 -ltpn=1 -lwalltime=600 /home/jdoe/user_job_script.sh
```
- The task file monte_sim.txt contains the list of task command lines that you want to execute.
- The "‑‑job‑dir" sets the path to which Nitro will write the job and task logs.
- The last parameter is the path of the user job script.
- NITRO_TASK_FILE, NITRO_OPTIONS and NITRO_COORD_OPTIONS are set in the user job script.