2.1 Prepare a Nitro Job
This topic provides information on the Nitro job's task file and performance tuning information.
In this topic:
The task file is a single file that contains a list of tasks to execute. Each line of the task file should contain only one task. You can add comments to your task file to help describe the tasks being performed, the data required, or other information that is pertinent to describing the tasks. Nitro also provides the capability to use task names and labels to help you organize your tasks.
Most of the tasks you create for a task file will probably run to completion fairly quickly, but it is possible that a task gets stuck in a loop or needs to run for a certain amount of time. Nitro by default limits tasks to 3,600 seconds (1 hour), but you can specify the limit to apply to the task by using the "maxtime" token. Time limits are specified in seconds. The following is an example of a task definition that limits a task to 30 seconds.
name=S23T01 maxtime=30 cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 1
2.1.1.A Tasks
A task line can be as simple as the command you want to execute. For example, if you want to run a program called "framegen", input a file from a shared directory, and process the frame starting at time index "0" (zero), the command line might look like as follows.
/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 0
Nitro uses name/value pairs before the command line that you want to execute to define Nitro-specific information, such as, specifying a task name, task labels (that you can use to categorize the task), maximum time a task will run, and the command to execute to run the tasks itself. The key words for these name/value pairs are:
"cores=<count>""env=<name=value>[,<name=value>,...]"
"labels=<label>[,<label>,...]"
"name=<task name>""maxtime=<time limit in seconds>""memory=<amount>""shell=[default | none | <shell path>]""cmd=<command line>"
The optional name/value pairs must be prepended to the line containing the task command line. As soon as Nitro sees something that isn't a name/value pair, the task line parsing stops and the rest is assumed to be part of the command line to execute.
To make it clear where the task options end and your command line begins, include "cmd=" before your task's command line. This token is optional but helps to make the task definition easier to read when you are specifying other options. The following is an example command line with the "cmd=" token.
cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 0
Nitro organizes the tasks for tracking.
Nitro tracks tasks by a task ID and line number. Nitro automatically generates a task ID for each task definition in the task file. The first task definition receives task ID "1".
Only a task definition will increment the task ID. Because a task file can have empty or comment lines, the task ID and the line number in the task file may not be the same for the task.
Nitro will create a report of all tasks run and will include the task ID and line number in this report. The task ID is passed to the task via the $NITROTASKID environment variable.
To make Nitro tasks easier to track, or to search for specific tasks in the task completion report, add a unique task name to your task definition. Task names don't have to be unique, but creating a unique task name helps you identify specific tasks.
You can use any naming scheme you want, as long as the name does not include spaces (which would indicate an end to the name/value pair).
For example, if you are processing data for scenes 21, 22, and 23, you can name the tasks according to scene and time index.
name=S21T00 cmd=/opt/framemaker/bin/framegen -i /shared/scene21.def -tindex 0 name=S21T01 cmd=/opt/framemaker/bin/framegen -i /shared/scene21.def -tindex 1 name=S22T00 cmd=/opt/framemaker/bin/framegen -i /shared/scene22.def -tindex 0 name=S22T01 cmd=/opt/framemaker/bin/framegen -i /shared/scene22.def -tindex 1 name=S23T00 cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 0 name=S23T01 cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 1
Nitro makes the task name available to the task via the $NITROTASKNAME environment variable when it executes the task. If the task command line includes the environment variable, it is substituted by its value before the command executes.
You can also use task labels to organize or identify the tasks. You can use multiple labels to describe a task. Multiple label values are separated by a comma between them; spaces are not allowed.
For example, if scene 22 contains a green screen that needs additional processing after this job completes, you can include the label "green" on all of the tasks for this scene.
name=S21T00 cmd=/opt/framemaker/bin/framegen -i /shared/scene21.def -tindex 0 name=S21T01 cmd=/opt/framemaker/bin/framegen -i /shared/scene21.def -tindex 1 name=S22T00 labels=green /opt/framemaker/bin/framegen -i /shared/scene22.def -tindex 0 name=S22T01 labels=green /opt/framemaker/bin/framegen -i /shared/scene22.def -tindex 1 name=S23T00 cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 0 name=S23T01 cmd=/opt/framemaker/bin/framegen -i /shared/scene23.def -tindex 1
2.1.2.A Assignment Size
Each set of tasks that a coordinator sends to a worker is called an assignment. Nitro is most efficient when it can send a large enough assignment to each worker to keep the worker busy for at least 10 seconds before requesting more work. On the other hand, if you have a heterogeneous set of nodes with a wide variance in performance characteristics, you don't want one worker taking a very long time completing its assignment after all of the other workers have finished.
The nitro.cfg file lets you specify an assignment size of 0 up to 1000. A size of 0 allows the assignment size to be calculated by the coordinator and dynamically calculated to target an assignment duration of 5 seconds.
You can also use the ‑‑assignment‑size command line option on the coordinator to change the tasks per assignment for your configuration. You can specify an assignment size as "0" (calculated by the coordinator), as small as "1" (which could be useful for tasks that need to use all of the available OS cores), and as large as "1000" (useful to keep worker cores busy with tasks of extremely short duration).
If submitting a job to a job scheduler, you can change the assignment size by setting the NITRO_COORD_OPTIONS environment variable so it contains the ‑‑assignment‑size command line option.
For example, if your nodes are all running 16 OS cores and each task takes 2 seconds to complete, each assignment of default size will take 31.25 seconds to complete (250 tasks at 2 seconds each divided by 16 OS cores), so you might want to change the assignment size to "80" to get an assignment time of closer to 10 seconds with the command line option.
--assignment-size 80
Assignment sizes don't need to be evenly divisible by the number of OS cores available. Nitro will try to send the worker a second assignment when the worker gets about half way done with the current assignment so the second assignment will start running tasks as soon as an OS core becomes idle from the previous assignment.
Adaptive Computing recommends a 10-20 second assignment duration to optimize node utilization and to prevent "tailing" jobs (job where at its end there is only one or a few workers executing a large assignment and other workers are idle).
2.1.2.B Thread Control
Nitro typically runs one task per available OS core on each worker. However, you can configure Nitro to run more tasks than OS cores (over-subscription), fewer tasks than OS cores (under-subscription) or a specific number of OS cores. You might want to over-subscribe the available OS cores if you are not utilizing the full capacity of the node. You may need to under-subscribe cores if background tasks are running on the nodes. To over- or under-subscribe, use the ‑‑thread‑ratio command line option.
--thread-ratio <ratio>
<ratio> only applies to worker nodes. However, if you are using the ‑‑run‑local‑worker command line option, then the thread ratio will be passed on to the coordinator's local worker.
‑‑thread‑ratio also lets you specify over- or under-subscription properly in a heterogeneous node environment where nodes have different numbers of processors, cores, or threads. For example, if your nodes are all single socket, oct-core with hyper-threading enabled (16 total OS cores), but you want to over-subscribe by a factor of 1.5x, you could accomplish this by adding "‑‑thread‑ratio 1.5" to the worker command line to give each worker the ability to run 24 concurrent tasks. Alternately, if your tasks are all designed to use 2 OS cores each (multi-threaded application), you could use "‑‑thread‑ratio 0.5".
There may also be cases where you want to specify the exact number of OS cores to be used by the worker, such as when you have tasks that will use all available OS cores. In that case, you would use the ‑‑thread‑count command line option to specify a thread count of 1 (‑‑thread‑count 1).
2.1.2.C Run a Worker on the Coordinator Node
In configurations where you will be running less than 20 worker nodes, the coordinator node may be underutilized. To remedy this situation you may want to run a worker on the coordinator node so you can use its resources more effectively. To run a worker on the coordinator node, include the ‑‑run‑local‑worker flag on the coordinator's command line or you can explicitly start a worker Nitro process on the node.
If using the nitrosub command, use ‑‑no‑local‑worker to prevent a worker from running on a coordinator node.
Nitro will calculate the number of threads that the local worker should run so the coordinator is not starved for CPU cycles; causing it to slow down all the other workers.
2.1.2.D Task Execution Environment Variables
Nitro will pass several environment variables to your tasks when it executes them. See 3.3 Environment Variables for more information.
Nitro reads a portion of the task file at a time. While the Nitro job is running, do not add or remove any task definitions or comment lines in the task file. Changes to the task file could cause line numbers to be changed and jobs to not run or be accidentally rerun.
Related Topics