2.4 Track Job Progress
This topic provides information on viewing job progress and output.
In this topic:
2.4.1 Introduction on How Nitro Tracks the Job
Nitro will print some job information to stdout, such as what workers attached, how many tasks have been run, if any tasks failed, etc.
If your Nitro job is submitted through a scheduler, you may not see any of this until the job has completed and the resource manager has copied the job output to your job's submission directory.
However, Nitro provides a tool called nitrostat to display status information while the job is running. nitrostat is located in the nitro/bin directory where Nitro was installed.
Nitro creates two files that you can use to stay up-to-date on the progress of your job.
- nitro_<jobid>.joblog.txt - Information about the job in general.
- nitro_<jobid>.tasklog.txt - Listing of individual tasks that have completed along with performance statistics collected from running the task (duration and memory usage) and the task output to stdout and/or stderr.
Both files are written to the job directory that you provide using the ‑‑job‑dir command line option when submitting your job, or to the default job directory $HOME/nitro/<jobid>.
To see job status using nitrostat, you will need the job ID. The job ID is the job ID reported to you when you submitted the job to the schuleder or that you set manually via the ‑‑job‑id command line option in the NITRO_OPTIONS environment variable or via the NITROJOBID environment variable.
- The default location for the job and task logs are in your "$HOME/nitro/<jobID>" directory.
- You can also use the "‑‑job‑dir" command line option to specify a different job directory if you are not using the default location.
2.4.2.A Nitro Job Progress Report
The Nitro job progress reports lets you see the current contents of a job log file.
For example, let's say you have a job that was run by your resource manager as job "23576", running "/opt/nitro/bin/nitrostat 23576"shows you the job's progress.
Nitro Job Progress Report
Start Time : 2016-02-10 09:10:11-0600
Current Time: 2016-02-10 09:10:42-0600
Elapsed Time: 31 seconds (00:00:31)
Job Id : 23576
Coordinator : node01
Load Pct : 5.6%
Task Log : /home/jdoe/jobs/23576/nitro_23576.tasklog.txt
Task File : /home/jdoe/jobs/survey03.tasks
File Size : 123366
Est Tasks : 3016
Processed : 75%
Tasks
------------
Pending : 500
In Progress : 500
Completed : 1250
Success : 1250
Failure : 0
InsufRes : 0
Timeout : 0
Invalid : 0
Tasks/sec : 40.3
Total Tasks : 2250
Workers
-------
Host Pid Thrds Status Assigned Running Completed Success Failure InsufRes Timeout Tasks/sec AsgmtDur
node02 6851 12 running 1250 250 1000 1000 0 0 0 36.0 8.0
node03 14988 4 running 500 250 250 250 0 0 0 9.3 27.0
The following describes the fields and their output descriptions.
- Start Time – Date and time the coordinator started running.
- Current Time – Current date and time the report was generated (reports are generated every 5 seconds).
- Elapsed Time – Amount of time the coordinator has been working on the tasks.
- Job Id – Job ID that Nitro was passed on its command line. Typically assigned by the resource manager, but can be assigned by the user.
- Coordinator – Host name on which the coordinator is running.
-
Load Pct – Percentage of coordinator load capacity.
- Task Log – Path and file name of the task log file that is generated by the coordinator.
- Task File – Path and file name of the task file.
- File Size – File size of the task file.
- Est Tasks – Number of tasks the coordinator estimates in the task file. Since Nitro doesn't read the entire task file on startup, an estimate is given based on lines read from the file so far.
- Processed– Percentage of the task file that has been read by the coordinator.
- Tasks Section: Lists the counts of tasks in each category
- Pending – Number of tasks that have been put into assignments and are waiting to be sent to a worker.
- In Progress– Number of tasks in assignments sent to the workers for which workers have not yet returned results.
- Completed – Number of tasks in assignments that have been completed (workers have returned results).
- Success – Number of completed tasks that were successful (the task returned an exit code of 0).
- Failure – Number of completed tasks that returned an exit code other than 0.
- InsufRes – Number of tasks that could not be run because the requested resources for the task were not available.
- Timeout – Number of completed tasks that ran longer than the task "maxtime" parameter and were terminated by the worker.
- Invalid – Number of task definitions that contained errors and could not be run.
- Tasks/sec – Number of tasks per second based on the time that the coordinator sends the first assignment until the time the report is generated. If in linger mode, this will only be calculated for the last 60 seconds.
- Total Tasks – Total number of tasks including completed and invalid tasks.
- Workers Section: List by worker
- Host – Host name and port (if not the default port) of the worker.
- Pid – Process ID of the worker.
- Thrds –Number of task launch threads the worker is using to run tasks.
- Status – Status of the worker. This may be "unconnected", "running", "unresponsive", "closing", or "closed".
- Assigned – Number of tasks assigned to this worker so far.
- Running – Number of tasks in assignments currently allocated to the worker.
- Completed – Number of tasks in assignments the worker has completed.
- Success – Number of successfully completed tasks.
- Failure – Number of tasks that returned an exit code other than 0.
- InsufRes – Number of tasks that could not be run because the requested resources for the task were not available.
- Timeout – Number of tasks that exceeded the tasks "maxtime" threshold and were terminated by the worker.
- Tasks/sec – Number of tasks per second that the worker has completed so far. In linger mode this is only calculated for the last 60 seconds.
- AsgmtDur – Average assignment duration in seconds.
2.4.2.B Job Completed Report
Once the job has completed, the job report will show "(final)" on the end of the first line of the report and Current Time is replaced with Finish Time (after Start Time). The following example is based on the previous example for job "23576" .
Nitro Job Progress Report (final)
Start Time : 2016-02-10 09:10:11-0600
Finish Time : 2016-02-10 09:11:36-0600
Elapsed Time: 85 seconds (00:01:25)
Job Id : 23576
Task Log : /home/jdoe/jobs/23576/nitro_23576.tasklog.txt
Task File : /home/jdoe/jobs/survey03.tasks
Tasks
------------
Pending : 0
Running : 0
Completed : 3000
Success : 3000
Failure : 0
InsufRes : 0
Timeout : 0
Invalid : 0
Tasks/sec : 35.3
Total Tasks : 3000
Coordinator
-----------
Host : node01
Threads : 8
Worker Resources
----------------
Workers : 2
Threads : 16
Workers
-------
Host Pid Thrds Status Assigned Running Completed Success Failure InsufRes Timeout Tasks/sec AsgmtDur
node02 6851 12 closed 2250 0 2250 2250 0 0 0 29.2 8.3
node03 14988 4 closed 750 0 750 750 0 0 0 8.8 35.7
The task log file contains a listing of all tasks that have been completed and some statistics about the tasks duration and memory consumption. This file is named nitro_<JobID>.tasklog.txt and is located in the same directory as the job log file.
The task log file is tab-delimited, so you can easily import it into a spreadsheet or database, or process it using another program. You can also view the task log using the nitrostat utility.
JobID TaskID Line Name Status ExitCode Hostname StartTime Duration UserCPU SystemCPU VirtualMem PhysicalMem Labels Output
foo 1 1 task001 Success 0 localhost:10004 2015-06-18_15:26:52.954-0600 1.005 0.000 0.000 7364608 630784 foo,foobar,foobaz,xyz
foo 2 2 task002 Success 0 localhost:10004 2015-06-18_15:26:52.954-0600 1.007 0.000 0.000 87834368 630784 foo,foobar,xyz
foo 3 3 task003 Success 0 localhost:10004 2015-06-18_15:26:52.954-0600 1.005 0.000 0.000 71728640 901120 foo,xyz
foo 4 4 task004 Success 0 localhost:10004 2015-06-18_15:26:52.955-0600 1.005 0.000 0.000 38837504 630784 foo,foobar,foobaz,abc
foo 5 5 task005 Success 0 localhost:10004 2015-06-18_15:26:53.960-0600 1.004 0.000 0.000 405946368 630784 foo,foobar,abc
foo 6 6 task006 Success 0 localhost:10004 2015-06-18_15:26:53.961-0600 1.005 0.000 0.000 405946368 946176 foo,abc
foo 7 7 task007 Success 0 localhost:10004 2015-06-18_15:26:53.961-0600 1.003 0.000 0.000 405946368 630784
foo 8 8 task008 Success 0 localhost:10004 2015-06-18_15:26:53.966-0600 1.003 0.000 0.000 405946368 700416
foo 9 9 task009 Success 0 localhost:10004 2015-06-18_15:26:54.965-0600 1.005 0.000 0.000 405946368 630784
foo 10 10 task010 Success 0 localhost:10004 2015-06-18_15:26:54.965-0600 1.003 0.000 0.000 405946368 630784
foo 11 11 task011 Success 0 localhost:10004 2015-06-18_15:26:55.973-0600 1.005 0.000 0.000 7364608 630784
foo 12 12 Success 0 localhost:10004 2015-06-18_15:26:55.973-0600 1.004 0.000 0.000 405946368 626688
foo 13 14 fail Failure 1 localhost:10004 2015-06-18_15:26:55.973-0600 0.005 0.000 0.000 8192 4096
foo 14 16 stderr Success 0 localhost:10004 2015-06-18_15:26:55.974-0600 0.005 0.000 0.000 405946368 536576
foo 15 18 stderr_fail Failure 1 localhost:10004 2015-06-18_15:26:55.979-0600 0.005 0.000 0.000 405946368 1228800 ERROR MESSAGE
foo 16 20 overtime Timeout -9 localhost:10004 2015-06-18_15:26:55.980-0600 2.006 0.000 0.000 405946368 970752 maxtime exceeded, process was killed
foo 17 21 Success 0 localhost:10004 2015-06-18_15:26:55.985-0600 1.002 0.000 0.000 405946368 626688
foo 19 23 Success 0 localhost:10004 2015-06-18_15:26:56.979-0600 1.007 0.000 0.000 405946368 970752
foo 20 24 Success 0 localhost:10004 2015-06-18_15:26:56.988-0600 1.003 0.000 0.000 405946368 724992
foo 21 25 Success 0 localhost:10004 2015-06-18_15:26:57.986-0600 1.005 0.000 0.000 405946368 724992
foo 22 26 Success 0 localhost:10004 2015-06-18_15:26:57.988-0600 1.005 0.000 0.000 405946368 970752
foo 23 27 Success 0 localhost:10004 2015-06-18_15:26:57.988-0600 1.005 0.000 0.000 405946368 630784
foo 24 28 Success 0 localhost:10004 2015-06-18_15:26:57.995-0600 1.005 0.000 0.000 405946368 630784
foo 25 29 Success 0 localhost:10004 2015-06-18_15:26:58.993-0600 1.005 0.000 0.000 405946368 974848
foo 26 30 Success 0 localhost:10004 2015-06-18_15:26:58.994-0600 1.004 0.000 0.000 405946368 626688
The task log contains the following fields.
- JobID – Job ID that was passed to Nitro using the "‑‑job‑id" command line option.
- TaskID – Task number within the Nitro job.
- Line – Line number in the task file of the task definition.
- Name – Task name supplied in the task definition by the "name=<name>" option.
- Status – One of "Success", "Failure", "InsufRes", "Timeout", or "Invalid".
- ExitCode – Numerical exit code returned by the task.
- Hostname – Name of the worker that executed the task.
- StartTime – Date and time the worker actually started the task.
- Duration – Number of seconds the task ran (millisecond resolution).
- UserCPU – Number of seconds the task ran in user mode (millisecond resolution).
- SystemCPU – Number of seconds the task run system calls (millisecond resolution).
- VirtualMem – Maximum virtual memory allocated to the task in bytes.
- PhysicalMem – Maximum physical memory allocated to the task in bytes.
- Labels – Optional task labels specified by the task definition.
- Output – stdout and/or stderr. If a task outputs to both stdout and stderr, both are displayed in the format <stdout>/<stderr>.
The operating system may allocate shared memory and may charge a proportion of this shared memory to random tasks.
Related Topics