2.8 Service Jobs
TORQUE service jobs are a special kind of job that is treated differently by TORQUE than normal batch jobs. TORQUE service jobs are not related to Moab's dynamic service jobs. A TORQUE service job cannot dynamically grow and shrink in size over time.
Jobs are marked as service jobs at the time they are submitted to Moab or TORQUE. Just like a normal job, a script file is specified with the job. In a batch job, the contents of the script file are taken by TORQUE and executed on the compute nodes. For a service job, however, the script file is assumed to respond to certain command-line arguments. Instead of just executing the script, TORQUE will use these command-line arguments to start, stop, and check on the status of the job. Listed below are the three command-line arguments that must be supported by any script submitted as part of a TORQUE service job:
- 'start' - The script should take this argument and launch its service/workload. The script should remain executing/running until the service stops.
- 'stop' - The script should take this argument and stop the service/workload that was earlier started.
- 'status' - The script should take this argument and return, via standard out, either "running" if the service/workload is running as expected or "stopped" if the service is not running.
This feature was created with long-running services in mind. The command-line arguments should be familiar to users who interact with Unix services, as each of the service scripts found in /etc/init.d/ also accept and respond to the arguments as explained above.
For example, if a user wants to start the Apache 2 server on a compute node, they can use a TORQUE service job and specify a script which will start, stop, and check on the status of the "httpd" daemon--possibly by using the already present /etc/init.d/httpd script.
Moab Version Required
If you wish to submit service jobs only through TORQUE, no special version of Moab is required. If you wish to submit service jobs using Moab's msub, then Moab 5.4 or later is required.
Submitting Service Jobs
There is a new option to qsub, "-s" which can take either a 'y' or 'n' (yes or no, respectively). When "-s y" is present, then the job is marked as a service job.
qsub -l walltime=100:00:00,nodes=1 -s y service_job.py
The example above submits a job to TORQUE with a walltime of 100 hours, one node, and it is marked as a service job. The script "service_job.py" will be used to start, stop, and check the status of the service/workload started on the compute nodes.
Moab, as of version 5.4, is able to accept the "-s y" option when msub is used for submission. Moab will then pass this information to TORQUE when the job is migrated.
Submitting Service Jobs in MCM
Submitting a service job in MCM requires the latest Adaptive Computing Suite snapshot of MCM. It also requires MCM to be started with the "--future=2" option.
Once MCM is started, open the Create Workload window and verify Show Advanced Options is checked. Notice that there is a Service checkbox that can be selected in the Flags/Options area. Use this to specify the job is a service job.
Managing Service Jobs
Managing a service job is done much like any other job; only a few differences exist.
Examining the job with qstat -f will reveal that the job has the service = True attribute. Non-service jobs will not make any mention of the "service" attribute.
Canceling a service job is done with qdel, mjobctl -c, or through any of the GUI's as with any other job. TORQUE, however, cancels the job by calling the service script with the "stop" argument instead of killing it directly. This behavior also occurs if the job runs over its wallclock and TORQUE/Moab is configured to cancel the job.
If a service job completes when the script exits after calling it with "start," or if TORQUE invokes the script with "status" and does not get back "running," it will not be terminated by using the "stop" argument.