4.4 Dynamic Workload
This topic identifies activities pertaining to dynamic workload.
Nitro jobs are flexible in the number of resources they can use to accomplish the tasks given them. Worker nodes can be added to a job or taken away without any adverse consequences (other than the job running more slowly). You can also add workload by appending the task file.
In this topic:
If nodes are needed for a more important task, the workers can be killed, and their assignments will be returned to the coordinator.
Killing the worker with a SIGTERM signal will allow the worker to send a partial assignment completion report to the coordinator. Be aware that if a worker is killed, the tasks that are running, may be run again when the assignment is given to a different worker to complete. Therefore, it is important to program your tasks to exit if the work has already been completed or overwrite the previous result.
4.4.2 Adding Worker Nodes to a Running Job
While the workload is being executed, workers can be added to the coordinator. The coordinator requires either a list of worker names or a session key that workers will use to attach to the coordinator to receive workload assignments. If you specify a list of worker names, only those workers will be authorized to connect to the coordinator. If you specify a session key, any worker with the session key will be able to connect to the coordinator.
Once the coordinator exits, the job is finished and workers won't be able to connect.
If you need to keep a coordinator up continually to respond to workload that could be added at any time, you can use the "‑‑linger" command line option on the workers and coordinator to allow Nitro to stay resident and not exit when the tasks are completed.
Nitro provides a message-based process to dynamically add workload to Nitro. Contact Adaptive Computing Professional Services for more information on dynamically adding workload.