3.36 Applying the msub submit filter

When you use msub to submit a job, msub processes the input, converts it to XML, and sends the job specification XML to the Moab scheduler. You can create a submission filter to modify the job XML based on the criteria you set before Moab receives and processes it.

Image 3-1: Job submission process

Click to enlarge

The filter gives you the ability to customize the submission process, which is helpful if jobs should have certain defaults assigned to them, if you want to keep detailed submission statistics, or if you want to change job requests based on custom needs.

The submit filter, is a simple executable or script that receives XML via its standard input and returns the modified XML in its standard output. It modifies the attributes of the job specification XML based on policies you specify. It can perform various other actions at your request, too; for instance, logging. Once the submit filter has modified the job XML based on your criteria, it writes the XML representing the actual job submission to stdout. The new XML could potentially match the original XML, depending on whether the job met the criteria for modification set in the job submit filter script. Job submissions you want to proceed will leave the filter with an exit code of 0 and continue to Moab for scheduling. If the job meets the filter's specified criteria for rejection, it exits with a non-zero value, aborting the job submission process. You can configure the filter script to write a descriptive rejection message to stderr.

Job submit filters follow these rejection rules: 1) msub will reject job XML with an exit code of anything other than zero, 2) the msub command displays filter's error output on the command line, 3) msub will reject the job if the filter outputs invalid job XML, and 4) msubwill reject the job if it violates any policies in your general Moab configuration; you cannot use a submit filter to bypass other policies.

To see the schema for job submission XML, please refer to Submitting Jobs via msub in XML.

3.36.1 Submit filter types

You can implement submit filters on either the client or server side of a job submission. The primary differences between the two submit filter types are the location from which the filter runs, the powers and privileges of the user running the filter, and whether a user can bypass the filter. Client-based submit filters run from the msub client as the user who submits the job and can be bypassed, and server-based submit filters run from the Moab server as the user as which the server is running and cannot be bypassed.

3.36.1.A Client-based submit filter

Client-based filters run from the msub client as the user who is submitting the job. Because they do not have elevated privileges, the risk of client-based submit filters' being abused is low; however, it is possible for the client to specify its own configuration file and bypass the filter or substitute its own filter. Job submissions do not even reach the server if a client-based submit filter rejects it.

To configure msub to use the submit filter, give each submission host access to the submit filter script and add a SUBMITFILTER parameter to the Moab configuration file (moab.cfg) on each submission host. The following example demonstrates how you might modify the moab.cfg file:

SUBMITFILTER /home/submitfilter/filter.pl

If you experience problems with your submit filter and want to debug its interaction with msub, enter msub --loglevel=9. This will cause msub to print verbose log messages to the terminal.

3.36.1.B Server-based submit filter

Server-based submit filters run from the Moab server as the user as which the server is running. Because it runs as a privileged user, you must evaluate the script closely for security implications. A client configuration cannot bypass the filter.

To configure Moab to automatically apply a filter to all job submissions, use the SERVERSUBMITFILTER parameter. SERVERSUBMITFILTER specifies the path to a global job submit filter script, which Moab will run on the head node and apply to every job submitted.

SERVERSUBMITFILTER /opt/moab/scripts/jobFilter.pl

3.36.1.C OutputFormat XML Tag

The "OutputFormat" element is used by a job submit filter to alter the output of the msub command when it reports the submitted job's job id. For example, if a job submit filter performs a complex procedure on behalf of the user, such as submitting system jobs for a pre-defined workflow to accomplish some function, the filter can set this element to a value that permits it to return the job ids of the system jobs it submitted in addition to the user's job id the msub command returns (The Moab integration with Cray's SSD-based DataWarp service does precisely this using a job submit filter).

To illustrate this element's functionality using the Moab/DataWarp integration example, a DataWarp job submit filter submits a "DataWarp instance creation/input data staging" script as a system job and a corresponding "output data staging/DataWarp instance destruction" script as another system job, and then ties them together with job dependencies in a "DataWarp job workflow" that causes the user job's execution to depend on the successful completion of the DataWarp creation/input staging job and the DataWarp output staging/DataWarp Destruction system job to depend on the user job, regardless whether it completes successfully or not, or is cancelled. This DataWarp 3-job workflow guarantees the proper creation and destruction of job-based DataWarp storage; all set up and accomplished by a job submit filter.

However, users often create job workflows that have dependencies between their own jobs and may require the job ids of all jobs to be made available in order to build a desired job workflow; i.e "jobB" may require "jobA" to complete before "jobB" is able to run. For example, if jobA were a DataWarp job and jobB should not run unless JobA successfully completes, but not until JobA's output data files are successfully staged, jobB must depend on jobA's job id as well as jobA's "output data staging/DataWarp instance destruction" system job's job id. The user can indicate jobB's job dependencies when jobA is a DataWarp job using the job submission option:

-l depend=afterok:<jobAid>:<jobAoutputSystemJobId>.

The OutputFormat XML tag provides a way for a job submit filter to pass the job ids of additional jobs it submitted to perform a service on behalf of the user's job.

For example, you might submit a job and a job submit filter submits two additional jobs to assist it; the first additional job, "job11", will run before your job, and the second additional job, "job12", needs to run after your job finishes. If the job submit filter requires them to output in the order of "pre", "user", and "post" job ids (which is the same order Moab outputs job ids for user jobs with input and output data-staging options), it would return the following OutputFormat element as the user's job id string.

<OutputFormat>moab.11 %s moab.12</OutputFormat>

This means that you can have all three job ids delivered to the end user, or a job workflow generation script in an easy to read format.

3.36.2 Sample submit filter script

The following example is a trivial implementation that will not affect whether a job is submitted. Use it as reference to verify that you are writing your filter properly.

#!/usr/bin/perl
use strict;

## Simple filter example that re-directs the output to a file.

my $file = "xmllog.out";

open FILE,">>$file" or die "Couldn't open $file: $!";
while (<>)
{
print FILE;
print;
}
close FILE;

© 2016 Adaptive Computing