Configuring jobs > Configuring TORQUE and Moab for streaming job output

4.6 Configuring TORQUE and Moab for streaming job output

The only resource manager for which streaming output is supported is TORQUE 2.4 and later.

To configure TORQUE and Moab for streaming job output

  1. Configure all pbs_mom daemons in TORQUE with the $spool_as_final_name parameter set to true.

    2. This causes the MOM daemons to not move the job output files to another location after the jobs finish running.

  2. Use submit filters to force all job output to go to the directory configured in Viewpoint.

    3. If both msub and qsub are used by users, a filter must be supplied for both. The following example demonstrates a submit filter for the msub command that forces all job output to go to /var/spool/torque/spool. The script relies on Ruby, RubyGems, the libxml-ruby gem, and the availability of native libxml2 libraries that libxml-ruby needs.

    #!/usr/bin/env ruby

    require 'rubygems'

    require 'libxml'

    include LibXML::XML

    SpoolDir = '/var/spool/torque/spool/'

    ScriptStartRe = %r{^\s*\START}

    ScriptDirectiveRe = %r{#{ScriptStartRe}\23!.+?\0a}

    job = Parser.io($stdin).parse

    e = job.find_first('/job/SubmitString')

    output_directives = "#PBS -o #{SpoolDir}

    #PBS -e #{SpoolDir}

    "

    { " " => "\\20", "

    " => "\\0a", "#" => "\\23" }.each do |a, b|

    output_directives.gsub!(a, b)

    end

    if e.content =~ ScriptDirectiveRe || e.content =~ ScriptStartRe

    e.content.insert($~.size, output_directives)

    else

    raise "Failed to recognize start of job submit string"

    end

    puts job

  3. Create job output files with file permissions such that the OS user Viewpoint runs under can read the files.

    4. Use the $job_output_file_unmask parameter to configure all MOM daemons to produce job output files that are world-readable. Specifically, the following line causes job output files to be created with read privileges for all users:

    $job_output_file_umask 022

  4. Verify the steps have been completed successfully. Once the following 4 criteria have been met, the job output Viewpoint feature and the "get-output" job control should function properly:
    1. TORQUE MOM daemons are configured to use the same location for job output while the job is running and after it is completed.
    2. Submit filters force all job output to a single directory.
    3. The job output umask is configured so that the Viewpoint user can read the job output files.
    4. Viewpoint is properly configured to look inside this directory for job output, the job output Viewpoint feature and the "get-output" job control should function properly.

Related topics 

 

© 2012 Adaptive Computing