Moab Web Services > Resources > Jobs

Jobs

This section describes behavior of the Job object in Moab Web Services. It contains the URLs, request bodies, and responses delivered to and from MWS.

The Job API is new with API version 2. The supported methods table below requires each resource to be accessed with a URL parameter of api-version=3 in order to behave as documented.

For more information, see Requesting Specific API Versions.

The Fields: Jobs reference contains the type and description of all fields in the Job object. It also contains details regarding which fields are valid during PUT and POST actions.

Supported methods

Resource	GET	PUT	POST	DELETE
/rest/jobs	Get All Jobs	--	Submit Job	--
/rest/jobs/<name>	Get Single Job	Modify Job Attributes	--	Cancel Job
/rest/jobs/<name>/<modifyAction>	--	Perform Actions on Job	--	--

This topic contains these sections:

Getting Job Information
- Get All Jobs
- Get Single Job
Submitting Jobs
- Submit Job
Modifying Jobs
Deleting (Canceling) Jobs
- Cancel Job

Getting Job Information

The HTTP GET method is used to retrieve Job information.

Quick reference

GET http://localhost:8080/mws/rest/jobs/<name>?api-version=3

Get All Jobs

URLs and parameters

GET http://localhost:8080/mws/rest/jobs?api-version=3

Parameter	Required	Type	Description	Example
query	No	JSON	Queries for specific results. It is possible to query by one or more fields based on MongoDB query syntax.	query={"isActive":true}
sort	No	JSON	Sort the results. Use 1 for ascending and -1 for descending.	sort={"name":-1}

Parameter

Required

Type

Description

Example

query

JSON

Queries for specific results.

It is possible to query by one or more fields based on MongoDB query syntax.

query={"isActive":true}

sort

JSON

Sort the results. Use 1 for ascending and -1 for descending.

sort={"name":-1}

See Global URL Parameters for available URL parameters.

How to get all jobs

GET http://localhost:8080/mws/rest/jobs?api-version=3&fields=name,flags&max=3
------------------------------------

{
  "totalCount": 8,
  "resultCount": 3,
  "results":   [
        {
      "flags": ["GLOBALQUEUE"],
      "name": "Moab.1"
    },
        {
      "flags": ["GLOBALQUEUE"],
      "name": "Moab.2"
    },
        {
      "flags": ["GLOBALQUEUE"],
      "name": "Moab.4"
    }
  ]
}

How to get a subset of jobs

Get active jobs
------------------------------------

http://localhost:8080/mws/rest/jobs?api-version=3&query={"isActive":true}

Get completed jobs
------------------------------------

http://localhost:8080/mws/rest/jobs?api-version=3&query={"isActive":false}

Get jobs owned by a particular user
------------------------------------

http://localhost:8080/mws/rest/jobs?api-version=3&query={"credentials.user":"fred"}

Known issues

Some jobs are not returned if DisplayFlags UseBlocking is set in the moab.cfg file.

Get Single Job

URLs and parameters

GET http://localhost:8080/mws/rest/jobs/<name>?api-version=3

Parameter	Required	Type	Valid values	Description
name	Yes	String	--	The name of the object.

See Global URL Parameters for available URL parameters.

The attributes field is only applicable in API version 2 and later, and the MOAB_TENANT field only applies if the job is attached to a tenant.

Sample response

JSON response
------------------------------------

{
  "arrayIndex": null,
  "arrayMasterName": null,
  "attributes": [],
  "blocks": [  {
    "category": "jobBlock",
    "message": null,
    "type": null
  }],
  "bypassCount": 0,
  "cancelCount": 0,
  "commandFile": "/tmp/test.sh",
  "commandLineArgs": null,
  "completionCode": null,
  "cpuTime": 0,
  "credentials":   {
    "account": null,
    "group": "adaptive",
    "jobClass": null,
    "qos": "NONE",
    "qosRequested": null,
    "user": "adaptive"
  },
  "customName": null,
  "dates":   {
    "completedDate": null,
    "createdDate": "2012-10-11 17:58:16 UTC",
    "deadlineDate": "2037-10-24 12:26:40 UTC",
    "dispatchedDate": null,
    "earliestRequestedStartDate": null,
    "earliestStartDate": "2012-10-11 17:58:18 UTC",
    "eligibleDate": "2012-10-11 17:59:19 UTC",
    "lastCanceledDate": null,
    "lastChargedDate": null,
    "lastPreemptedDate": null,
    "lastUpdatedDate": "2012-10-11 17:59:19 UTC",
    "startDate": null,
    "submitDate": "2012-10-11 17:58:16 UTC",
    "terminationDate": "2037-10-24 12:26:40 UTC"
  },
  "deferCount": 0,
  "dependencies": [],
  "description": null,
  "duration": 8639999,
  "durationActive": 0,
  "durationQueued": 31,
  "durationRemaining": 0,
  "durationSuspended": 0,
  "emailNotifyAddresses": [],
  "emailNotifyTypes": [],
  "environmentRequested": false,
  "environmentVariables": {},
  "epilogScript": null,
  "flags": ["GLOBALQUEUE"],
  "holdDate": null,
  "holdReason": null,
  "holds": [],
  "initialWorkingDirectory": "/tmp",
  "isActive": true,
  "jobGroup": null,
  "masterNode": null,
  "memorySecondsDedicated": 0,
  "memorySecondsUtilized": 0,
  "messages": [],
  "migrateCount": 0,
  "minimumPreemptTime": 0,
  "mwmName": "Moab",
  "name": "Moab.15",
  "nodesExcluded": [],
  "nodesRequested": [],
  "nodesRequestedPolicy": null,
  "partitionAccessList":   [
    "msm",
    "SHARED"
  ],
  "partitionAccessListRequested":   [
    "msm",
    "SHARED"
  ],
  "preemptCount": 0,
  "priorities":   {
    "run": 0,
    "start": 1,
    "system": 0,
    "user": 0
  },
  "processorSecondsDedicated": 0,
  "processorSecondsLimit": 0,
  "processorSecondsUtilized": 0,
  "prologScript": null,
  "queueStatus": "blocked",
  "rejectPolicies": [],
  "requirements": [  {
    "architecture": null,
    "attributes":{
      "matlab":   [
             {
            "restriction":"must",
            "comparator": "<=",
            "value": "7.1",
            "displayValue": null
         }
      ],
      "MOAB_TENANT":   [   {
            "value": "1234567890aabbccddeeff00",
            "displayValue": "ResearchGroup"
      }],
      "soffice":   [
             {
            "restriction":"must",
            "comparator": "%=",
            "value": "3.1",
            "displayValue": null
         }
      ]
    },
    "features": [],
    "index": 0,
    "featuresRequested": [],
    "featuresRequestedMode": "AND",
    "featuresExcluded": [],
    "featuresExcludedMode": "AND",
    "metrics": {},
    "nodeAccessPolicy": null,
    "nodeAllocationPolicy": null,
    "nodeCount": 0,
    "nodes": [],
    "nodeSet": null,
    "image": null,
    "reservation": null,
    "resourcesPerTask":     {
      "processors":       {
        "dedicated": 1,
        "utilized": 0
      },
      "memory":       {
        "dedicated": 0,
        "utilized": 0
      },
      "disk":       {
        "dedicated": 0,
        "utilized": null
      },
      "swap":       {
        "dedicated": 0,
        "utilized": null
      }
    },
    "taskCount": 4,
    "tasksPerNode": 0
  }],
  "reservationRequested": null,
  "resourceFailPolicy": null,
  "resourceManagerExtension": null,
  "resourceManagers": [  {
    "isDestination": false,
    "isSource": true,
    "jobName": "Moab.15",
    "name": "internal"
  }],
  "rmStandardErrorFilePath": null,
  "rmStandardOutputFilePath": null,
  "standardErrorFilePath": null,
  "standardOutputFilePath": null,
  "startCount": 0,
  "states":   {
    "state": "Idle",
    "stateExpected": "Idle",
    "stateLastUpdatedDate": null,
    "subState": null
  },
  "submitHost": "0:0:0:0:0:0:0:1",
  "systemJobAction": null,
  "systemJobType": null,
  "targetedJobAction": null,
  "targetedJobName": null,
  "templates": [{"name": "DEFAULT"}],
  "triggers": [],
  "variables": {},
  "virtualContainers": [],
  "virtualMachines": [],
  "vmUsagePolicy": null
}

Job arrays

If a job is the master of a job array, the response will have some additional fields set as shown in the following example. The name field is chosen by the Moab, and the customName field comes from the Fields: Job Arrays name field.

Job array master
------------------------------------

{
  "name": "Moab.5",
  "customName": "myarray",
  "flags":   [
    "ARRAYMASTER",
    "GLOBALQUEUE",
    "CANCELONFIRSTFAILURE",
    "CANCELONANYSUCCESS"
  ]
}

If a job is a sub-job of an array, the response will have other fields set as shown in the following example.

Array sub-job
------------------------------------

{
  "name": "Moab.5[21]",
  "customName": "myarray",
  "arrayIndex": 21,
  "arrayMasterName": "Moab.5",
  "flags":   [
    "ARRAYJOB",
    "GLOBALQUEUE",
    "CANCELONFIRSTFAILURE",
    "CANCELONANYSUCCESS"
  ]
}

Submitting Jobs

The HTTP POST method is used to submit Jobs.

Quick reference

POST http://localhost:8080/mws/rest/jobs?api-version=3[&proxy-user=<username>]

Restrictions

No more than one virtual container can be specified in the request. The virtual container must already exist.
The credentials.user and credentials.group properties are used to submit a job as the specified user belonging to the specified group.
Job variables have the following restrictions:
- Variable names cannot contain equals (=), semicolon (;), colon (:), plus (+), question mark (?), caret (^), backslash (\), or white space.
- Variable values cannot contain semicolon (;), colon (:), plus (+), or caret (^).
When submitting jobs, the only supported hold type is User.
The proxy-user parameter is ignored unless you set ENABLEPROXY=TRUE in the moab.cfg file. For example:

ADMINCFG[1]           USERS=root,ted ENABLEPROXY=TRUE

When the JOBNODEMATCHPOLICY is set to EXACTNODE in Moab, avoid using requirements.tasksPerNode. Use requirements.resourcesPerTask.processors.dedicated instead.

Submit Job

URLs and parameters

POST http://localhost:8080/mws/rest/jobs?api-version=3[&proxy-user=<username>]

Parameter	Required	Type	Valid values	Description
proxy-user	No	String	--	Perform the action as this user.

See Global URL Parameters for available URL parameters.

Request body

JSON request body (specified host list)
------------------------------------

{
  "attributes":   [
    "attr1",
    "attr2"
  ],
  "commandFile": "/tmp/test.sh",
  "commandScript": "c2xlZXAgNjAK",
  "commandLineArguments": "-x -v",
  "credentials":   {
    "account": "account",
    "group": "group",
    "jobClass": "BATCH",
    "qosRequested": "QOS1",
    "user": "saadmin"
  },
  "customName": "custom name for job",
  "dates":   {
    "earliestRequestedStartDate": "2012-11-08 13:18:47 UTC",
    "deadlineDate": "2014-02-17 14:00:00 UTC"
  },
  "dependencies":   [
        {
      "type": "set",
      "name": "vc1.varA"
    },
        {
      "type": "set",
      "name": "vc2.varB"
    },
        {
      "type": "set",
      "name": "vc3.varC"
    }
  ],
  "duration": 600,
  "emailNotifyAddresses":   [
    "[email protected]",
    "[email protected]"
  ],
  "emailNotifyTypes":   [
    "JobStart",
    "JobEnd"
  ],
  "environmentRequested": true,
  "environmentVariables":   {
    "var1": "val1",
    "var2": "val2"
  },
  "epilogScript": "/tmp/epilog.sh",
  "flags":   [
    "RESTARTABLE",
    "SUSPENDABLE"
  ],
  "holds": ["User"],
  "initialWorkingDirectory": "/tmp",
  "jobGroup": "job_group",
  "nodesExcluded":   [
    {"name": "node07"},
    {"name": "node08"}
  ],
  "nodesRequested":   [
    {"name": "node01"},
    {"name": "node02"}
  ],
  "nodesRequestedPolicy": "SUBSET",
  "partitionAccessListRequested":   [
    "p1",
    "p2"
  ],
  "priorities": {"user": 5},
  "prologScript": "/tmp/prolog.sh",
  "requirements": [  {
    "architecture": "x86_64",
    "attributes":{
      "matlab":   [
             {
            "restriction":"must",
            "comparator": "<=",
            "value": "7.1"
         }
      ],
      "soffice":   [
             {
            "restriction":"must",
            "comparator": "%=",
            "value": "3.1"
         }
      ]
    },
    "featuresRequested":     [
      "a",
      "b",
      "c"
    ],
    "featuresRequestedMode": "OR",
    "featuresExcluded":     [
      "d",
      "e",
      "f"
    ],
    "featuresExcludedMode": "AND",
    "nodeAccessPolicy": "SINGLEJOB",
    "nodeAllocationPolicy": "PRIORITY",
    "nodeCount": 6,
    "nodeSet":"FIRSTOF:FEATURE:vlan2",
    "image": "linux",
    "resourcesPerTask":     {
      "disk": {"dedicated": 1024},
      "memory": {"dedicated": 512},
      "processors": {"dedicated": 2},
      "swap": {"dedicated": 4096},
      "matlab": {"dedicated": 6},
      "intellij": {"dedicated": 2}
    },
    "taskCount": 4,
    "tasksPerNode": 14
  }],
  "reservationRequested": {"name": "rsv.1"},
  "resourceFailPolicy": "RETRY",
  "resourceManagerExtension": "x=PROC=4",
  "standardErrorFilePath": "/tmp/error",
  "standardOutputFilePath": "/tmp/out",
  "submitHost": "admin-node",
  "templates":   [
    {"name": "template1"},
    {"name": "template2"}
  ],
  "variables":   {
    "var1": "val1",
    "var2": "val2"
  },
  "virtualContainers": [{"name": "vc1"}],
  "vmUsagePolicy": "CREATEVM"
}

Sample response

The response of this task is one of three possibilities:

An object with a single messages property containing a list of error messages on failure.

{"messages":["Could not create job - invalid requirements"]}

An object with a name property containing the name of the newly created job.

{"name":"Moab.1"}

An object with a name property and a virtualContainers list containing the name of the newly created virtual container.

{ "name": "Moab.1", "virtualContainers": [{"name": "vc1"}] }

The virtual container will only be reported when a new virtual container has been created by Moab for the job.

Examples of job submission

This section includes some sample job submission requests.

Example 4-5: Submit job to run on node2 and node3

POST http://localhost:8080/mws/rest/jobs?api-version=3
------------------------------------

{
  "commandFile": "/tmp/test.sh",
  "credentials":   {
    "group": "adaptive",
    "user": "adaptive"
  },
  "initialWorkingDirectory": "/tmp",
  "nodesRequested":   [
    {"name": "node2"},
    {"name": "node3"}
  ]
}

Example 4-6: Submit job that requires 20 processors

POST http://localhost:8080/mws/rest/jobs?api-version=3
------------------------------------

{
  "commandFile": "/tmp/test.sh",
  "credentials":   {
    "group": "adaptive",
    "user": "adaptive"
  },
  "initialWorkingDirectory": "/tmp",
  "requirements": [{"taskCount": 20}]
}

Example 4-7: Submit job to run after a certain time

POST http://localhost:8080/mws/rest/jobs?api-version=3
------------------------------------

{
  "commandFile": "/tmp/test.sh",
  "credentials":   {
    "group": "adaptive",
    "user": "adaptive"
  },
  "dates": {"earliestRequestedStartDate": "2012-10-11 18:36:35 UTC"},
  "initialWorkingDirectory": "/tmp",
  "requirements": [{"taskCount": 20}]
}

Example 4-8: Submit job based on msub example

Given this msub command:

msub -l nodes=3:ppn=2,walltime=1:00:00,pmem=100 script2.pbs.cmd

Here is an equivalent MWS request:

POST http://localhost:8080/mws/rest/jobs?api-version=3
------------------------------------

{
  "duration": 3600,
  "commandFile": "/home/adaptive/script2.pbs.cmd",
  "credentials":   {
    "group": "adaptive",
    "user": "adaptive"
  },
  "initialWorkingDirectory": "/home/adaptive",
  "requirements": [  {
    "resourcesPerTask": {"memory": {"dedicated": 100}},
    "taskCount": 6,
    "tasksPerNode": 2
  }]
}

To emulate what msub does, make commandFile an absolute path, and add credentials.user, credentials.group, and initialWorkingDirectory.

As shown above, nodes=3:ppn=2 is equivalent to setting taskCount to 6 and tasksPerNode to 2.

Example 4-9: Submit a job array

For information on how to submit a job array, see Submitting Job Arrays.

Modifying Jobs

The HTTP PUT method is used to modify Jobs.

Quick reference

PUT http://localhost:8080/mws/rest/jobs/<name>[/<modifyAction>]?api-version=3[&proxy-user=<username>]

Restrictions

The proxy-user parameter is ignored unless you set ENABLEPROXY=TRUE in the moab.cfg file. For example:

ADMINCFG[1]           USERS=root,ted ENABLEPROXY=TRUE

Modify Job Attributes

URLs and parameters

PUT http://localhost:8080/mws/rest/jobs/<name>?api-version=3[&proxy-user=<username>][&change-mode=set]

Parameter	Required	Type	Valid values	Description
name	Yes	String	--	The name of the object.
proxy-user	No	String	--	Perform the action as this user.

See Global URL Parameters for available URL parameters.

Additional URL parameters

Parameter	Required	Valid values	Description
change-mode	No	set (default) add remove	If set, replace all fields with the fields specified. If add, add the specified fields to existing fields. If remove, remove the specified fields from existing fields.

Parameter

Required

Valid values

Description

change-mode

set (default)

add

remove

If set, replace all fields with the fields specified.

If add, add the specified fields to existing fields.

If remove, remove the specified fields from existing fields.

Request body

The request body below shows all the fields that are available when modifying a job, along with some sample values.

JSON request body
------------------------------------

{
  "credentials":   {
    "account": "account",
    "jobClass": "BATCH",
    "qosRequested": "QOS1"
  },
  "customName": "custom name for job",
  "dates": {"earliestRequestedStartDate": "2012-11-08 13:18:47 UTC"},
  "duration": 600,
  "flags":   [
    "RESTARTABLE",
    "SUSPENDABLE"
  ],
  "holds": ["User"],
  "messages":   [
    {"message": "Message one"},
    {"message": "Message two"}
  ],
  "nodesRequested":   [
    {"name": "n015"},
    {"name": "n016"},
    {"name": "n017"},
    {"name": "n018"}
  ],
  "partitionAccessListRequested":   [
    "p1",
    "p2"
  ],
  "priorities":   {
    "system": 3,
    "user": 5
  },
  "requirements": [  {
    "features":     [
      "vlan1",
      "vlan2"
    ],
    "resourcesPerTask":     {
      "matlab": {"dedicated": 1},
      "tape": {"dedicated": 2}
    }
  }],
  "reservationRequested": {"name": "rsv.1"},
  "variables":   {
    "var1": "val1",
    "var2": "val2"
  }
}

Sample response

These messages may not match the messages returned from Moab exactly, but are given as an example of the structure of the response.

Not all messages are shown for the above request body.

JSON response
------------------------------------

{"messages": [
  "Account modified successfully",
  "Messages modified successfully",
  "Variables modified successfully"
]}

Restrictions

Old messages are not removed from jobs; only new messages are added.
Job variables have the restrictions documented in Submitting Jobs.
Although the client can modify features and resourcesPerTask, Moab only considers these elements when they appear in the first element of the requirements array. If the requirements array contains two or more elements, all elements but the first are silently ignored.

Generic Resources

Jobs can require configurable, site-specific consumable resources called generic resources. For example, some jobs may require a matlab license. Only one job at a time may legally consume this license. Matlab is not a standard resource and may only be available on some sites. Nevertheless Moab allows this to be configured and tracked as is explained in Managing Consumable Generic Resources.

You must specify generic resources in the requirements.resourcesPerTask portion of the JSON document. Any resource in requirement.resourcesPerTask that is not a standard resource is considered a generic resource. Standard resources include disk, memory, processors, and swap. Assume a job has the following in requirement.resourcesPerTask:

  "resourcesPerTask":{

    "processors":{

      "dedicated":4,

      "utilized":0

},

    "memory":{

      "dedicated":2048,

      "utilized":0

},

    "disk":{

      "dedicated":4096,

      "utilized":0

},

    "swap":{

      "dedicated":1024,

      "utilized":0

},

    "tape":{

      "dedicated":1,

      "utilized":0

},

    "matlab":{

      "dedicated":2,

      "utilized":0

The standard resources the job requires are:

4 processors
2048 MB of memory
4096 MB of disk
1024 MB of swap

The generic resources the job requires are

1 tape
2 matlab

To modify a job so that it requires 1 matlab license, run the following:

PUT http://localhost:8080/mws/rest/jobs/Moab.2?api-version=3

  "requirements":[

      "resourcesPerTask":{

        "matlab":{

          "dedicated":1

Perform Actions on Job

URLs and parameters

PUT http://localhost:8080/mws/rest/jobs/<name>/<modifyAction>?api-version=3[&proxy-user=<username>]

Parameter

Required

Type

Valid values

Description

name

Yes

String

The name of the object.

modifyAction

Yes

String

cancel

checkpoint

execute

hold

requeue

rerun

resume

suspend

unhold

If cancel, attempts to cancel the job (equivalent to deleting a job).

If checkpoint, attempts to checkpoint the job. Note that the OS must support checkpointing for this to work.

If execute, executes the job (if possible).

If hold, attempts to hold the job using the holds set in the request body.

If requeue, attempts to requeue the job.

If rerun, attempts to rerun the job.

If resume, attempts to resume the job.

If suspend, attempts to suspend the job.

If unhold, attempts to release the holds set in the request body.

proxy-user

String

Perform the action as this user.

See Global URL Parameters for available URL parameters.

Request body

Request bodies are only required for holding or unholding jobs. All other actions do not require request bodies of any kind.

JSON request body to add holds to a job
------------------------------------

{"holds": ["User"]}

JSON request body to remove holds from a job
------------------------------------

{"holds": ["User"]}

If no holds are specified when unholding a job, all holds will be removed. This is equivalent to specifying holds as a list with a single element of All.

Sample response

This message may not match the message returned from Moab exactly, but is given as an example of the structure of the response.

JSON response
------------------------------------

{"messages": ["Job modified successfully"]}

Deleting (Canceling) Jobs

The HTTP DELETE method is used to cancel Jobs.

Quick reference

DELETE http://localhost:8080/mws/rest/jobs/<name>?api-version=3[&proxy-user=<username>]

Restrictions

The proxy-user parameter is ignored unless you set ENABLEPROXY=TRUE in the moab.cfg file. For example:

ADMINCFG[1]           USERS=root,ted ENABLEPROXY=TRUE

Cancel Job

URLs and parameters

DELETE http://localhost:8080/mws/rest/jobs/<name>?api-version=3[&proxy-user=<username>]

Parameter	Required	Type	Valid values	Description
name	Yes	String	--	The name of the object.
proxy-user	No	String	--	Perform the action as this user.

See Global URL Parameters for available URL parameters.

Sample response

JSON response for successful DELETE
------------------------------------

{}

Additional information about the DELETE can be found in the HTTP response header X-MWS-Message.