17.0 Object triggers > References > Trigger components

Conventions

17.16 Trigger components

Required trigger components

AType

Action type Description
cancel Cancels the object
changeparam Causes Moab to give a parameter to a new value
exec Launches an external program or script on the command line when the dependencies are fulfilled. See Using a trigger to execute a script for more information.
internal Modifies Moab without using the command line. See Using a trigger to perform internal Moab actions for more information.
jobpreempt Indicates the preempt policy to apply to all jobs currently allocated resources assigned to the trigger's parent reservation
mail Causes Moab to send mail. See Using a trigger to send email for more information.

Action

Cancel Action
Format NONE
Description Indicates that Moab should cancel the reservation when the event occurs. No action should be specified.
Example
Etype=threshold,Threshold=usage<10,FailOffset=1:00,AType=cancel

When less than 10% of the reservation has been in use for a minute, Moab cancels it.

Changeparam Action
Format Action="<STRING>"
Description Specifies the parameter to change and its new value (using the same syntax and behavior as the changeparam command)
Example
Atype=changeparam,Action="JOBCPURGETIME 02:00:00"

Moab maintains detailed job information for two hours after a job has completed.

Jobpreempt Action
Format Action="cancel|checkpoint|requeue|suspend"
Description Signifies PREEMPTPOLICY to apply to jobs that are running on allocated resources
Example
RSVPROFILE[adm1] TRIGGER=EType=start,Offset=-240,AType=jobpreempt,Action="cancel"

40 minutes after the reservation adm1 starts, all jobs using the reservation's resources adopt a PREEMPTPOLICY of cancel.

Mail Action
Format Action="<MESSAGE>"
Description When AType=mail, the Action parameter contains the message body of the email. This can be configured to include certain variables. See Using a trigger to send email for details.

Mail triggers can be configured to launch for node failures, reservation creation or release, scheduler failures, and even job events. In this way, site administrators can keep track of scheduler events through email.

The email comes from moabadmin, has a subject of moab update, and has a body of whatever you specified in the Action attribute. The recipient list depends on the type of object the trigger is attached to.

  • Node - The primary user (first listed in ADMINCFG[1]), typically root
  • Scheduler - The primary user
  • Job - The user who owns the job
  • Reservation - The primary user
Example
NODECFG[DEFAULT] TRIGGER=EType=fail,AType=mail,Action="node $OID will failed.",Offset=05:00:00

This example sends an email to the primary administrator informing him/her that the node (including the node ID) has failed.

Exec Action
Format Action="<script>"
Description Exec triggers will launch an external program or script when their dependencies are fulfilled. The following example will submit job.cmd and then execute monitor.pl three minutes after the job is started. See Using a trigger to execute a script for more information.
Example

> msub -l trig=EType=start\&AType=exec\&Action="/tmp/monitor.pl" job.cmd\&Offset=03:00

Internal Action
Format Action="<objectType>:-:<cancel|reserve>"
Description A couple different actions are valid depending on what type of object the internal trigger is acting upon. The following list shows the available actions:
  • Reserve a node
  • Cancel a job
  • Cancel a reservation

See Using a trigger to perform internal Moab actions for more information.

Example
NODECFG[node01] TRIGGER=EType=start,AType=internal,Action="node:-:reserve"

When node01 starts, it becomes a reservation.

> msub moab.3 -l 'trig=EType=fail\&AType=internal\&Action="job:-:cancel"

If moab.3 fails, Moab cancels it.

> mrsvctl -c -a user==joe -h node50 -T EType=start,AType=internal,Action="reservation:-:cancel",Offset=10:00

User joe's jobs are given a ten-minute window to start, then the reservation cancels.

EType

Event type Description
cancel The event is triggered when the parent object is either canceled or deleted.
checkpoint Triggers fire when the job is checkpointed. checkpoint triggers can only be attached to jobs.
create Triggers fire when the parent object is created. create triggers can be attached to nodes, jobs, reservations, and the scheduler (when attached to the scheduler, triggers fire when Moab starts).
discover Triggers fire when the node is loaded from a resource manager and Moab cannot recorgnize it nor find it in the checkpoint file.
end Triggers fire when the parent object ends. end triggers can be attached to nodes, jobs, reservations, and the scheduler (when attached to the scheduler, triggers fire when Moab shuts down).
fail Triggers fire when the resource manager is in a corrupt or down state for longer than the configured fail time, or when Moab detects a corruption in a node's reservation table. fail triggers can be attached to jobs, nodes, resource managers, and the scheduler.
hold Triggers fire when the job is put on hold. hold triggers can only be attached to jobs.
modify Triggers fire when the parent object is modified. modify triggers can be attached to jobs and reservations
preempt Triggers fire when the job is preempted. preempt triggers can only be attached to jobs.
standing Triggers fire multiple times based on a certain period. They can be used with Period and Offset attributes. standing triggers can be attached to nodes and the scheduler.
start Triggers fire when the parent object or Moab starts. start triggers can be attached to jobs, reservations, resource managers, and the scheduler (when Moab starts and at the beginning of Moab's first iteration).
threshold

Triggers fire when a threshold, such as usage or a gmetric comparison, is true. threshold triggers can be attached to nodes and reservations.

Triggers with ETypes set to threshold must include the Threshold attribute.

Event-modifying trigger components

The following trigger attributes modify the event that causes the trigger to fire.

RearmTime
Possible Values [[HH:]MM:]SS
Description The amount of time that must pass before a trigger can fire again. RearmTime is enforced from the trigger event time.
Usage Notes ---
Offset
Possible Values [-][[HH:]MM:]SS
Description The relative time offset from event when trigger can fire
Usage Notes
  • Only end triggers can have a negative value for Offset.
  • Offset cannot be used with cancel.
Period
Possible Values Minute, Hour, Day, Week, Month, Infinity
Description The period at which the trigger will regularly fire
Usage Notes ---
Threshold
Possible Values Threshold={<metric>[<metricName>]}{> >= < <= ==}<FLOAT>
Where <metric> is one of:
  • gmetric
  • usage
Description

When the object meets, drops below, or increases past the configured Threshold, the trigger will fire.

Usage Notes

Threshold triggers allow sites to configure triggers to launch based on internal scheduler statistics, such as the usage of a reservation.

FailOffset
Possible Values [[HH:]MM:]SS
Description The time that the threshold condition must exist before the trigger fires
Usage Notes Use with fail triggers to avoid transient triggers.

Action-modifying trigger components

Flags
Possible Values

Flags=<flag>[:<flag>] or Flags=[<flag>][[<flag>]]

attacherror - If the trigger outputs anything to stderr, Moab attaches it as a message to the trigger object.

cleanup - If the trigger is still running when the parent object completes or is canceled, Moab kills the trigger.

checkpoint - Moab always checkpoints this trigger. For more information, see Checkpointing a trigger.

objectxmlstdin - Trigger passes its parent's object XML information into the trigger's stdin. This only works for exec triggers with reservation type parents.

resetonmodify - The trigger resets if its object is modified, even if RearmTime is not set.

user - The trigger executes under the user ID of the object's owner. If the parent object is the scheduler, you may explicitly specify the user using the format user+<username>. For example: Flags=user+john.

Description Specifies various trigger behaviors and actions
Usage Notes When specifying multiple flags, each flag can be delimited by colons (:) or with square brackets; for example:

Flags=[user][cleanup] or Flags=user:cleanup

BlockTime
Possible Values [[HH:]MM:]SS
Description The amount of time Moab will suspend normal operation to wait for trigger execution to finish
Usage Notes Use caution; Moab will completely stop normal operation until BlockTime expires.
ExpireTime
Possible Values <INTEGER>
Description The time at which trigger should be terminated if it has not already been activated
Usage Notes ---
Timeout
Possible Values [+|-][[HH:]MM:]SS
Description The time allotted to this trigger before it is marked as unsuccessful and its process (if any) killed
Usage Notes ---
MaxRetry
Possible Values MaxRetry=<INTEGER>
Description The number of times Action will be attempted before the trigger is designated a failure
Usage Notes If Action fails, the trigger will restart immediately (up to MaxRetry times). If it fails more than MaxRetry times, the trigger has failed. This restart ignores FailOffset and RearmTime.

Organizational trigger components

Name
Possible Values Name=<STRING>
Description Name of the trigger
Usage Notes

Because Moab uses its own internal ID to distinguish triggers, the Name need not be unique. Only the first 16 characters of Name are stored by Moab.

Description
Possible Values Description=<STRING>
Description Description of the trigger
Usage Notes ---