(Click to open topic with navigation)

22.0 Workload-Driven Cloud Services > How-to's > Creating a cloud workflow

Conventions

22.3 Creating a cloud workflow

ClosedContext

The following procedure details the steps required to create a workflow that builds a VM. This workflow contains four job templates:

  1. createVM - The system job that creates the VM. It starts the workflow but may not run immediately. The trigger sets the IP address of the VM as a variable when it successfully completes.
  2. installSoftware - The system job that installs software on the new VM. It depends on the successful completion of createVM and uses the IP address variable to locate the correct VM.
  3. createVMWithSoftware - The VM tracking job the user submits to request the workflow. It is the final job in the workflow and depends on the successful completion of installSoftware.
  4. destroy - The job that, if createVMWithSoftware is canceled, runs a script to delete the VM.

Image 22-2: Sample cloud workflow in submit order

Click to enlarge

Image 22-3: Sample cloud workflow in run order

Click to enlarge

To create a generic system job workflow

  1. Create a generic system job template in moab.cfg that will set up the VM.
    1. Give the template a unique name (createVM).
    2. Set the INHERITRES attribute to TRUE.
    3. Configure the trigger to launch a script that creates the VM (see Generic system job trigger requirements). Set a variable to hold the VM's IP address.
    JOBCFG[createVM] GENERICSYSJOB=EType=start,AType=exec,Action="$HOME/installVM.py $HOSTLIST",Timeout=1:00:00,sets=^IPAddr
    JOBCFG[createVM] INHERITRES=TRUE
  2. Create a generic system job template to install software on the new VM.
    1. Give the template a unique name (installSoftware).
    2. Set the INHERITRES attribute to TRUE.
    3. Specify a dependency on the createVM job using the TEMPLATEDEPEND attribute (see the Job Dependency Syntax table for options). In this example, the installSoftware job begins after the createVM job completes successfully.
    4. Configure the trigger to launch a script that installs the software on VM, located via the IP address provided in the IPAddr variable.
    JOBCFG[installSoftware] GENERICSYSJOB=EType=start,AType=exec,Action="$HOME/setupSoftware.py $IPAddr",Timeout=30:00,Flags=objectxmlstdin
    JOBCFG[installSoftware] INHERITRES=TRUE
    JOBCFG[installSoftware] TEMPLATEDEPEND=AFTEROK:createVM

    If you are using your workflow in Moab Web Services or Viewpoint, the trigger attached to the last setup job must pass up the Deployed variable upon success.

  3. Create a template for the job that will act as the gate to the workflow. Do not make it a generic system job.
    1. Give the template a unique name (createVMWithSoftware).
    2. Set a dependency on the previous job. In this case, createVMWithSoftware should begin after installSoftware completes successfully.
    3. Set the SELECT attribute to TRUE to indicate that users can request the workflow using this job (see Applying templates based on job attributes for more information). The SELECT attribute also makes the workflow available in the Viewpoint service templates.
    4. Set the VMTRACKING and NORMSTART flags. When you set the VMTRACKING flag on a job, Moab automatically changes the SYSTEMJOBTYPE to vmtracking.
    5. Configure the DESTROYTEMPLATE attribute to point to the destroy job in the workflow. When this job is canceled, the destroy job runs a script to delete the service.
    JOBCFG[createVMWithSoftware] TEMPLATEDEPEND=AFTEROK:installSoftware SELECT=TRUE FLAGS=NORMSTART,VMTRACKING
    JOBCFG[createVMWIthSoftware] DESTROYTEMPLATE=destroy
  4. Create the destroy job template, specifying that it is a generic system job. Attach a trigger with a script that will take the location of the VM or other workflow service from the IPAddr variable and destroy the workflow. 
    JOBCFG[destroy] GENERICSYSJOB=EType=start,AType=exec,Action="$HOME/destroy.py $IPAddr",timeout=5:00

  5. If you wish to implement an automatic VM migration policy, you must similarly configure a migration job and link to it via the MIGRATETEMPLATE attribute.

    To request the VM, submit a job with the createVMWithSoftware job template and any other desired resources. You must specify a script to satisfy the msub syntax, but it will not actually run. 
    > msub -l walltime=2:00:00,template=createVMWithSoftware,nodes=1:ppn=4,mem=1024 anyScript.py

    If ENABLEVMDESTROY is FALSE (default), Moab puts the VM-tracking job on hold when a VM build times out rather than deleting it. The job is visible in the showq report. Releasing the job hold causes Moab to attempt to pick up the VM again.

    Moab creates a job and applies the createVMWithSoftware template to it. This includes the creation of the installSoftware and createVM jobs. The INHERITRES attribute causes the new jobs to adopt the same resource request as createVMWithSoftware (4 processors and 1 GB of memory). Moab then applies their own templates.

    The createVM job runs first, the trigger script returning the IP address of the new VM and setting it as a variable. The job passes its allocation to the installSoftware job. installSoftware uses the IP address variable to locate the VM and install software on it. The job returns its resources to createVMWithSoftware, which is now the VM-tracking job.