Checkpointing Jobs with Preemption
Context
CHECKPOINT is one of the PREEMPTPOLICY types (for more information, see PREEMPTPOLICY Types). For systems that allow checkpointing, the CHECKPOINT attribute allows a job to save its current state and either terminate or continue running. A checkpointed job may restart at any time and resume execution from its most recent checkpoint.
You can tune checkpointing behavior on a per-resource manager-basis by setting the CHECKPOINTSIG and CHECKPOINTTIMEOUT attributes of the RMCFG parameter.
For information about PREEPMPTEE and PREEMPTOR flags, see Preemption Flags
The following outlines some benefits of using CHECKPOINT and also lists some things you should be aware of if you choose to use it.
Advantages:
This attribute allows you to restart a job from its last checkpoint.
Cautions:
Jobs tend to take longer to complete when you use CHECKPOINT.
To preempt jobs using CHECKPOINT
Make the following configurations to the moab.cfg file:
- Set GUARANTEEDPREEMPTION to TRUE. (This causes Moab to lock PREEMPTOR jobs until JOBRETRYTIME expires.)(This locks the job on a node and keeps trying to preempt.)
- Make sure that JOBNODEMATCHPOLICY is not set to EXACTNODE, which is not currently supported for preemption (for more information, see Testing and Troubleshooting Preemption).
- Set PREEMPTPOLICY to CHECKPOINT (for more information, see PREEMPTPOLICY Types).
- Make sure that the PREEMPTEE job has a lower priority than the PREEMPTOR job (for more information, see Preemption Flags).
For example:
GUARANTEEDPREEMPTION TRUE PREEMPTPOLICY CHECKPOINT QOSCFG[test1] QFLAGS=PREEMPTEE MEMBERULIST=john PRIORITY=100 QOSCFG[test2] QFLAGS=PREEMPTOR MEMBERULIST=john PRIORITY=10000
Related Topics