(Click to open topic with navigation)
There are multiple steps associated with setting up a working preemption policy. With preemption, issues arise because it appears that Moab is not allowing preemptor jobs to preempt preemptee jobs in the right way. To diagnose this, use the following checklist:
| Verify that preemptor jobs are marked with the PREEMPTOR flag. (Verify with checkjob <JOBID> | grep Flags.) | |
| Verify that preemptee jobs are marked with the PREEMPTEE flag. (Verify with checkjob <JOBID> | grep Flags.) | |
| Verify that the start priority of the preemptor job is higher than the priority of the preemptee job. (Verify with checkjob <JOBID> | grep Priority.) | |
| Verify that the resources allocated to the preemptee job match those requested by the preemptor job. | |
| Verify that the preemptor job is within the 32-preemptee limit. | |
| Verify that there are no policies preventing preemption from occurring. (Verify with checkjob -v -n <NODEID> <JOBID>.) | |
| Verify that the PREEMPTPOLICY parameter is properly set. (See PREEMPTPOLICY types.) | |
| Verify that the preemptee job is properly marked as restartable, suspendable, or checkpointable. (Verify with checkjob <JOBID> | grep Flags.) | |
| Verify that GUARANTEEDPREEMPTION is set to TRUE. | |
| Verify that JOBNODEMATCHPOLICY is not set to EXACTNODE. Moab does not currently consider EXACTNODE when it handles preemption, resulting in unexpected behavior when EXACTNODE is set in an environment with preemption. | |
| Verify that NODEACCESSPOLICY is not set to SINGLEUSER. (SHARED is recommended.) | |
| Verify that BACKFILLPOLICY is set to FIRSTFIT. | |
| Verify that the resource manager is properly responding to preemption requests. (Use mdiag -R.) | |
| If there is a resource manager level race condition, verify that Moab is properly holding target resources. (Verify with mdiag -S and set RESERVATIONRETRYTIME if needed.) |
Related topics