(Click to open topic with navigation)

15.0 Cluster Analysis, Testing, and Simulation > Simulations > Configuring Simulation

Conventions

15.3.1 Configuring Simulation

This section explains how to set up simulation mode in Moab.

Configuring Moab Simulation

  1. Determine the performance metrics. The first step of most simulations is to determine the primary purpose of the simulation. Purposes may include identifying impact of certain resource or workload changes on current cluster performance. Simulations may also focus on system utilization or workload distribution across resources or credentials. Further, simulations may also be used for training purposes, allowing risk-free evaluation of behavior, facilities, and commands. With the purpose known, metrics of success can be specified and a proper simulation created. While performance metrics may not be critical to training based simulations, they are key to successful evaluation in most other cases.
  2. Select resources. As in the real world, a simulation requires a set of resources (compute hosts) on which to run. In Moab, this is specified using a resource trace file. This resource trace file may be obtained from specific hardware or generated for the specific purpose.
  3. Select workload. In addition to resources, a simulation also requires a workload (batch jobs) to schedule onto the available resources. This workload is specified within a workload trace file. Like the resource traces, this workload information may be based on recorded data or generated to meet the need of the particular simulation.
  4. Select policies. The final aspect of a simulation is the set of policies and configuration to be used to determine how a workload is to be scheduled onto the available resources. This configuration is placed in the moab.cfg file just as would be done in production (or normal) mode operation.
  5. Set up the initial configuration using the sample traces. While mastering simulations may take some time, initial configuration is straightforward. To start, edit the moab.cfg file and do the following:
    1. Change the SCHEDCFG attribute MODE from NORMAL or MONITOR to SIMULATION.

      Once SIMULATION is set, the following parameters control the environment, behavior, and policies used within the simulation:

    2. You may need to add these lines to the moab.cfg file:
      3. SIMWORKLOADTRACEFILE   samples/workload.testcluster.txt
STOPITERATION          0
      CREDDISCOVERY	       TRUE
SIMAUTOSHUTDOWN     	false
SIMSTARTTIME		1196987696
USERCFG[DEFAULT] 	ENABLEPROFILING=true 
GROUPCFG[DEFAULT] 	ENABLEPROFILING=true
ACCOUNTCFG[DEFAULT] 	ENABLEPROFILING=true
CLASSCFG[DEFAULT] 	ENABLEPROFILING=true
QOSCFG[DEFAULT] 	ENABLEPROFILING=true

      The first two lines specify that the scheduler should run in simulation mode and use the referenced resource and workload trace files. In addition, leaving the STOPITERATION parameter at zero indicates that Moab should stop before the first scheduling iteration and wait for further instructions. If you want the simulation to run as soon as you start Moab, remove (or comment out) this line. To continue scheduling, run the mschedctl -r command.

      The second set of parameters is helpful if you want to generate charts or reports from Moab Cluster Manager. Since events in the workload trace may reference credentials that are not listed in your moab.cfg file, set CREDDISCOVERY to true, which allows Moab to create simulated credentials for credentials that do not yet exist. Setting SIMAUTOSHUTDOWN to false prevents Moab from terminating after it has finished running all the jobs in the workload trace, and it allows you to generate charts after all the simulated jobs have finished. Ensure that SIMSTARTTIME is set to the epoch time (in seconds) of the first event in your workload trace file. This causes the internal clock in Moab to be set to the workload trace's first event, which prevents issues caused by the difference between the time the workload trace was created and the time reported by the CPU clock. Otherwise, Moab thinks the current time is the time that the CPU clock reports, yet simulated jobs that are reported by showq as currently running will really be running at the time the workload trace was created. To avoid confusion, set the SIMSTARTTIME. The lines that specify ENABLEPROFILING is true are necessary for Moab to keep track of the statistics generated by the simulated jobs. Not setting these lines will cause charts and reports to contain all zero values.

  6. Start a simulation. As in all cases, Moab should be started by issuing the command moab. It should be noted that in simulation mode, Moab does not daemonize itself and so will not background itself. Verification of proper operation is possible using any common user command such as showq. If the showq command is run, it will display the number of jobs currently in the scheduler's queue. The jobs displayed by the showq command are taken from the workload trace file specified earlier and those that are marked as running are running on resources described in the resource trace file. At any point, a detailed summary of available resources may be obtained by running the mdiag -n command.