Z.15 Configuring Simulation

Simulation Setup

This section explains how to set up a simulation in Moab.

To prepare for Moab simulation

  1. Determine objectives. The first step of any simulation is to determine the primary purpose of the simulation. Purposes may include identifying impact of certain policy, resource, or workload changes on the behavior and performance of real or hypothetical clusters. These changes include impacts on system utilization, workload responsiveness, or workload distribution across resources or credentials. Further, simulations may also be used for training purposes, allowing risk-free evaluation of behavior, facilities, and commands. With the purpose known, metrics of success can be specified and a proper simulation created. While performance metrics may not be critical to training based simulations, they are key to successful evaluation in most other cases.
  2. Specify resources. As in production scheduling, a simulation requires a set of resources (compute hosts) on which to run. In Moab, this is specified using a native resource manager. The resource information passed to the NativeRM may be obtained from specific hardware or generated for the specific purpose. This resource information is specified via a native resource manager using the RMCFG parameter as in the example below. The contents of the file specified with the CLUSTERQUERYURL attribute follow the Wiki resource format.
    # moab.cfg
    RMCFG[simcluster]      TYPE=native CLUSTERQUERYURL=file:///home/user1/moabsim/etc/sim.nodes
    
  3. Specify workload. In addition to resources, a simulation also requires a workload (jobs) to schedule onto the available resources. This workload is specified within a workload trace file. Like the resource traces, this workload information may be based on recorded data or generated to meet the needs of the particular simulation. The workload information is specified using the SIMWORKLOADTRACEFILE parameter. This parameter points to a flat text file containing job information stored in the Moab job event record format.
    # moab.cfg
    SIMWORKLOADTRACEFILE   sim.jobs
    
    
    NOTE: The location of the workload trace file can be specified as either an absolute or relative path. If relative, it is relative to $MOABHOMEDIR/etc.

    Parameters which modify workload specification, queueing, and job management within the simulation are included in the table below.

    ParameterDescription
    SIMINITIALQUEUEDEPTHSpecifies simulation backlog.
    SIMJOBSUBMISSIONPOLICYSpecifies how simulation backlog is managed.
    SIMPURGEBLOCKEDJOBSRemoves jobs that can never run.
    SIMWORKLOADTRACEFILESpecifies source of job traces.

  4. Specify timeframe. Simulations continue over a specified timeframe or run until a specified workload is processed. Which approach is used depends on the results which are desired. By default, the simulation continues running forever, even if there is no workload remaining.

    The table below contains parameters which modify simulation time management.

    ParameterDescription
    SIMEXITITERATIONSpecifies simulation backlog.
    RMPOLLINTERVALSpecifies the duration (in seconds) of the simulation tme step.
    SIMPURGEBLOCKEDJOBSRemoves jobs that can never run.
    SIMWORKLOADTRACEFILESpecifies source of job traces.

  5. Specify policies/behaviors. The next aspect of a simulation is the set of policies and configurations to be used to determine how a workload is to be scheduled onto the available resources. This configuration is placed in the moab.cfg file just as would be done in production (or normal) mode operation. All moab policies including fairshare, reservations, priorities, and quotas can be configured using standard Moab parameters. In addition, a few behavior modifying simulation parameters are provided including those listed in the table below:
    ParameterDescription
    SIMFLAGSSpecifies modifications to how the simulation loads and creates jobs from the simulation job trace.

  6. Enable simulation mode. The final step of enabling a simulation is to configure Moab to run in simulation mode using the MODE attribute of the SCHEDCFG parameter. If running on a server which is running another instance of Moab, be certain to set this up in a different Moab home directory and have it run with a different server port to avoid any conflicts with the other Moabs running.

Running a sample Simulation

This section explains how to run a sample Moab simulation.

To run the sample Moab simulation

A sample simulation can be run using the sample simulation traces included with the Moab package. While mastering simulations may take some time, initial configuration is straightforward. To start, edit the moab.cfg file and do the following: Change the SCHEDCFG attribute MODE to SIMULATION.

Once SIMULATION is set, the following parameters control the environment, behavior, and policies used within the simulation:

  • Simulation Workload Specification, Queuing, and Management
  • ParameterDescription
    SIMINITIALQUEUEDEPTHSpecifies source of job traces.
  • Time/Iteration Management
  • ParameterDescription
    SIMAUTOSHUTDOWNShuts down when all jobs have been scheduled.
    SIMSTARTTIMESets simulation clock to specified time.
    STOPITERATIONPauses simulation on specified iteration.
  • Example moab.cfg file:
  • # moab.cfg
    SCHEDCFG[clusterA]     SERVER=mgmt.mydomain:42559 MODE=SIMULATION
    SIMWORKLOADTRACEFILE   samples/workload.testcluster.txt
    RMCFG[simcluster]      TYPE=native CLUSTERQUERYURL=file:///home/user1/moabsim/etc/sim.nodes
    STOPITERATION          0
    SIMAUTOSHUTDOWN        false
    SIMPURGEBLOCKEDJOBS           true
    REJECTINFEASIBLEJOBS      true
    RECORDEVENTLIST        jobend
    

    The first two lines specify that the scheduler should run in simulation mode and use the referenced resource and workload trace files. In addition, leaving the STOPITERATION parameter at zero indicates that Moab should stop before the first scheduling iteration and wait for further instructions. If you want the simulation to run as soon as you start Moab, remove (or comment out) this line. To continue scheduling, run the mschedctl -r command.

    The second set of parameters is helpful if you want to generate charts or reports from Moab Cluster Manager. Since events in the workload trace may reference credentials that are not listed in your moab.cfg file, set CREDDISCOVERY to true, which allows Moab to create simulated credentials for credentials that do not yet exist. Setting SIMAUTOSHUTDOWN to false prevents Moab from terminating after it has finished running all the jobs in the workload trace, and it allows you to generate charts after all the simulated jobs have finished. Ensure that SIMSTARTTIME is set to the epoch time (in seconds) of the first event in your workload trace file. This causes the internal clock in Moab to be set to the workload trace's first event, which prevents issues caused by the difference between the time the workload trace was created and the time reported by the CPU clock. Otherwise, Moab thinks the current time is the time that the CPU clock reports, yet simulated jobs that are reported by showq as currently running will really be running at the time the workload trace was created. To avoid confusion, set the SIMSTARTTIME. The lines that specify ENABLEPROFILING is true are necessary for Moab to keep track of the statistics generated by the simulated jobs. Not setting these lines will cause charts and reports to contain all zero values.

    Launching a Simulation

    As with production execution, Moab simulation is started by issuing the command moab. In simulation mode, Moab does not daemonize itself and so will not background itself. If desired, Moab simulations can be started with the Moab daemon in the background using 'moab &'.

    When starting a simulation, a few considerations:

    Verifying a Simulation

    Verification of proper operation is possible using any common user command such as showq. If the showq command is run, it will display the number of jobs currently in the scheduler's queue. The jobs displayed by the showq command are taken from the workload trace file specified earlier and those that are marked as running are running on resources described in the resource trace file. At any point, a detailed summary of available resources may be obtained by running the mdiag -n command. To determine the current simulation time, issue the showstats -v command.

    Terminating a Simulation

    At any point, a Moab simulation can be shut down using the command mschedctl -k. This will terminate the simulation and write out final performance statistics into the Moab log file.

    © 2016 Adaptive Computing