Overcommit Factor and Threshold

21.0 VMs > Overcommit Factor and Threshold

21.2 Overcommit Factor and Threshold

The two main configuration settings that govern how migrations work are the Overcommit Factor and Overcommit Threshold. Both can be applied to the processors and memory of virtual machines (VM's).

The Overcommit Factor and Threshold can be defined as a global default or on a per-node basis.

NODECFG[DEFAULT]  OVERCOMMIT=PROC:2.0,MEM:2.0  # This is the default global policy
NODECFG[node42]   OVERCOMMIT=PROC:3.0,MEM:3.0  # This is a node-specific policy for node42

Overcommit Factor defines the upper bound or maximum amount of VCPUs that can be created on any given hypervisor (HV). For example, if you have a hypervisor with 12 processors or cores (Moab sees them as 12 processors), and have an Overcommit Factor of 2.0 for procs, then Moab will not allow, under any condition, more than 24 VCPU's to be allocated on this hypervisor. Remember: a VM can have one or more VCPU's. So, in this example, the HV could only support 8 VM's if they all had 3 VPCU's each. It could support 4 VM's if they had 6 VPCU's each, and so forth.

The Overcommit Threshold defines how many VM's are allowed on a node if those VM's have a load being reported. The Overcommit Factor defines the maximum under any condition, but the Overcommit Threshold controls how many can be practically supported by a hypervisor due to load.

An Overcommit Threshold is a number between 0 and 1 and is interpreted as a percentage that is applied to the number of configured processors. It is not applied to the overcommitted processor count. For example, if we have an Overcommit Threshold of 0.7 for CPUs and a hypervisor with 12 configured processors, then that HV can support a CPU load of up to 8.4 before Moab will try to migrate VM's off of it. Moab uses the CPULOAD reported for the hypervisor to determine if the threshold is exceeded.

An example using both the Overcommit Factor and Overcommit Threshold is as follows:

We have a hypervisor with 12 procs, an overcommit factor of 2.0 and a threshold of 0.7.
We initially create VM's on the hypervisor. Each VM has a VCPU count of 1 and a CPULOAD of 0.1. We keep creating VM's on the hypervisor until we have 24. At this point, Moab will not any more VMs to be created because the Overcommit Factor has been reached (12x2.0=24). The hypervisor reports a load of about 2.4 (this will usually be greater than the sum of the VM loads due to overhead and VMs that are not under Moab's control). The load is well below the threshold of 0.7 (threshold) * 12 (number of processors) = 8.4, so no VM's need be migrated.
```
#Example of the above implemented in Moab.cfg 
NODEAVAILABILITYPOLICY      UTILIZED
VMMigrationPolicy           OVERCOMMIT,CONSOLIDATION@10:00:00
NODECFG[DEFAULT]            VMOCThreshold=PROC:0.7,MEM:0.95
NODECFG[DEFAULT]	     OVERCOMMIT=PROC:2.0,MEM:2.0
```
The example above first specifies NODEAVAILABILITYPOLICY UTILIZED. This tells Moab to use what the resource manager is reporting as being used for resources. Next the OVERCOMMIT flag must be set on VMMigrationPolicy for the VMOCTHRESHOLD attribute to function. CONSOLIDATION is then also specified with an Overcommit migration even time of 10:00:00 or 10 minutes. Next VMOCThreshold is then specified to overcommit PROC by .7 and MEM by .95 allowing PROC utilization to reach 8.2 and MEM to reach 22.8 Gigs (.95 x 24) before action needs to be taken. Finally each node is given a default OVERCOMMIT for PROC by 2 and MEM by 2. This could allow up to 24 VM's at 1 processor and 2 Gigs of memory each.
Over time, the VMs start to get used more and their CPULOAD increases. Soon, four of the VM's load shoots up to 3. This brings the load to 12 x 3.0 = 36. This is above the Overcommit Threshold of 8.4, so Moab will now need to migrate VM's during the next Overcommit Migration event.
When the Overcommit Migration event occurs, Moab will migrate enough of the high-load VMs off of the hypervisor to bring the load back down below 8.4. In this example, Moab would need to migrate at least TWO of the high-load VMs, bringing the total load down to 6.2. Once these two VM's have been migrated, Moab should not migrate any more, as we are now below our threshold. Moab should migrate these two VM's in the same Overcommit event--it should not take multiple events to migrate both of them.

Example of Overcommit Migration event

#GMETRIC threshold based triggers
# WLM Metric Threshold to check file system utilization
GEVENTCFG[disk_free] ACTION=fail SEVERITY=4
NODECFG[DEFAULT] TRIGGER=atype=exec,etype=threshold,failoffset=1:00,threshold=gmetric[disk_free]>90,action="/opt/moab/tools/filesize_fault.py $OID $METRICTYPE"

In the above example, a GEVENT of type disk_free is created. This is one of the predefined GEVENTS provided by Moab. The action is set to fail if the event is triggered and give it an arbitrary SEVERITY of 4. Next, the nodes to be defined with this event are specified. In this example this is applied to all nodes. If the threshold is over 90, Moab is informed the following action will take effect: /opt/moab/tools/filesize_fault.py $OID $METRICTYPE

Example of supported GMETRIC. Note this is not an exhaustive list:

"bytes_out"Number of net bytes out per second 
"cpu_num" Number of CPUs 
"cpu_speed" processor speed (in MHz) 
"disk_free" Total free disk space (GB) 
"disk_total" Total available disk space 
"load_one" One minute load average 
"machine_type" cpu architecture 
"mem_free" Amount of available memory (KB) 
"mem_total"  Amount of available memory 
"os_name"
"os_release" operating system release 
"pkts_in" NYI */ / Packets in per second (packets/sec) 
"pkts_out" NYI Packets out per second 
"swap_free" Amount of available swap memory (KB) 
"swap_total"  Total amount of swap memory

Note that the Overcommit Factor and Threshold should also apply when selecting a VM destination. If a VM needs to be migrated off of a loaded hypervisor Y, but moving it to hypervisor X would cause X's load or overcommit factor to be violated, Moab cannot move it to X. It must try to find another location. Also, the example above dealt only with CPU or processor counts, but Overcommit Factor and Threshold also apply to a wide array of system resources. You can also create your own with Ganglia.