BIOS settings in a VMware ESX(i)/vSphere Environment

Over the last period I’ve noticed that a lot of customers aren’t aware of BIOS settings that can be changed as a best practice for VMware ESX(i)/vSphere environments. In this article I want to outline some of these best practices specifically based on HP Hardware, but feature-like options are available within other vendors as well.

ASR: The Automatic Server Recovery (ASR) feature is a hardware-based timer. If a true hardware failure occurs, the Health Monitor might not be called, but the server will be reset as if the power switch is pressed. The ProLiant ROM code may log an event to the Integrated Management Log (IML) when the server reboots.

Personally I prefer to disable ASR since it’s possible that an ESX Host crashes (Purple Screen of Death – PSOD), VMware HA starts VM’s on the other hosts, ASR restarts the server (default timeout: 10 minutes), the ESX Hosts gets reconnected to the cluster, VMware HA is automatically reconfigured and based on the load DRS could move VM’s back to the host, waiting for another crash to occur…

Basically when an ESX Host crashes I suggest getting this Host out of service to let the administrators find the root cause of the problem first.

Virtual Install Disk: Since this feature is currently only available for Windows Operating systems it can be easily disabled for VMware vSphere deployments. The virtual install disk is a holding place within the system ROM that contains embedded boot drivers (such as SCSI or RAID controller drivers) that could be required to complete the operating system installation. Typically, boot drivers that are placed in the virtual install disk are either not included as part of the operating system media or updated for new controllers. Supported operating systems automatically find these drivers, eliminating the need for user intervention.

Power Management Options: There are a lot of CPU Power Management options currently available from both hard- and software perspective. Looking at the hardware we can summarize the following settings that are available on new generation HP servers:

• HP Dynamic Power Savings Mode (default)
• HP Static Low Power Mode
• HP Static High Performance Mode
• OS Control Mode

Since ESX(i) 4.1 we can fully control the Power Management options from within ESX when setting the BIOS to “OS Control Mode”. This method is currently recommended by HP as described in this document (pg. 5) (Thanks to my colleague Wade Holmes for pointing this out).

When the host is running lower utilization workloads, HP Power Regulator manages the CPU speed and voltage to save power. By default, the Power Regulator on the host server is in ‘Dynamic’ mode, which automatically manages CPU states to save power. However, this setting can result in incorrect measurement of CPU utilization by VMware ESX, which can affect DRS algorithms. In OS Control mode, processors will run at their maximum power/performance state at all times unless the OS enables a power management policy. A virtualized server is more likely to be highly utilized with multiple VMs, especially if using VMotion and DRS, which actively balance workloads. In this environment, it is better for administrators to configure Power Regulator in OS Control mode and allow DRS to manage the workloads and CPU efficiency.

See thisVMware KB article for additional information. Some applications may suffer whenever processor power management is implemented by either ESX or the server hardware.

The following message is displayed when OS Control mode is not active:

The following options are available when OS Control mode is active:

Since there isn’t a best practice on this setting I would recommend to simply testing it out. If there are any (application) performance issues and you suspect the CPU, it might be worth playing around with the Power Management Settings. Please note that although you can change the settings from ESX(i) instantly, a reboot of the server is still required.

CPU Vendor specific settings

AMD Virtualization: AMD Virtualization Technology is a set of hardware extensions to the x86 architecture that allows better utilization of resources. This feature needs to be activated for EVC and is required to run x64 Virtual Machines.

No-Execute Page-Protection: A technology used in the CPU’s to segregate areas of memory for use by either storage of processor instructions (or code) or storage of data. This feature needs to be activated for EVC compatible CPU’s to be supported.

Or

Intel VT: Intel VT is the hardware assistance for processors running virtualization platforms. Intel VT includes a series of extensions for hardware virtualization. This feature needs to be activated for EVC and is required to run x64 Virtual Machines.

Intel eXecute Disable: A technology used in the CPU’s to segregate areas of memory for use by either storage of processor instructions (or code) or storage of data. This feature needs to be activated for EVC compatible CPU’s to be supported.

To summarize I’ve created the following overview of settings that you should consider or are mandatory in your environment dependent on the features that are in use.