First of all lets get the definitions of these objects summarized (as stated in the vSphere Datacenter Administration Guide)

Clusters

A collection of ESX/ESXi hosts and associated virtual machines intended to work together as a unit. When you add a host to a cluster, the host’s resources become part of the cluster’s resources. The cluster manages the resources of all hosts. VMware Features like EVC, DRS, DPM and HA are enabled on a per Cluster basis.

Datacenters

Unlike a folder, which is used to organize a specific object type, a datacenter is an aggregation of all the different types of objects needed to do work in virtual infrastructure:

• Virtual machines (and templates)
• Hosts (and clusters)
• Networks
• Datastores

The datacenter defines the namespace for networks and datastores. The names for these objects must be unique within a datacenter. For example, you cannot have two datastores with the same name within a single datacenter, but you can have two datastores with the same name in two different datacenters.

Virtual machines, templates, and clusters need not be unique within the datacenter, but must be unique within their folder.

Objects with the same name in two different datacenters are not necessarily the same object. Because of this, moving objects between datacenters can create unpredictable results. For example, a network named networkA in datacenterA might not be the same network as a network named networkA in datacenterB. Moving a virtual machine connected to networkA from datacenterA to datacenterB results in the virtual machine changing the network it is connected to.

Back to the the discussion about creating multiple Datacenters, this was mainly started because the organization wanted a static ”Folder” hierarchy per environment. Since the Datacenter object is the boundary for Folders their environment looked like this:

This design argument can easily be set aside since it’s possible to create Subfolders as well. So it could also look like this:

while preserving the one Datacenter design which, from a logic perspective they had.

The key message here is to be sure that you have thought well about the design decisions and that you preserve logics as much as possible to keep a clear view even when the the environment grows.

This is by design within ESXi and stated in this VMware KB article: ESXi uses UTC time and does not support changing time zones

Although it’s not supported to change the time zone it could come in handy to get the correct time-stamps for custom scripts within the ESXi Shell.
To get the correct time-stamps you only need to add the following line in the beginning of your script:

export TZ=MET<-/+><no of hours>

In my case UTC is off with -1 hour with the actual time, which means that I need: export TZ=MET-1

Please note that this command only changes the time zone for the current Shell session as long as it’s active, so it doesn’t change the system time zone.
Also don’t forget to check the output with daylight saving time if applicable to your current time zone.

Note that  Frank Denneman, one of the two great authors of the VMware vSphere Clustering Technical Deepdive (available now!) will also be attending. This is a great moment to meet up with him!

Drinks will not be paid for, there will not be a tab. When you buy a drink please pay for it as no one else will be paying for your drinks.

• Location: ‘Cafe de Omval’ Amsterdam
• Address: Weesperzijde 250 (hoek Omval), 1097 EB Amsterdam
• Nearest Train Station: Amsterdam Amstel
• Time: 6:00pm
• Location: Map

One of the design considerations is around the placement of the Virtual Machine Swap File (.vswp) which I want to give some more information about in this article.

Let’s first take a look at the VM Swap File (.vswp), by default this file is placed in the VM “working directory” which also contains all the other VM files. The .vswp is created every time the VM is started and equals the size to the unreserved memory configured on the VM. If the VM is configured with 2 GB and memory reservation is set to 0 MB (default) the VM Swap File will be 2 GB. If memory reservation in this example would be 1 GB than the .vswp file will be 2GB – 1GB = 1 GB total.

The design considerations are about:

1. Keeping the .vswp file in its default “working directory”;
2. Placing the .vswp on a separated non-replicated datastore.

Keeping the .vswp file in its default “working directory” means that the .vswp file will be replicated to the recovery site as indicated in the next overview:

Pros:

• Ease of manageability, all VM files are together and it’s default;

Cons:

• More replication bandwidth is needed for files (.vswp) that aren’t used at the recovery site;
• Cost is higher since more replicated storage space is used;
• Increases the recovery speed of both the test and real failover. This is due to the fact that SRM explicitly deletes the useless .vswp files on the recovery site before starting the VM’s.

Placing the .vswp on a separated non-replicated datastore involves some manual work, possibly even reconfiguration of all the current available Virtual Machines. The following overview shows the configuration:

Pros:

• Does not consume unused storage replication traffic;
• Uses less replicated storage, which could be more expensive than non-replicated storage.

Cons:

• Can be more difficult to manage because some parts of a virtual machine reside on a separate datastore.
• Requires additional configuration and management processes within SRM since the VM would be detect with a non-replicated datastore which consequently causes SRM to remove the VM from its protection group.

An important note that needs to be made is around NFS storage. As indicated, one of the drawbacks on keeping the .vswp file in the “working directory” is the fact that this increases the recovery speed since SRM deletes the replicated, useless, .vswp file before starting the VM.

Deleting the .vswp file from a newly recovered NFS datastore can take up some time since ESX needs to wait for the replicated file lock to expire (default 35 seconds).  A quote from the the Best Practices on NAS Whitepaper:

Once a lock file is created, VMware periodically (every NFS.DiskFileLockUpdateFreq seconds) send updates to the lock file to let other ESX hosts know that the lock is still active. Changing any of the NFS locking parameters will change how long it takes to recover stale locks. The following formula can be used to calculate how long it takes to recover a stale NFS lock:

(NFS.DiskFileLockUpdateFreq * NFS.LockRenewMaxFailureNumber) + NFS.LockUpdateTimeout

If any of these parameters are modified, it’s very important that all ESX hosts in the cluster use identical settings. Having inconsistent NFS lock settings across ESX hosts can result in data corruption!

This timeout isn’t applicable on VMFS datastores because the auto-resignaturing process  drops the file locks automatically.

As with every design decision it’s all about knowing the pros/cons of the available options you have and as such select the best option for your environment.

As a contractor I will be available to do Virtualization Consultancy, primarily focused on designing, implementing and/or validating VMware Virtualization solutions with a strong focus on business continuity, high availability and disaster recovery.

I want to thank VMware for their understanding and I want to thank all the great people I met while working for VMware since they definitely have the best people around!

Thanks guys, all the best and we will certainly stay in touch!

The Device Identifiers are separated into the following items:

Getting the Device Identifiers on your ESX(i) Host is done by taking a look at the current running PCI config by issuing the following command:

For an ESX Host: cat /proc/vmware/pci

For an ESXi Host: lspci –p

The output will display all the Device Identifiers needed to verify the I/O device against the VMware HCL. (in this example the details of a HBA).

More information on Device Identifiers in combination with PowerShell can be found in this excellent article of Luc Dekens.

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

Running out of VMFS3 Heap Space can occur when a large quantity of virtual disk space (.vmdk files) is open on a single ESX Host.

The default VMFS3.MaxHeapSizeMB settings are:

• 16 MB for ESX 3.5/4.0 – allows for a maximum of 4TB of active virtual disk space on a single ESX Host.
• 80 MB for ESX 4.1 – allows for a maximum of 20TB active virtual disk space on a single ESX Host.

The maximum value that can be set is 128MB which allows for a maximum of 32TB of active virtual disk space on a single ESX Host.

Important Note: All limits of “active virtual disk space” that are described above are applicable only for 1 MB VMFS Blocksizes. 2 MB VMFS Blocksize will double these numbers and 4 MB VMFS Blocksize will quadruple these numbers, etc.

The following example shows the maximum values when using a VMFS Blocksize of 8MB with again the default VMFS3.MaxHeapSizeMB settings:
(If you want to read more about VMFS Blocksizes I would recommend this excellent article by Didier Pironet)

• 16 MB for ESX 3.5/4.0 – allows for a maximum of 32TB of active virtual disk space on a single ESX Host.
• 80 MB for ESX 4.1 – allows for a maximum of 160TB active virtual disk space on a single ESX Host.

Using the maximum of 128 for the VMFS3.MaxHeapSizeMB setting allows up to 256TB of active virtual disk space on a single ESX Host.

It is important to know about these values for your consolidation ratio/design and to know what the effect is of increasing it. Nowadays physical servers become bigger and bigger, allowing you to run more and more Virtual Machines on them which increases the risk of hitting the limit of active virtual disk space on a single ESX Host.

As stated the maximum value can be set to 128MB. The Heap is part of the kernel memory so increasing its memory will increase memory consumption of the kernel which results in less memory for other consumers on the system.

• Real-time data
• Historical data

Real-time Data
 The Real-time data is displayed for the last hour only and is directly queried from the ESX Host (vpxa) every 20 seconds.
Because the ESX Host stores the performance data locally for the last hour you can have a vCenter outage for up to one hour without getting any gaps in the historical data.

Historical Data
The process outlined below described the steps that are taking place to collect and store the Historical Data.

1. Every 5 minutes 1/6^th of the total amount of ESX Hosts (vpxa) is queried for performance data;
2. The vpxa sends its data to vCenter (vpxd) which stores the data in the VPX_HIST_STAT1 table.

This process happens every 5 minutes which means that every ESX Host is queried once every 30 minutes.

Let’s take a look at a 10 ESX Host cluster for example:

10.00am: ESX01, ESX02 are queried
10.05am: ESX03, ESX04 are queried
10.10am: ESX05, ESX06 are queried
10.15am: ESX07, ESX08 are queried
10.20am: ESX09, ESX10 are queried
10.25am: ESX11, ESX12 are queried
10.30am: ESX01, ESX02 are queried again.
Etc.

Every 30 minutes the final statistics data rollup and purge store procedures will run. These database roll-up jobs are either controlled by the SQL Agent (in both MS SQL and Oracle) or by vCenter (vpxd) internally. The vpxd internal scheduler is only used with Microsoft SQL Express Editions.

The corresponding database tables that are used to store the rollup data are outlined below:

The interval duration for the stored data is a configurable item from within vCenter as you can see in the screenshot below.

Please note that changing the Interval durations can have a major impact on the database size and vCenter performance due to excessive load.

To conclude, knowing the difference between the Real-time and Historical Data querying and the way the data is processed might benefit you during troubleshooting.

Before discussing the options you have in teaming/load balancing lets focus a bit more on the VMware FT Interface. VMware FT keeps the Primary and Secondary VMs in lockstep using VMware vLockstep technology. vLockstep technology ensures that the Primary and Secondary VMs execute the same x86 instructions in an identical sequence. The Primary VM captures all nondeterministic events and sends them across the VMware FT logging network to the Secondary VM. The Secondary VM receives and then replays those nondeterministic events in the same sequence as the Primary VM, typically with a very small lag time.

The FT interface is configured per ESX(i) Host as a vmkernel port and needs an unique IP address assigned. This unique IP address is used to send out the logging traffic as shown in the figure below.

Looking at a typical design with two vmnics configured for FT you can set the failover configuration to either active/active or active/passive as shown in the overview below.

or    

The active/passive connection is pretty straight forward, in case the active adapter fails the standby adapters takes over seamlessly.

The active/active connection however could cause some confusion since we have a few options in there which I like to explain a bit more.

Active/Active with default “Route based on the originating virtual port ID” load balancing

Since the FT Interface has only got one MAC address it will distribute the traffic to only one vmnic in this configuration. The vmnic that will be used is the first vmnic that listed in the “Active Adapters” as shown in the overview below. vmnic1 will utilize all FT logging traffic and vmnic5 will only be used whenever vmnic1 fails.

Note that you can change the order of active vmnic dynamically by just moving another vmnic to the first position in the list.

Active/Active with “Route Based on source MAC hash”

Equally to the “Route based on the originating virtual port ID”

 Active/Active with “Route based on IP Hash”

IP Hash load balancing is another policy that you can use on the FT logging network. This policy uses both source and destination IP addresses of the ESX(i) Hosts that are running FT-protected VM’s. Etherchannel /Link Aggregation is mandatory on the physical network to be able to correctly use the IP Hash algorithm.

In my opinion there is a risk in using the “Route based on IP Hash” load balancing policy. Generally you only enable FT on important VMs that need to meet very high availability SLAs. Using the IP Hash load balancing method will utilize both vmnics, spreading out the load, but  keep in mind what happens whenever one vmnic will fail. You don’t want to get in a situation with congestion on the FT network which could impact Fault Tolerance since this is a key element in your high availability.

The key take-away here is to hand out the design decisions and let you make the best selection for your specific scenario.

By default vCenter wants to use port 80 for the HTTP service and using the netstat command you can verify which process is listening on port 80. In my case Windows wasn’t able to show the actual process name other than Process ID 4 which basically tells that the “system” is using the port.

The netstat -abo output displays: “Cannot obtain ownership information” and only shows Process ID 4

A lot of forums on the Internet talk about the fact that this is caused by Windows Remote Management (WinRM) that by default is using port 80. This story isn’t true since Windows Remote Management isn’t listening by default even while it is started as described in this Microsoft article:

By default, no WinRM listener is configured. Even if the WinRM service is running, WS-Management protocol messages that request data cannot be received or sent

Also note that only the WinRM 1.1 and earlier is using port 80 by default when configured.

WinRM 1.1 and earlier: The default HTTP port is 80.

WinRM 2.0: The default HTTP port is 5985.

There is a VMware KB article about this problem that states that it could be IIS or Microsoft SQL Server Reporting Services (SSRS). Since I personally only know this error message from IIS that is using port 80, I wanted to know what the difference was in SQL 2008.

Basically the story is that SSRS now handles HTTP via HTTP.SYS, a native OS kernel that intercepts and handles HTTP requests just like IIS. The reason for Microsoft to use this method is to make the SSRS independent of IIS.

So with this knowledge you can relate to the fact that netstat isn’t displaying the process name other than the process ID 4 which is pointing out to the “system”. To get vCenter Installed on port 80 ensure that the SSRS Service is stopped (or uninstalled when it’s not needed on the system.).