The Quest of Encrypted VMotion

Last week I was dreaming (don’t even ask…. 😉 about the new VMotion encryption technology that VMware has implemented in vSphere.
Since that moment I decided that I want to know more about this feature since there isn’t much said about it at the moment.

Let’s go back before vSphere: Ever since VMotion exists I’m spreading the word that VMotion isn’t secure and that you need to take that knowledge into account when designing a Virtual Infrastructure.
Most of the time this resulted in creating a separate (V)LAN which is only accessible for the VMotion interfaces, making it hardly impossible for anyone to sniff the guest memory as it unencrypted crosses the line.

Starting sniffing the VM guest memory
In my home-lab I’ve created a scenario in where I can sniff the VM guest memory as it passes the network during a VMotion.  To go more into detail: 

1) Sniffing is done with Wireshark installed on a VM connected to the same VLAN ID as VMotion;
2) Promiscuous mode is set to accept on the VM Portgroup where the “Wireshark” VM is connected to.

First of all I calculated the average time it takes for multiple a default Windows 2003 Enterprise (x86) to get VMotionned on my isolated test network and at the same time I’ve sniffed the guest memory for further analyses.

Unencrypted Guest Memory
On my Windows 2003 test VM I’ve opened notepad with some text in it. (Didn’t write it to disk so I’m sure it’s in memory).
When analyzing the sniff I noticed that I indeed could see my whole Notepad contents:

 

And of course some other plain text system properties:

Enabling VMotion Encryption
So now off to the fun part, enabling VMotion Encryption! (this is done within the “vCenter Server Settings” > “Advanced Settings”) What I noticed first is that you have to make a selection between either “Best Effort” and “Required”

 

 

 

Well, for my test I selected “Required” and applied by pressing “OK”
Ready to go, let’s fire that Encrypted VMotion so I can evaluate how much impact this encryption has on my environment.

Unfortunately my Wireshark sniffing still shows me plain text that’s crossing the network. How can this be?
I’ve restarted several services and even the ESX Hosts to make sure that the “VirtualCenter.VMotionEncryption” setting was applied correctly, but neverentheless my VMotion stays unencrypted.

In my journey for an explanation I’ve first checked the VirtualCenter Logging (Trivia Logging Level) to see if something is logged while I change the “VirtualCenter.VMotionEncryption” property, this was the case so that should be fine than. Next stop, searching the ESX log files. The /var/log/vmware/hostd.log has got some interesting information:

VMotionPrepare (1249920019465893): Sending ‘to’ srcIp=10.0.0.2 dstIp=10.0.0.1, type=1, encrypted=false, remoteThumbprint=

So it’s saying that it’s unencrypted. Rather strange though since I set the “VirtualCenter.VMotionEncryption” option to “Required”. Shouldn’t it fail my VMotion than?

Just as a side note, I’m running a clean out-of-the-box vSphere environment with self-signed certificates. SSL is set to “vCenter requires verified host SSL certificates” and all the ESX Hosts thumbprints are verified by me. The “remoteThumbprint=” part in the hostd.log doesn’t satisfy me so I’m guessing the problem has something to do with SSL.

Next step: I’ve created a Root Certificate Authority and signed my vCenter server as well as all my ESX Hosts. No invalid certificate errors exist anymore while connecting with the VIClient towards vCenter nor to the ESX Hosts.

Let’s try again……. Unfortunately… still no encryption.

I’m breaking my head over this one at the moment, since I’m running out of options at the moment. I’ve even tried several different network settings, I even tried running it through a dvSwitch (well, I didn’t specifically setup a dvSwitch for this, it’s just that my second cluster was running with these network settings 😉

Obviously I will continue my research on this item but for now I guess it’s time for a good night sleep.

To be continued…