Dave Hill recently wrote an article about running nested ESXi and a gotcha with EVC (Enhanced vMotion Compatibility). In vSphere 4.x, you could not join a nested ESXi host into a cluster with EVC enabled. With vSphere 5, there's actually a way to connect a nested ESXi 5 host to an EVC enabled cluster AND still power on 64bit nested guestOSes.
I have to thank my friend and partner in crime Tuan Duong for showing me this trick awhile back. Tuan was performing some tests using both nested and physical ESXi 5 hosts and discovered this method after a bit of tinkering. At the time, I was not sure if others would find this useful and I did not document the process.
Disclaimer: As usual, this is not officially supported by VMware, use at your own risk.
Here are the steps:
1. You must be running vSphere 5, create a nested ESXi 5 host using this article How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5
2. Create an EVC enabled cluster or use an existing cluster with whatever baseline you would like and click on the "Current CPUID Details" in cluster settings.
3. Copy down the CPU mask flags for that particular EVC baseline, you will need this in the next step
4. Shutdown your nested ESXi 5 host and edit the VM's settings and under "Options" tab click on "CPUID Mask->Advanced". You will take the CPU mask from the above step and update the nested ESXi 5 VM to make it match
5. Go ahead and power on your nested ESXi 5 host and join it to the EVC enabled cluster you created earlier. You should not see any errors when connecting to the cluster and after that you can create a nested 64bit VM within that virtualized ESXi 5 host.
There you have it, running a nested ESXi 5 host and joined to an EVC enabled cluster! Isn't VMware technology awesome! 🙂