WilliamLam.com

Having Difficulties Enabling Nested ESXi in vSphere 5.1?

by // 21 Comments

I noticed there were a few folks having some difficulties enabling Nested ESXi (VHV Virtual Hardware Virtualization) in the latest release of ESXi 5.1 and I thought I share some additional info and tips on troubleshooting your setup in case you are running into similar problems.

*** DISCLAIMER **** This is not officially supported by VMware, do not bother asking if it is supported or calling into VMware support for details or help.

If you wish to run nested ESXi or other hypervisors on ESXi 5.1 and run 32-bit nested virtual machines, you must meet the following hardware requirement:

  • CPU supporting Intel VT-x or AMD-V

If you wish to run nested 64-bit virtual machines in your nested ESXi or other hypervisors, in addition to the requirement above, you must also meet the following hardware requirement:

  • CPU supporting Intel EPT or AMD RVI

If you only meet the first criteria, you CAN still install nested ESXi or other hypervisors on ESXi 5.1, BUT you will only be able to run 32-bit nested virtual machines. When you create your virtual machine shell using the new vSphere Web Client, in the expanded CPU view, the "Hardware Virtualization" box will be grayed out. This is expected as you do not have full support for VHV, but you can still continue with your installation of ESXi or other hypervisors.

In ESXi 5.0, you may have been able to run 64-bit nested virtual machines without EPT/RVI support but performance was extremely poor. With ESXi 5.1, VHV now requires EPT/RVI.

Note: During the installation of ESXi, you may see the following message "No Hardware Virtualization Support", you can just ignore it.

If you are using sites such as Intel's ark.intel.com to check your CPU requirements, be aware that it is COMMON even for the hardware vendors to publish incorrect information about their websites. However, there is a quick way you can validate on your ESXi host whether you have full VHV support.

In vSphere 5.1, there is a new capability property called nestedHVSupported which specifies whether your physical ESXi 5.1 host has full VHV support. This property will only be true IF your CPU has both Intel-VT+EPT or AMD-V+RVI. A quick and easy way to validate this is using the vSphere MOB to retrieve the value.

To check nestedHVSupported property, please enter the following into a web browser (substitute the IP Address/hostname of your ESXi host):

https://himalaya.primp-industries.com/mob/?moid=ha-host&doPath=capability

After you login, search for the nestedHVSupported property on the page and you should see a value of either true or false. As mentioned earlier, if it is false, you might still be able to install nested ESXi or other hypervisors but you will not be able to run nested 64-bit virtual machines. I would also recommend taking a look at your system BIOS to ensure things like Intel-VT/EPT and AMD-V/RVI are enabled and sometimes it might just be as simple as a BIOS upgrade (you can always confirm by contacting the hardware vendor if you have further questions).

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.

Additional Resources: 

2gbsparse Disk Format No Longer Working On ESXi 5.1

by // 4 Comments

I was recently made aware of an issue with my ghettoVCB script that after upgrading to ESXi 5.1, the ability to clone (or in this case backup) using the 2gbsparse disk format with vmkfstools was no longer working. The error that users were seeing was "The system cannot find the file specified." and I also confirmed this behavior by manually creating a VMDK and then trying to clone using the 2gbsparse format.

To give you some background, the 2gbsparse disk format is not a VMFS virtual disk format, it is part of the hosted desktop product (VMware Fusion, Workstation, Server & Player) disk format. This disk format was created to prevent cross-platform file system compatibility issue as pointed out in this VMware KB article. This issue does not exists on VMFS and hence this extra disk format is not necessary.

After some investigation, I found to use the 2gbsparse format in vmkfstools, you will need to load a specific VMkernel module called "multiextent". 2gbsparse was never officially supported on ESXi, you can not run a virtual machine with 2gbsparse disk format on ESXi and that is why a conversion maybe required when moving from a hosted product to ESXi. So by disabling unnecessary VMkernel modules that were not used made sense to help reduce amount of resources needed to load up. This is especially important with stateless deployments, where you want your ESXi host to load up as fast as possible.

Once you have enabled this VMkernel module, the 2gbsparse format will function again with vmkfstools. I also found that this was mentioned in the vSphere 5.1 release notes (yes, you should read the release notes)

To load the multiextent VMkernel module, run the following ESXCLI command:

esxcli system module load -m multiextent

To check whether the multiextent VMkernel module has loaded, run the following ESXCLI command:

esxcli system module list | grep multiextent

If you wish to persist this configuration after a system reboot, I found that you need to add the following command in a start-up script /etc/rc.local.d/local.sh as just setting the "enabled" flag is not sufficient for this particular VMkernel module.

localcli system module load -m multiextent

Note: We are using localcli because hostd may not be completely ready and you can either add a sleep/timer or just use localcli.

A Pretty Cool Method of Upgrading to ESXi 5.1

by // 44 Comments

I recently came across an interesting article by Andreas Peetz which shows you how to patch an ESXi host using an image profile that is directly available on VMware's online depot within the ESXi shell. I knew that VMware had online depots for use with VUM and Auto Deploy but I was not aware of this particular method, especially directly from the host.

Disclaimer: This method assumes you can install the default ESXi Image Profile with no additional drivers or packages, else you may have connectivity issue after the upgrade. If you still need to customize the ESXi Image Profile before installation, you will still need to use something like Image Builder and then upload that to your online depot.

Note: There are many ways that you can patch/upgrade your ESXi hosts, here is another article that provides more details for command-line only methods.

Before you get started, you will need to make sure that your ESXi host has the httpClient firewall rule enabled, else you will not be able to connect to VMware's online depot. To enable this, run the following ESXCLI command:

esxcli network firewall ruleset set -e true -r httpClient

Also make sure that your ESXi host can reach the following URL (you can specify a proxy if needed):

https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

To view the available ESXi Image Profiles, run the following ESXCLI command (use the --proxy if you need to specify a proxy to reach VMware's online depot):

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you are able to successfully connect to the online depot, you see a list of all the ESXi Image Profiles that are available to you. You will see two ESXi 5.1 Image Profiles (these were recently published), one with VMware Tools and one without VMware Tools.

Note: Before you begin, make sure you do not have any running VMs and put your host into maintenance mode.

Let's go ahead and upgrade our ESXi 5.0 Update 1 host to latest ESXi 5.1. To install the new Image Profile, run the following command:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.1.0-799733-standard

This can take a few minutes to complete depending on how fast you can pull down the Image Profile. Once it is done, you will see all the new VIBs that have been updated and you will be asked to reboot for the changes to go into effect and then you will be running ESXi 5.1! Pretty cool IMO!