There are a ton of new features with the latest release of vSphere 5.1, but the one "unsupported" feature I always test first is "Nested Virtualization" (aka Nested ESXi) and with the latest release, it seems to have gotten even better. You will still need to have the same physical CPU prerequisites as you did in the past to run "Nested Virtualization" as well as nesting 64-bit VMs.
- Intel VT-x or AMD-V is required for running "Nested Virtualization" which supports nested 32-bit VMs
- Intel EPT or AMD RVI is required for running nested 64-bit VMs.
A quick way to verify whether your CPU truly supports both Intel-VT+EPT or AMD-V+RVI, you can paste the following into a browser:
https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability
You will need to login with your root credentials and then look for the "nestedHVSupported" property and if it states false, it means you maybe able to install nested ESXi or other hypervisor, but you will not be able to run nested 64-bit VMs, only 32-bit VMs, assuming you have either Intel-VT or AMD-V support on your CPUs.
For more details, take a look at this article for troubleshooting: Having Difficulties Enabling Nested ESXi in vSphere 5.1?
Disclaimer: This is not officially supported by VMware, please use at your own risk.
There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.
- The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
- vhv.allow = "true" is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = "true" that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware's hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter.
- You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = "true" parameter to the VM's .VMX configuration file.
Note: You can run a nested ESXi 5.1 VM on top of a physical ESXi 5.0 host, just follow the instructions here.
Enabling VHV (Virtual Hardware-Assisted Virtualization)
Step 1 - Create a new Virtual Hardware 9 Virtual Machine using the new vSphere Web Client that's available with vCenter Server 5.1.
Step 2 - Select "Linux" as the guestOS Family and "Other Linux (64-bit)" as the guestOS Version.
Step 3 - During the customize hardware wizard, expand the "CPU" section and select "Hardware Virtualization" box to enable VHV.
Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x + EPT or AMD-V + RVI which is required to run VHV OR that you are not using Virtual Hardware 9. If your CPU only supports Intel-VT or AMD-V, then you can still install nested ESXi, but you will only be able to run nested 32-bit VMs and not nested 64-bit VMs.
Step 4 - It is still recommended that you change the guestOS Version to VMware ESXi 5.x after you have created the VM shell, as there are some special settings that are applied automatically. Unfortunately with the new vSphere Web Client, you will not be able to modify the guestOS after creation, so you will need to use the C# Client OR manually go into the .VMX and update guestOS = "vmkernel5"
Now you are ready to install nested ESXi VMs as well as run nested 64-bit VMs within.
If you have followed my previous article about How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 you may recall a diagram about the levels of "Inception" that can be performed with nested ESXi. That is, the number of times you could nest ESXi and still have it be in a "functional" state. With vSphere 5.0, the limit that I was able to push was 2 levels of nested ESXi. With latest release of vSphere 5.1, I have been able to push that limit now to an extraordinary 3 levels of inception!
You might ask why would someone want to do this ... well I don't have a good answer other than ... because I can? 😉 VHV is one of the coolest "unsupported" feature in my books and I'm glad it is working beyond what it was designed for.
For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.
Nesting "Other" Hypervisors
For those of you who feel inclined to run other hypervisors such as Hyper-V, you can do so with latest release of ESXi 5.1. The process if very straight forward just like running nested ESXi host.
Step 1 - Create a Virtual Hardware 9 VM and select the appropriate guestOS. In this example, I selected Windows Server 2012 (64-bit) as the guestOS version.
Step 2 - Enable VHV under the CPU section if you wish to create and run nested 64-bit VMs under Hyper-V
Step 3 - You will need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE
Great post William as always
One thing to note, don't do the vhv.enable on the physical host if you plan on running VSE on it. It will complain that it doesn't support Virtual Hardware-Assisted Virtualization. Just enable it per VM.
It's actually best to enable VHV on a per VM basis, that way you won't hit this problem due to the virtual hardware version differences in the environment.
Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x or AMD-V which is required to run VHV OR that you are not using Virtual Hardware 9.
I cannot understand above part.
I have enabled Intel-VT in BIOS on PowerEdge 1950 server. I added vhv.enable = "TRUE" as well - yet, I cannot select "Hardware Virtualization" box to enable VHV when creating virtual machine using web client. Where am, I wrong. Appreciate your input.
Hi Ram. Same situation here. We have two VT-supported processors, one 5130 and one E5335. The latter works, the first doesn't. What processor do you have?
I've updated the article, take a look at the very top and search for "nestedHVSupported"
Hey Ram,
Same thing here Dell Precision T5400 with dual X5460, "hardware Virtualization" is grayed out, and when I start a VM I get a VM Message stating "Virtualized Intel VT-x/EPT is not supported on this platform".
Output of esxcfg-info shows supported and enabled
~ # esxcfg-info | grep "HV Support"
|----HV Support............................................3
|----World Command Line.................................grep HV Support
Worked fine with 4.1 and 5 -
I've updated the article, take a look at the very top and search for "nestedHVSupported"
Hi,
Same thing here. Hp dl380g5 e5345 and dl380g6 l5520 both show
~ # esxcfg-info | grep "HV Support"
|----HV Support............................................3
|----World Command Line.................................grep HV Support
But no luck getting it enabled.
I've updated the article, take a look at the very top and search for "nestedHVSupported"
Same thing for me too!
HP DL380 G5 with 1 of Dual-Core 5160 Intel CPU
In Bios, "Intel Virtualisation" is enabled.
But "Hardware Virtualisation" on vSphere Web Client 5.1 is greyed out for any newly created HW V.9 Virtual Machine.
I've updated the article, take a look at the very top and search for "nestedHVSupported"
Having the same problem on Dell 1950 -
In Bios, "Intel Virtualisation" is enabled.
But "Hardware Virtualisation" on vSphere Web Client 5.1 is greyed out for any newly created HW V.9 Virtual Machine.
Used to work ok on vSphere 5.0.
Yes thank you
You just need to create the VM in the webclient and then
The option is not grayed out.
If you create the vm in the viclient and then goto the webclient you can not select it.
Still strange that there is a difference between the web and
Windows client. But on the other hand probably going away from Ms windows dependency
Regards Toine
Are you sure about last entry, just tried creating VM via Web Client, still greyed out. on Dell 1950 Host.
There some inconsistency here. Works for some and not others.
I put together some screenshots.
Yes, ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs!
http://tinkertry.com/esxi-5-1-running-hyper-v-server-2012
I am also having same problem here. I was able to create virtual esxi5.1 - Created vm server 2012 through web client - yet Hardware Virtualization box is grayed out.
I even checked intel site for xeon e5410 processor with regard to EPT - It is available but, I don't see an option to enable it in BIOS.
Even checked esxcfg-info command to check the status of HV Support
The output is HV Support...3
World Command Line Grep HV Support
I even ran coreinfo -v to check the status of EPT. It shows a DASH which means the processor does not support EPT. If you have * then the processor support EPT.
Coreinfo -v output
Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Intel64 Family 6 Model 23 Stepping 6, GenuineIntel
HYPERVISOR * Hypervisor is present
VMX - Supports Intel hardware-assisted virtualization
EPT - Supports Intel extended page tables (SLAT)
Same issues here:
Dell PowerEdge 2950III with dual Xeon E5430's
Dell Precision T7400 with dual Xeon E5420's
both have latest BIOS revisions, all virtualisation switches enabled in BIOS, and Intel ARK clearly states that these CPU's support VT-X with EPT.
Not a happy bunny at the moment 🙁
Ram/Seymour,
Can you guys provide a bit more details on how you're configuring your nested ESXi VMs? Which method are you using? Please provide the step by step that you've walked through
Completed steps on physical pe 1950 esxi 5.1
1. Enabled vhv on physical esxi 5.1
2. Enabled promiscuous mode to accept on physical esxi 5.1
3. Created Virtual machine using web client 5.1 - Guest OS - Other Linux x64.
4. At Customize hardware - Expanded CPU to select Hardware Virtualization - Greyed out.
What is going on, I can't understand.
Can you open a browser and paste the following URL: https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability
Login using the root credentials and then look for "nestedHVSupported" and see what it says. This basically lists the capabilities of the host and this should tell you whether vSphere believes you can run VHV.
Also is your hardware on the HCL for ESXi 5.1?
Same problem here. No option for Hardware Virtulisation. I'm running an E3-1240 which should have the extensions required.
This comment has been removed by the author.
It looks like my CPU does support nesting, so why can't I install a Hyper-V role on Windows Server 2012? I followed the instructions as above. No option for VHV when creating the VM via the web GUI.
https://dl.dropbox.com/u/6104595/Screen%20Shot%202012-09-28%20at%2023.55.26.png
Did you create a VM that has ESXi 5.1 compatibility (e.g. virtual hardware 9)? Please take a closer look at the steps on where to enable VHV.
nestedHVSupported boolean false
Now, I understand why, I cannot run nested vm's.
By the way PowerEdge 1950 III is listed in HCL - VMware esxi 5.1
@Ram,
Let me ping a few folks to see if there's anything else you could do. As you know this is all "not supported" and YMMV 😉
@Ram,
It looks like the issue is that you don't have EPT support. I know you pointed to ark.intel.com but I was told that even their site is not always 100% accurate and these actual bits are detected and that is why you see nestedHVSupported = false.
What this means is you won't be able to check the "Hardware Virtualization" box and you WILL be able to install nested ESXi or other hypervisors, just that you won't be able to run 64-bit guestOSes, only 32-bit just like previous releases.
@William - Thanks for all your effort.
Yes coreinfo did point out that my pe1950 does not support EPT. Wrong info on intel site.
My main purpose to run vesxi is for learning purpose only. I will play with my lab using 2 other physical pe 1950.
@Ram
If you CPU has Intel-VT support, you can still install nested ESXi. Please take a look at the note under Step 3
@ William - I was able to install vESXi successfully. As pointed out b4 cannot run 64bit OS. That's OK!
By the way, I am getting this error on DP - "the most common cause for this error is that the time on the vdp appliance and your sso server are not in sync" Any thoughts......
Ignore DP issue, I was able to fix. Once again thanks for updated nested esxi 5.1 instructions.
@Ram - I am curious on what your resolution was for the VDP "most common cause for this error is that the times on the VDP appliance..." error message?
Failed to install nested Red Hat Enterprise Virtualization Hypervisor v6.3 on ESXi v5.1:
I enabled promiscuous mode on the portgroup and used Web Client to create a new VM HW v9 as Linux/Other Linux (64-bit) and successfully enabled Hardware Virtualization.
When I boot VM from ISO and select "Install or Upgrade" option - it just stays frozen there.
Any thoughts?
Thank you!
Found the workaround: hit to get further to boot: prompt, enter "install" and the installation will continue.
Ended up with fully installed and configured RHEV Hypervisor and RHEV Manager VMs on ESXi 5.1, but inner-guest VMs (Win7 & WinXP) will not boot at all (staling with "Wait For Launch" message)...
Any hint would be appreciated.
prezha
Solved (by restart and some manual work).
Bottom line: latest RHEV v6.3 *can* run virtualized on latest ESXi v5.1.
Enjoy!
Interesting!
Can you share some tricks please?
Thanks in advanced.
Hello all,
I have to many problems put RHEV v6.3 run virtualized on latest ESXi v5.1.
Can some one off you guys help me please?
When I boot VM from ISO and select "Install or Upgrade" option (or any other option) it just stays frozen.
@prezha can you tel me better wht you do on your workaround?
tks in advance.
Regards.
prezha was missing a VERY important word. You must press the "Escape" key at the install screen. That will present you with a "boot:" prompt at which point you must enter "install". Wizard continues from then on as normal.
How do I enable hardware assisted virtualization without the web client, I only have the c# client (free version)? I have a Xeon E3-1230 Sandy bridge CPU.
The why is easy,
Virtual Labs Today
Using HPC FT for entire host workloads tomorrow
(FT Multi CPU + Infiniband + RDMA + Fusion IO Mezanine Card + VMware Disk IO Caching and Replication).
*protected email*
Hopefully this helps someone at some point:
Was able to turn on "Expose hardware assisted virtualisation", but when I powered on a VM, vCenter reported that "The target host does not support the virtual machine's current hardware requirements"
This was odd, as my Intel x5650 is capable of EPT, and nestedHVSupported was listed as "true"...
Turns out the EVC mode on my cluster was too low, hiding the functionality from the VMs. Increasing the EVC level solved the problem.
hi, i have installed server 2k8 r2 Hyper v on ESXi 5.1, i am able to create vm in server manager but when i tried to power on it i am getting the following error message "the virtual machine could not be started because the hypervisor is not running", checked the HyperV services are running. thanks in advance.
Cheers
Venkat Kurela
Thanks William. Excellent as always.
What's the best choice for the SCSI Controller and NIC adapter type?
Thanks from Singapore
e1
I use e1000 & LSI Logic
Have you been able to create a filer at level 1 and present shared storage to subsequent levels? I cannot get this to work. I'd like to run iSCSI over vmxnet3 from level 2 to level 3 so that I don't lose as much disk I/O, ideally.
Hi William,
Great article, as always. What base hardware specs would you suggest for a nested 5.1 host to be used for training purposes?
Thanks!
As a note, when I created teaching pods for 5.0, this is what I used:
CPU: 1 socket, 1 core
RAM: 4GB
Net Card: E1000
SCSI: LSI Logic SAS
Disk: 40GB Thin, Independent, Persistant
Has this changed again? The advanced option setting "vhv.enable='TRUE'" for the VM does not persist from the vSphere Client. However, after tweaking the option in the Web Client, the exposed configuration parameter was "featMask.vm.hv.capable='Min:1'" and the nested feature works... (vSphere 5.1)
Looks to hide the "vhv.enable" from vSphere Client while adding and exposing the "featMask.vm.hv.capable" setting. Hmmm...
(prior to power-on)
# diff vm03\ \(ESXi\ 4.0\).vmx /tmp/vm03\ \(ESXi\ 4.0\).vmx
--- vm03 (ESXi 4.0).vmx
+++ /tmp/vm03 (ESXi 4.0).vmx
@@ -120,5 +120,4 @@
config.readOnly = "FALSE"
-softPowerOff = "FALSE"
-featMask.vm.hv.capable = "Min:1"
-vhv.enable = "TRUE"
+softPowerOff = "TRUE"
floppy0.present = "FALSE"
Appears that when adding the setting "vhv.enable" from the vSphere Client, it will not show-up in subsequent listings and will not be persistent.
However, if you remove it from inventory, add it via shell and re-inventory it, the setting sticks (but is still not listed in advanced settings) - this is verifiable in the Web Client.
Have you worked-out the process for migrating a running vESXi VM from a 5.0 to a 5.1 host?
I had the same issue - vhv.enable is not persisted when using vCenter Client with free ESXi 5.5
I used this approach to manually edit .vmx configuration file:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017022
Hi William,
In my nested ESXi, everything is OK. I have a problem after build 50 ESXi VMs. 50 ESXi VMs can ping outside and anywhere but can't ping each other. I don't know the reason why, I use E1000 & E1000E Nic for testing but the result is the same. Can you help me?
Thanks.
Randy
Nice post. Keep in way of good job! 🙂
Great informative post friend.
I had trouble getting Nested HVH to work when nesting OracleVM 3.x cluster within ESXi 5.1 until I read this great post.
Thanks
--Tibbz
Wonderful post.
Thank you very much!
William, thank you for sharing this information. I'm a newbie on VMware and recently started playing around with ESXi 5.5 on two self built, i7 980X 6-core & i7 950 4-core based servers. I need virtualization for my network labs (exam prep). On the fly I try to pick up some knowledge of Windows Server, Linux and VMware.
Looking for a way to share my hardware with others I stumbled upon your article about nested 64 bit on vSphere v5. After following all the leads and reading some more documentation I was able to install ESXi 5.5 as guest on my ESXi host. But I got the "HARDWARE_VIRTUALIZATION WARNING: Hardware Virtualization is not a feature of the CPU ..." message.
When I looked further I ran into this post, for nested 64 bit on vSphere v5.1. The comments on this post gave me the last push. Here's what I do:
- I create my guest based on VMware Hardware Version 9 in accordance with your articles
- I do some tweaking in ESXi shell
- I finalize my configuration with vSphere Client 5.5 (that comes with ESXi 5.5); this INDEED removes vhv.enable="TRUE" from my guest's VMX file
- when I'm done with configuring I add vhv.enable="TRUE" through ESXi shell; as long as I refrain from further reconfiguration through vSphere Client it will stay there (I guess)
- After bootup of my ESXi 5.5 guest, "https:///mob/?moid=ha-host&doPath=capability" shows me "nestedHVSupported boolean true" so I think I'm OK with this procedure
But still: Would you know if there is a more elegant (permanent) solution to the disappearing vhv.enable="TRUE" line with ESXi 5.5 and vSphere 5.5?
Thanx once again,
Steijn van Essen
Network Professional
The Netherlands
From i8088 to i7-980X in 25 years and still waiting ...
Sorry, need to correct two omissions:
- 2nd paragraph: ... I was able to install ESXi 5.5 as guest on my ESXi 5.5 host ...
- 5th paragraph: ... disappearing vhv.enable=”TRUE” line with ESXi 5.5 and vSphere Client 5.5?
Final note: I run ESXi 5.5 host and vSphere Client 5.5 with an evaluation license (restricted in functionality, but unlimited in time)
Steijn van Essen
From i8088 to i7-980X in 25 years and still waiting …
When following the instructions, I am able to get the nested hardware support - thank you for the efforts here.
Now trying to script the same steps from a Packer build server and upload the image to ESX 5.1, the vhv.enable flag is being stripped from my .vmx! Everything looks good before the upload with ovftool 4.0, but when I check the contents on the ESX server the settings are gone.
Any thoughts in how I might get the flag to pass through ovftool?
I'm trying to start a VM under Windows 10 Hyper-V, but the VM always fails to start. I'm trying to run the Windows Phone emulator inside the VM.
I have an ESXi 5.5 Update 3 and I created a Windows 10 VM, following the 3 steps above for configuring an Hyper-V guest on ESXi. Windows 10 is at the latest build/update level. Then I enabled the Hyper-V feature on Windows 10 and created an Hyper-V VM. The Hyper-V VM is not able to start.
If I replace, in the same VM, Windows 10 with Windows 8.1 and create an Hyper-V VM or start the Windows Phone Emulator, everything work fine.
So it appears there is a problem with the Hyper-V that ships with Window 10. Has anyone been able to run an Hyper-V guest in windows 10?
I am having the same issue. Did you get this resolved?
I'm using Esxi 6.x and I want to enable KVM in one of our guest, web client is not working hence only option is vSphere Client, using earlier vmx files I set parameter for featMask.vm.hv.capable = "Min:1", however it's still not working.
I there any thing else I need to setup.
Hello everyone. I have been running nested Hyper-V inside of ESXi6 for sometime now as well as I also have a ESXi5.5 nested inside of my same ESXi6 environment. The question that I have is that I am able to install the Hyper-V role no problem and create VMs etc. The issue is that I have not been successful in trying to install the Windows Server RDS Virtualization Host role on a nested Windows Server 2016 Technical Preview 4 release VM in my ESXi6 environment. It keeps giving me an error that Hardware Virtualization is not present etc. Now keep in mind I have made all of the changes needed and this very server has Hyper-V installed and running VMs no problem at all. It is just the RDS VDI role that can not be installed. I have not been able to find any information pertaining to this being possible. If someone knows how to get this to work it would be great. Thanks
THANK YOU!!! Without vSphere web client I really struggled with this but knowing how to edit the vmx by adding vhv.enable = "true" Was a life saver!
Hi, its nice article... I have my nested esxi. Now, if I want my nested esxi to have ip from dhcp server rather than 192.168.1.190, how can I do that? Thanks in advance
William, just passing by to say how much your articles have helped me thru the years. Thanks a lot, mate. All the best to you and keep up the outstanding job !
Hi Will, I need to setup a nested lab environment on esxi 6.7 , please send me required link.