With the release of vSphere 5, one of the most sought out feature from VMware is the ability to run nested 64bit and Hyper-V guest virtual machines in a virtual ESXi instance. Previous to this, only 32bit virtual machines were supported as the VT-x/AMD-V Hardware Virtualization CPU instructions could not be virtualized and presented to the virtual ESX(i) guest. This feature is quite useful for home and lab setups in testing new features or studying for VMware certifications and running multiple vESX(i) instances.
You will still be required to have a 64bit capable system and CPU and you will need to be running ESXi 5.0, this will not work for ESX(i) 4.x or older.
Note: I'm not sure why these two additional guestOS type is not available from the default creation menu, but are available after the initial VM shell is created.
3. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance as denoted from the picture below
Creating vESXi 5.0 Instance using Hardware Version 4/7:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client
2. Now you will need to add the following advanced .vmx parameter: monitor.virtual_exec = "hardware" which can be done through the vSphere Client and/or editing the .vmx parameter manually.
3. Next you will need to add some cpuid bits, depending if you are running an Intel or AMD CPU, the respective entries are required:
Intel Hosts:
cpuid.1.ecx = "----:----:----:----:----:----:--h-:----"
AMD Hosts:
cpuid.80000001.ecx.amd = "----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
4. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance
By using a VM that is hardware version 8, you can easily automate the creation of vESXi 5.0 instance by changing the guestOS string in the .vmx parameter to "vmkernel" and the above configurations other than "vhv" string needed for either an Intel or AMD system are automatically configured.
For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.
Creating a vHyper-V Instance on physical ESXi 5.0:
1. To create a virtual Hyper-V instance, start off by creating a Windows 2008 Server R2 64bit VM using the vSphere Client
2. If you are using Hardware Version 7, you will need to follow the instructions in "Creating vESXi 5.0 Instance using Hardware Version 4/7" to add the additional parameters to the VM. If you are using Hardware Version 8, you just need to change the guestOS type to VMware ESXi 5.0
3. You need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE
4. You are now ready to install Hyper-V in a virtual machine and you can also spin up nested 64bit guestOSes in this virtual Hyper-V instance.
As you can see, now you can even run Hyper-Crap, err I mean Hyper-V as a virtualized guest under ESXi 5.0. I did not get a chance to try out Xen, but I'm sure with the ability to virtualize the Hardware Virtualization instructions, you should be able to run other types of hypervisors for testing purposes.
This is a really awesome feature but note that this is not officially supported by VMware, use at your own risk.
Great Stuff!
This comment has been removed by the author.
Found a way to add the entry using VI editor.
Excellent tip William!
Hi,
Do you have a trick to do the same on VMware workstation 7.1 or workstation 8.0 beta ?
thanks 🙂
For information, in VMware Workstation 8.0 we have to choose “ESXi 5.0″ in “Guest Operating System” but the CPU must have Intel EPT or AMD RVI… mine does not 🙁 .
I am unable to run 2008 r2 nested. I have 2 vesxi5 running. I have made necessary changes on the physical host for vhv.allow. Created virtual cluster and added 2 vesxi5 host. Was able to create virtual machine and installed 2008 r2. After finishing the install, I restarted the machine. The OS boot but BLACK screen. Why, I am unable to run nested virtual machine on vesxi5 host. No clue. Still searching for an answer. Hope somebody here can help me.
i tried to Creating a vHyper-V Instance on VMware Fusion & i didn't work..you think it's possible? any helpful lead?
For others: I missed the config change to the host, and got the "Black Screen" issue above. Fixing the host config worked.
anyone tried with kvm like a nested hypervisor ? I tried but with no success 🙁
Have the same. Did you already resolved this issue?
Pretty strange that I did everything correct, but can`t install Hyper-V role inside VM, and if I set GuestOS type to ESXi5 or 4 I get Black screen, while safe mode works.
can we use same settings to create Citrix Xen Server 5.6 too on vSphere 5i?
Has anyone been able to install the VMware tools within a nested VM? That was my problem with ESX 4.x
Could you post the VMX file for the Hyper-V Instance and the physical hardware specs?
I have followed all the steps for VHyper-V on a physical ESXi 5.0 and receive a black screen when booting up the 2008 server as well. Has anyone found a solution to this yet? I need to run hyper-V for a program that requires it but don't have another physical server to install it on. Any help would be great.
Not quite sure what I'm missing. The nested vESXi5.0 starts to install but fails as it can't see a disk to install on. Any ideas?
Hi, Can anyone help me here. Im able to create Nested 64 bit Windows 2008 servers, but those manys are not reachable from outside. I understand this is a problem with Esx 5.0, as 32 bit windows 2008 when created from Esx 4.1 works fine.
Does any one faced this issue before, and any tweaks to fix this?
Keep in mind that the vSwitch might block the VM's created in Xen. You need to configure the vSwitch to promiscuous mode ACCEPT.
This comment has been removed by the author.
Has any one had issues with the networking on their nested VMs? From my nested VM I'm not able to ping the GW; from the nested ESXi server I'm able to ping it without issue.
I had a similar issue with pings inside nested VMs. What remedied this issue for me was to set "Promiscuous Mode" to "Accept" on the physical host's vswitch.
I am able to create vESXi 5.0 v8 guest hosts in my existing ESXi 5.0 cluster with no issue, and can create 32bit v8 guests on the nested host. However, I do not have the option to choose ESXi as the OS after creation, and I cannot create 64 bit nested guests.
I've set vhv.allow = "TRUE" and hardware virtualization is enabled on my pyhsical hosts.
Any suggestions as to what my issue may be?
@JC
Not sure I follow what your statments, you mention you can create vESXi 5.0 v8 host but then later say you can not select ESXi as guestOS. This is contradicting? If you running ESXi 5.0 on your physical host and follow the steps above, you can run nested ESXi hosts and power on 64bit nested VM within that nested ESXi host.
Recommend you re-read the article and perform some troubleshooting such as checking "monitorControl.vhv" in your vmware.log file
Hi William,
Thank you for your reply and my apologies for the confusion.
I have added the vhv.allow = "TRUE" line to etc/vmware/config on my physical hosts
I have confirmed that hardware virtualization is enabled on the physicsal hosts.
I have created guests running ESXi 5.0 by choosing "Red Hat Enterprise Linux 5" as the guest OS.
On these vESXi hosts I have successfully created 32-bit guests. (Hardware ESXi > vESXi > 32-bit VM)
However...
I do not have the option to choose "VMware ESXi 5.x" in the edit settings dialog of my vESXi VM.
I cannot make 64-bit nested guests off of the vESXi
Any thoughts?
@JC,
Have you checked vmware.log as I recommended in my last reply? Also are you running ESXi 5 on your physical host?
My apologies, forgot that in my last post.
The grep in vmware.log returns nothing. In fact, the only instances of "vhv" are:
vmx| DICT vhv.allow = TRUE
vmx| DICT vhv.allow = TRUE
vmx| OvhdMem OvhdUser_vhvMSRBitmap
vmx| OvhdMem OvhdUser_vhvMSRBitmap
I checked on multiple vESXi 5 machines, and got the same result.
My physical hosts are running ESXi 5, and VT is enabled at the BIOS.
I ran into that problem. Depending on the vintage of your processor you might have to force CPU and MMU virtualisation. Thus:
Edit your VM config -> Options -> CPU/MMU Virtualisation. Then check Use Intel VT-x and Intel EPT.
This worked for an older E5440 CPU running ESXi 5.0.721882.
All
I'm running Esx5 on top of Vmware fusion and it's working well , but when i try to run 64 bit VM on ESX5 it doesn;t boot and say that 64bit support is not available. Let me know what can be done, Appreciate the help.
It appears that Intel VTX alone is NOT enough. Intel EPT feature is required to allow nested virtualization.
I landed at this article searching "installing vm tools on a vESXi host. (one of the comments in this thread, the question was asked..... still searching for that answer.....)
I just got all this working right now..(kind of impressed with myself......so I thought I'd brag to the masses......):
Physical box: HPDV8T Laptop(!) (all this running on 1 5400 RPM Laptop HDD! its slow as hell, everything works though (mostly......see lower).
Host OS: Windows 7 Ult x64, with VMWorkstation 8.0.(I did try to actually install esx on this laptop but it wouldn't work...(long story of failure.....)
VMW8 VM Guests:
2x ESXi5 servers (2cpu's, 2gb of ram each (just enough for one guest)
1x Open filer 2.99.1 x64 (1 cpu, 1gb of ram) Hosting 2 100GB Iscsi luns (so I can test storage vmotion as well as vmotion).
The vc was installed on another machine (8GB is not enough to work with, and as it is, windows, VMW8, three nested guests, are pushing that 1! 5400rpm! HDD to its breaking point......
VC5, build Data center, build cluster, add hosts, create 1 2008r2 guest (just enough cpu and ram on the hosts to give it one cpu and one 1 gb of ram).
Install VM tools on nested 2008r2 guest. Reboot.
Start Windows update on the nested guest.
Initiate vmotion of nested guest while windows updating.
Response in nested guest console slows to a crawl, mouse barely functional.
Vmotion successful (eventually....slowly). Guest doesn't abend though...she hold on.
Storage Vmotion successful (eventually....SLOWLY). Guest doesn't abend though...she holds on.
Wierd things begin to happen in vsphere client connected to vc during migrations:
Hosts start alerting,or disconnecting from VC (greying out in the host list etc). (but not really abending or rebooting etc......
HA fires off alerts that guests should be moved, due to host loosing connectivity to VC.
This all due to the HDD LED on my laptop now solidly lit.
Windows starts to bug out just slightly (can't move things around on desktop. mouse button starts to stop working on and off as resources are pegged).
I'm exteremely impressed that I got this all working on one laptop running windows (minus vc). An SSD and maybe 12/16gb of ram to play with, and they'd all run (barely...to get one working nested guest.....maybe two).
I'm extrememly impressed that one 5400rpm laptop HDD can push all this.
I'm extremely impressed that ESX finally has virtualized Intel VT so that this all works now.......
Now I just need to save up enough cash to do this with 3 physical boxes, so it will actually perform beyond being able to post an article in a board saying "It works! (barely)".
Thanks for sharing! Definitely helped me out.
yeah..tried this on an IBM x3590M2 with no love. HyperV never installs, says it requires a VT enabled processor etc. Did everything, same issue with KVM..installs but can never create the vms. So I think it really is what type of processor you have in the box...will try it on another v5 system...or just reload v5 with HyperV 😉
can you do the same with server 2012 beta/RC 's hyper V role or is it not yet ??
can you do the same with server 2012 beta/RC 's hyper V role ?
Good job!!!
I do the step "Creating Hyper-V instance on ESXi 5" but the VM stucked when completing installation windows server 2008 R2.
Have you tried this with ESXi 5.1 ?
Thanks a lot.
Marcus
Please take a look at this article http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html
I just figured out the problem for when a guest gets stuck, and won't boot. You must use the VMXNET 3 Adapter for nested hosts to boot correctly. This will also require VMware Tools to be installed. I was having a problem nesting Windows 7 64bit as the first layer, then switching it over to ESXi as the Guest OS.
Does the "hypervisor.cpuid.v0 = FALSE" work if more than one core is selected for the outer guest? Or is the "cpuid.v0" referring to virtual socket (0)?
Great Tip!
Hey William,
As mentioned above I am also getting a black screen when I try to boot the VM.Can you help me here resolving this.I have a x3550 M3 on which I have installed ESXI 5.0.Have added property as mentioned in the link to host.Then I created a new VM with Win2k8R2 and installed the same operating system.100GB Disk, 2 core CPU,8 GB memory for the VM.OS was installed correctly then I shutdown the VM and make the changes as mentinoed for hyperviousr.cupid and VMWare ESXI 5.0 selection.Once I power on the VM and then there is black screen nothing happens after this
Awesome! Saved me a bunch of time
Very good, Promiscuous mode on vswitch allows VM in Hyper-V in ESXi 5.1 to ping things on the physical host ESXi 5.1 VM Network - great article !
HyperV 2008 2008R2 2012 - PROBLEM RESOLVED
Simply just add few options to following
1. Shutdown machine and upgrade it to version 9 by right click on Machine name
2. Add/change few MachineName.vmx options
featMask.vm.hv.capable = "Min:1" (add at the end)
guestOS = "winhyperv" (change from previous)
virtualHW.productCompatibility = "esx" (change from hosted)
Power On Machine and use
The times they are a changing...
This might help someone trying to do this using ESXi 5.5...
I am using ESXi 5.5 and following your instructions did not work. After doing more research, I had to add an additional string to '/etc/vmware/config' to make it work. So in addition to the string you provided:
vhv.allow = "true"
, I also had to add the following enable string:
vhv.enable = "true"
Also, after you create the vESXi VM, you can go back to edit settings, and change the OS to "VMware ESXi 5.x".
Great tips i'm very thanks for this...
Finally i got a resolve my problem.
I'm having trouble getting this working for a physical ESXI host of 6.0. after adding vhv.allow= "TRUE" to /etc/vmware/config, the vmware.log does not update.
is this still a live place, and can I get some help ?