WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple

How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5

07.12.2011 by William Lam // 66 Comments

With the release of vSphere 5, one of the most sought out feature from VMware is the ability to run nested 64bit and Hyper-V guest virtual machines in a virtual ESXi instance. Previous to this, only 32bit virtual machines were supported as the VT-x/AMD-V Hardware Virtualization CPU instructions could not be virtualized and presented to the virtual ESX(i) guest. This feature is quite useful for home and lab setups in testing new features or studying for VMware certifications and running multiple vESX(i) instances.

You will still be required to have a 64bit capable system and CPU and you will need to be running ESXi 5.0, this will not work for ESX(i) 4.x or older.

 
The above diagram depicts the various levels of inception where pESXi is your physical ESXi 5.0 hosts. We then create a vESXi 5.0 host which will contain the necessary Hardware Virtualization CPU instructions to support a 64bit nested guest OS which I've created as another ESXi host called vvESXi.
Note: You will not be able to run a 4th level nested 64bit VM (I have tried by further passing the HV instructions in the nested guest) and it will just boot up and spin your CPUs for hours.
This feature by default is disabled in ESXi 5.0, to enable this virtualized HV (Hardware Virtualization) you will need to add the following string vhv.allow = "TRUE"  to /etc/vmware/config of your Physical ESXi 5.0 host
Once the configuration change has been made, the feature goes into effect right away. A reboot of the system is not necessary. To verify, you should now be able to power on a 64bit guest OS and see that the HV instructions bits are being passed into the guestOS which will then allow you to run a nested 64bit guestOS. You can also verify by looking in the vmware.log file of the virtual machine and grep for the string "monitorControl.vhv" and if you see the following message, then Virtualized HV is not enabled.
In the past to run a virtual ESX(i) instance, a few advanced .vmx configuration entries were needed as documented here. With ESXi 5.0, if you are using virtual hardware version 8, then you do not need to make any additional changes. If you are using hardware version 4 or 7, then you will need to add a few changes to the VM's configuration file.
Creating vESXi 5.0 Instance using Hardware Version 8:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client
2. Once the VM has been created, edit the settings of the VM and change over to the "Options" and now have the ability to select a new guestOS type: VMware ESXi 5.x or VMware ESXi 4.x under the "Other" section.

Note: I'm not sure why these two additional guestOS type is not available from the default creation menu, but are available after the initial VM shell is created.

3. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance as denoted from the picture below

Creating vESXi 5.0 Instance using Hardware Version 4/7:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client

2. Now you will need to add the following advanced .vmx parameter:  monitor.virtual_exec = "hardware" which can be done through the vSphere Client and/or editing the .vmx parameter manually.

3. Next you will need to add some cpuid bits, depending if you are running an Intel or AMD CPU, the respective entries are required:

Intel Hosts:

cpuid.1.ecx = "----:----:----:----:----:----:--h-:----"

AMD Hosts:

cpuid.80000001.ecx.amd = "----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"

4. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance

By using a VM that is hardware version 8, you can easily automate the creation of vESXi 5.0 instance by changing the guestOS string in the .vmx parameter to "vmkernel" and the above configurations other than "vhv" string needed for either an Intel or AMD system are automatically configured.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.  

Creating a vHyper-V  Instance on physical ESXi 5.0:
1. To create a virtual Hyper-V instance, start off by creating a Windows 2008 Server R2 64bit VM using the vSphere Client

2. If you are using Hardware Version 7, you will need to follow the instructions in "Creating vESXi 5.0 Instance using Hardware Version 4/7" to add the additional parameters to the VM. If you are using Hardware Version 8, you just need to change the guestOS type to VMware ESXi 5.0

3. You need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

4. You are now ready to install Hyper-V in a virtual machine and you can also spin up nested 64bit guestOSes in this virtual Hyper-V instance.

As you can see, now you can even run Hyper-Crap, err I mean Hyper-V as a virtualized guest under ESXi 5.0. I did not get a chance to try out Xen, but I'm sure with the ability to virtualize the Hardware Virtualization instructions, you should be able to run other types of hypervisors for testing purposes.

This is a really awesome feature but note that this is not officially supported by VMware, use at your own risk.

Categories // ESXi, Home Lab, Nested Virtualization, Not Supported Tags // ESXi 5.0, hyper-v, nested, vesxi, vhv, vSphere 5.0

How to Disable a vmnic in ESX(i)

06.24.2011 by William Lam // 8 Comments

There was another interesting thread today on the VMTN community forums about disabling an unused vmnic from an ESXi host due to false alarms being generated from HP SIM. Due to the specific hardware version, disabling the vmnic through the BIOS was not an option and there were no alternatives from the discussion.

I decided to dig a bit to see what I could find and stumbled upon a neat little utility called vmkchdev (VMkernel Change Device?) and from what I can tell provides a method passing a particular device to be controlled by either the VMkernel or as a passthrough device to a virtual machine (think VMDirect Path).

Disclaimer: Please note, this is using an undocumented utility. You should test this out in a development/lab environment before using and you may want to also contact VMware support to get their blessings***

~ # vmkchdev
Usage:
-s (scan device)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

What I found while testing the utility is by passing it over as a passthrough device, the vmnic is actually unrepresented to the VMkernel and does not show up under network adapters or even the unused/unlinked adapter list in the vSwitch configurations. It seems that it is just masks the device away from the VMkernel as you can still see the active configuration in esx.conf and you can see the device listed using vmkchdev.

Here is an example of an unused physical nic vmnic1 that we would like to disable and unpresent to an ESXi host.

Here is the output from esxcfg-nics:

First we need to identify the vmnic's PCI slot, we do by running the "-l" or list operation and searching for the particular vmnice device.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 vmkernel vmnic1

Next we will pass the device from the VMkernel to passthrough/VM using the "-p" flag and specifying the PCI slot, which in this case it is 000:002:01.0. We will also need to refresh the network section so the changes are reflected in the vSphere Client by using vim-cmd.

~ # vmkchdev -p 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

If we list our vmnic again using vmkchdev, you will notice the device is now owned by passthru versus the VMkernel.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 passthru vmnic1

Now if we check the output of esxcfg-nics and the vSphere Client, you will notice that vmnic1 is no where to be found

If you would like to enable or re-present the disabled vmnic, you just need to pass the device back over to VMkernel by using the "-v" flag.

~ # vmkchdev -v 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

Your vmnic should now re-appear on all your screens and any existing NIC teams that may have exists is automatically restored. This trick actually works on both a used and unused vmnic

If you are trying to do this on ESX, vmkchdev actually has an additional option called "console" for the Service Console.

[root@esx41-1 ~]# vmkchdev
Usage:
-s (scan device)
-c (give device to console)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

I found that you need to pass the vmnic from VMkernel to Console, passing it to passthru/VM will not work and an error is thrown if you do. Again, you can easily re-enable by passing it back to the VMkernel

vmkchdev -c 000:002:01.0

If you would like to automatically persist this change across reboots, specifically for ESXi as changes are not saved. You will need to add the following lines to /etc/rc.local which will execute the disabling of the vmnic's after bootup.

/sbin/vmkchdev -p 000:002:01.0
vim-cmd hostsvc/net/refresh

You will also need to run /sbin/auto-backup.sh to ensure the changes to /etc/rc.local are saved and reloaded upon the next reboot. For ESX, you can place it in /etc/rc.local without having to do anything extra as the changes persists across reboots for classic ESX

Now you can play hide and seek with your vmnic's without resorting to a system reboot or touching the BIOS. Though ideally, if you do have unused devices, you should definitely disable them in the BIOS if you have the option. On a side note, this might be a fun trick to play on one of your co-workers by hiding all vmnics 😉

Categories // ESXi, Not Supported Tags // ESX 4.0, ESXi 4.1, vmkchdev, vmkdevmgr, vSphere 4.1

Semi-Interactive automated ESXi installation

05.09.2011 by William Lam // 18 Comments

There was a recent thread in the VMTN community forums where a user wanted to provide user input prior to an automated ESXi installation. This may sound odd at first, especially when we are talking about an unattended installation, the last thing you want is any type of user interaction. The reason for this requirement was to maintain a generic kickstart configuration file and based on the site specific input (IP Address, Netmask, Gateway, Hostname and DNS Servers), the ESXi installation would be able to deploy and configure itself the same across multiple locations.

UPDATE (01/10/19) - For ESXi 6.5 or greater, please take a look at this blog post for an updated solution

UPDATE (10/28/15) - Please take a look at this blog post on how to prompt for user input during an interactive or scripted installation of ESXi.

The other reason for this requirement is that all hosts in the user's environment must be configured with a static IP Address, this is not an uncommon requirement for many production environments to not have DHCP enabled networks. I can only assume the initial network the host is being built is either a private build network or booting off of local media such as a USB or CD-ROM device.

Whether you are booting off of local media or via PXE over the network, you have the ability to specify some boot parameters which includes things like IP Address, Netmask, Gateway and DNS Servers. These "bootstrap" options are fully supported and documented in the ESXi Installation Section. You can also specify advanced VMkernel boot parameters which may not be officially supported by VMware, but you can take a look here for more details.The concept of specifying these boot parameters is nothing new and has been supported since the early days of classic ESX and other popular UNIX/Linux distros for PXE boot/installations.

When you boot the ESXi installer, you may have seen a screen similar to the following in which you have a few seconds to hit the "tab" key to edit the boot options.

Once you hit the "tab" key, you will be able to see what the default boot options are and if you are PXE booting, you will also see some IP information appended towards the end of the string. This is where you can append or update additional parameters and later read in by your kickstart script.

Here is an example of an ESXi installation being PXE booted over the network and I have added 4 supported boot parameters and 2 custom ones.

If you take a look at the screenshot, the "+++" IP information towards the end is what was given by the DHCP server but I am interested in specifying a different IP Address for the ESXi installer to boot from. I provided new entries for ip, netmask, gateway and nameserver. You can also see that I introduced two new variables called hostname and dc, these will be used to specify the hostname of the ESXi host and also the name of the datacenter which will be used later in the kickstart script to rename a datastore.

Note: If you redefine the IP information, you do not need to change the IP information found after the "+++", these will just be over-written with the new IP information.

When specifying these boot parameters, you need to make sure it is after the "vmkboot.gz" but before "--- vmkernel.gz". To help make this clear, I have colorized the section that was appended to the default options.

vmkboot.gz dc=ghettoDC hostname=vesxi41-2.primp-industries.com ip=172.30.0.200 netmask=255.255.255.0 gateway=172.30.0.1 nameserver=172.30.0.100 ks=http://172.30.0.108/esxi41u1/ks.cfg --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

Note: You can create any custom variables, the key in using the custom variables will be parsing from the boot command line using vsish which will be defined in your kickstart script. This will work also work for local media, but you MUST use a kickstart to perform the actual installation or inject the kickstart script into a custom ISO.

After you have provided all the input, you will then boot the installer and the the following kickstart configuration file is used to parse the boot options using vsish.

accepteula
autopart --firstdisk --overwritevmfs
rootpw vmware
install url http://172.30.0.108/esxi41u1
reboot

%include /tmp/networkconfig

%pre --unsupported --interpreter=busybox

CMDLINE_FILE=/tmp/cmdline
ESXI_INSTALL_LOG=/var/log/esxi_install.log

# extract boot options
vsish -e get /system/bootCmdLine > ${CMDLINE_FILE}

# extract and set variables
HOSTNAME=$(cat ${CMDLINE_FILE} | grep hostname | sed -e 's/.*hostname=\([^ ]*\).*/\1/')
IPADDR=$(cat ${CMDLINE_FILE} | grep ip | sed -e 's/.*ip=\([^ ]*\).*/\1/')
NETMASK=$(cat ${CMDLINE_FILE} | grep netmask | sed -e 's/.*netmask=\([^ ]*\).*/\1/')
GATEWAY=$(cat ${CMDLINE_FILE} | grep gateway | sed -e 's/.*gateway=\([^ ]*\).*/\1/')
NAMESERVER=$(cat ${CMDLINE_FILE} | grep nameserver | sed -e 's/.*nameserver=\([^ ]*\).*/\1/')
DC=$(cat ${CMDLINE_FILE} | grep dc | sed -e 's/.*dc=\([^ ]*\).*/\1/')

# create networkline based on boot options
echo "network --bootproto=static --hostname=${HOSTNAME} --ip=${IPADDR} --netmask=${NETMASK} --gateway=${GATEWAY} --nameserver=${NAMESERVER} --addvmportgroup=0" > /tmp/networkconfig

# persist custom variables in ESXi install log for %post section
echo "GHETTO_CUSTOM_VARIABLE-DC ${DC}" >> ${ESXI_INSTALL_LOG}

%firstboot --unsupported --interpreter=busybox --level=9999

#extract custom variables in ESXi install log
DC=$(grep "^GHETTO_CUSTOM_VARIABLE-DC" /var/log/esxi_install.log | awk '{print $2}')

vim-cmd hostsvc/datastore/rename datastore1 "${DC}-datastore1"

vim-cmd hostsvc/maintenance_mode_enter
vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm
vim-cmd hostsvc/enable_local_tsm
vim-cmd hostsvc/start_local_tsm

The script does the following:

  1. Reads the boot options using vsish and temporarily stores the output to /tmp/cmdline for later use
  2. Extracts all the relevant boot parameters in /tmp/cmdline and stores them in variables to be used in the script.
  3. Creates the "network" stanza for static IP assignment of the ESXi host which includes: Hostname, IP Address, Netmask, Gateway and Nameserver
  4. Writing out the custom variable "dc" into /var/log/esxi_install.log because this will be persisted through the reboot and can be later read in for any %firstboot operations.
  5. Upon the reboot, %firstboot will execute and read from /var/log/esxi_install.log to extract the "dc" variable in which it will use to rename the local datastore

Note: The above is just an example of what you can do with custom parameters. You can easily add as many as you need for site specific configurations and then use those input in your post configure your ESXi host. Also note that you do not need to make sure of ip,netmask,gateway,dns parameters to use custom variables, these are all optional.

You can add as many custom entries as you would like, but this can easily get error prone due to the amount of typing required. One thing you can do to minimize the amount of typos is to pre-specify the custom variables in your PXE/TFTP configuration file.

To do so, your PXE/TFTP configuration file would look something like this:

IMEOUT 300 #30 seconds
PROMPT 1
DEFAULT menu.c32
SAY -
SAY vga - Install esxi410u1 (vga console)
SAY -

LABEL www.virtuallyghetto.com - Semi-Interactive ESXi Install
KERNEL mboot.c32
APPEND vmkboot.gz dc= hostname= ip= netmask= gateway= nameserver= ks=http://172.30.0.108/esxi41u1/ks.cfg --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz
IPAPPEND 1

This not only reduces the amount of typing but also lets the user know what variables must be defined in order for the installation to proceed.

I totally understand where the user is coming from and I have had this requirement in the past but I think having a static DHCP entry and maintaining a simple configuration file can easily solve this problem. This not only makes the deployment process hands-off as it should be in the first place but also removing the human factor out of the equation. Can you imagine deploying 100-300 ESXi hosts in a day? I sure can not if I had to manually type out all those addresses by hand.

Categories // Automation, ESXi Tags // ESXi 4.1, kickstart, ks.cfg

  • « Previous Page
  • 1
  • …
  • 128
  • 129
  • 130
  • 131
  • Next Page »

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC) across Private, Hybrid and Public Cloud

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • ESXi 8.0 Update 2 not detecting Apple NVMe on Apple Mac Mini 2018 11/28/2023
  • Is there a vSphere REST API for ... 11/21/2023
  • Custom vSAN HCL JSON for VMware Cloud Foundation (VCF) 5.1 and vSAN ESA using Nested ESXi 11/20/2023
  • Heads Up - Performance Impact with VMware Workstation on Windows 11 with Intel Hybrid CPUs 11/19/2023
  • Quick Tip - Retry VMware Cloud Foundation (VCF) deployment with modified bringup spec 11/15/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023