WilliamLam.com

w00t! VMware Tools for Nested ESXi!

by // 42 Comments

I have been working with Nested ESXi since it original inception and this technology has greatly benefited me and the entire VMware community, especially when it comes to learning about VMware software and being able to easily prototype something before installing it on actual hardware. However, one thing that I felt that has been missing for awhile now is the ability to run an instance of VMware Tools within a Nested ESXi VM. I have personally been asking for this feature for a couple of years and I know many in the VMware community have expressed interests as well.

I am super excited to announce that VMware has just released a new Fling that provides you with a VIB that you can install VMware Tools inside a Nested ESXi host. I originally showed a demo of this at VMworld Barcelona in my vBrownBag Tech Talk and as I mentioned we would be releasing this as a VMware Fling very soon. So here it is!

UPDATE (08/20/15) - An updated version of VMware Tools for Nested ESXi was just published, make sure to download latest version and you can find more details here.

Requirements:

  • Nested ESXi running 5.0, 5.1 or 5.5 

Installation:

To install the VIB, you simply just need to download it and upload the VIB it to your Nested ESXi datastore and then run the following commands:

esxcli system maintenanceMode set -e true
esxcli software vib install -v /vmfs/volumes/[VMFS-VOLUME-NAME]/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f
esxcli system shutdown reboot -r "Installed VMware Tools"

You can also install the VIB directly from VMware.com if you have direct or proxy internet connectivity from your ESXi host by running the following commands:

esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi_tools_for_guests/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f

Once the VIB has been successfully installed, you will need to reboot the host for the changes to take effect. To verify, you can now login to either your vSphere Web/C# Client and you should now see the status for VMware Tools for your Nested ESXi host showing green and the IP Address of the Nested ESXi host should be displayed.

So why would you want to do this? Well, there’s a couple of reasons actually. The first one is pretty basic, which is when I need to reboot or shutdown a Nested ESXi VM, instead of having to jump into the VM console or SSH into ESXi host, I could just right click in the vSphere Web/C# Client and just say shutdown or reboot. I also tend to do all sorts of craziness in my lab (I’m sure this is an understatement for folks that know me) and may often break networking connectivity to my Nested ESXi VM. In vSphere 5.0, we introduced the Guest Operations API (formally known as VIX API) which is now part of the vSphere API. This API is actually quite handy as it allows you to perform guest operations within the VM without needing network connectivity as it relies on the fact that VMware Tools is running (pretty cool stuff!).

Here is a screenshot demonstrating the executing of vmkfstools through the Guest Operations API to one of my Nested ESXi VM:

A couple of things to note:

  • If you install VMware Tools on Nested ESXi VM, you will NOT be able to just right click in the UI and say install/upgrade
  • If you wish to integrate this into you ESXi image, you can take a look at a community tool  called ESXi-Customizer created by Andreas Peetz which I have used in the past and works great. Image Builder does not support raw VIBs, only zip files which may need to contain additional metadata information. If you want to create an offline bundle instead to then use Image Builder to create your custom ISO, Andreas has a new tool you can take a look at here.

Finally, if you have any feedback (likes/dis-likes), thanks, comments please head over to the VMware's Fling page for VMware Tools for Nested ESXi and leave a comment. I am sure the Jim Mattson the engineer who built this Fling would greatly appreciate any feedback you may have.

ESXi 5.5 introduces a new Native Device Driver Architecture Part 2

by // 4 Comments

Following up from Part 1 where I provided an overview of the new Native Device Driver architecture introduced in ESXi 5.5, we will now take a deeper look at how this new device driver model works in ESXi. A new concept of driver priority loading is introduced with the Native Device Driver model and the diagram below provides the current ordering of how device drivers are loaded.

As you can see OEM drivers will have the highest priority and by default Native Drivers will be loaded before "legacy" vmklinux drivers. On a clean installation of ESXi 5.5 you should see at least two of these directories: /etc/vmware/default.map.d/ and /etc/vmware/driver.map.d/ which contains driver map files pertaining to Native Device and "legacy" vmklinux drivers.

Here is a screenshot of the map files for both of these directories on an ESXi host:

The following inbox Native Drivers are included in default installation of ESXi 5.5:

Device Device Driver Name
Emulex 10GBe NIC elxnet
Emulex FC lpfc
LSI Megaraid lsi_mr3
LSI mptsas lsi_msgpt3
Micron SSD mtip32xx_native
QLogic FC qlnativefc
SAS/SATA rste
vmxnet3 & graphics vmkernel

As I mentioned earlier, Native Drivers by default will always load before vmklinux drivers, however if you need to perform some troubleshooting, one option is to disable the specific driver in question by using ESXCLI which is applicable to both Native Drivers as well as vmklinux drivers.

To do so, run the following ESXCLI command:

esxcli system module set --enabled=false --module=[DRIVER-NAME]

How to run Nested ESXi on top of a VSAN datastore?

by // 35 Comments

Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course piqued my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.