WilliamLam.com

How to run Nested ESXi on top of a VSAN datastore?

by // 35 Comments

Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course piqued my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.

VMworld Barcelona #NotSupported Tips/Tricks for vSphere 5.5 Slides Posted

by // 3 Comments

I hope everyone got back home quickly and safely from VMworld Barcelona (unlike myself for those of you who follow me on Twitter, took a bit longer than I had hoped). For those of you who attended my #NotSupported Tips/Tricks for vSphere 5.5 at VMworld Barcelona earlier in the week, I have posted my Sliderocket presentation below as well as the recording thanks to the vBrown Bag team. If you could not attend the session or did not go to VMworld you can view all vBrown Bag Tech Talk recordings by checking out this link here. I highly recommend you check out the ENTIRE presentation as there was an exciting announcement that I made at the VERY END of the presentation.

Disclaimer: I think it should be pretty obvious, but things discussed in the presentation is not officially supported by VMware. Use at your own risk.

I would like to thank everyone that attended my session, really enjoyed the crowd and the questions/discussions afterwards. I know you could have been else where such as the Solution Exchange with a nice beverage, so thank you all for attending and hopefully everyone enjoyed it. I would also like to give a big shout out to vBrown Bag team Jon Harris, Kyle Murray, Damian Karlson and Gregg Robertson for putting together such an awesome event up for the VMware community and for their assistance on quickly getting the AV all setup for my presentation.

Here is the presentation:

Here is the video recording:

Here is a couple of pictures from the audience:

Quick Tip: New Hyper-V guestOS identifier in vSphere 5.5

by // 16 Comments

For those of you who are so inclined to run Hyper-V as a nested VM on ESXi 5.5 (not sure why anyone would want to do such a thing), you should be aware that there is a new guestOS identifier that you can configure your VM to for the most optimal configuration. The main reasons you would want to use this configuration is that by default Windows Enlightenment is enabled and this will prevent Hyper-V from running as it will detect it is running inside of a VM. This configuration will disable Windows Enlightenment to allow you to run Hyper-V.

I noticed a new guestOS identifier called "windowsHyperVGuest" while browsing through the vSphere 5.5 API Reference guide, but when I checked the vSphere Web Client, I did not see this guestOS type as an available option. Perhaps this was not a supported guestOS, after all Nested Virtualization is not officially supported by VMware. In any case, you can still configure your VM by leveraging the vSphere API.

Here is a quick vSphere SDK for Perl script called changeGuestOSID.pl which allows you to reconfigure a VM with a valid guestOS identifier from the vSphere API Reference guide. You can of course easily do this using PowerCLI as well as any other language that can speak to the vSphere API.

Once updated, you should now see it reflected in both the vSphere Web/C# Client:

Note: I did not do extensive testing other than basic installation of latest Hyper-V Server and I do not believe you need any additional settings. If you wish to run nested 64-bit VMs, then you will need to enable VHV.