Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course peaked my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.