Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:
This program has encountered an error:
Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320
This of course peaked my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.
Here is a quick screenshot of error message:
I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.
Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.
To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.
Run the following ESXCLI command (either locally on ESXi Shell or remotely)
esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1
A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂
Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.
Hi ,William!
Thank you for your great article.
The command "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1" should be run on the ESXi Host(The really one),right?
I have a question with it. If I run the command,does it make any negative consequence for the VSAN?I mean it sounds like to hack the ESXi, so I worry about that.
Thank you very much.(Sorry for my English.)
Hi William!! great great article... so much useful... thanks and regards =)
Hi, I'm getting the same 1048320 error during the /usr/sbin/vmkfstools stage, but I'm not trying to do a nested config, I'm doing a bare metal install on a Cisco C240 with RAID6 4Tb drive config (36Tb total).
We were able to install no problem on 2 other C240's (only difference was they had 2Tb drives - 18Tb total)
Had an open VMware case for a while and not much to show - latest suggestion is to make the 36Tb volume smaller as a test.
Any other ideas?
Fletcher. Did you ever come up with a solution? I am having the same problem you were having.
thanks brother
Hey I was wondering if this has an performance implications on the vSan Datastore?
Just a note that this command needs to be run on every host in the Vsan cluster. Thought this would be fixed by vsan 6, but it's not 🙁
I have a question about whether this will cause any side-effects on VSAN hosts which also run real LUNs.. Will faking reservations carry over to my real block storage which require it? Is the host smart enough to which one to fake it on and which one not to?
I guess it doesn't seem to have any negative effect, as I enabled it and my LUNs are all still working. They do have hardware accelerated ATS locking enabled (don't use scsi-2 or 3 locking) so not sure if that is relevant or not, and at any rate the command from this article does have the path VSAN in it, as if it only changes it for VSAN..
esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1
I wonder why they don't just default it to enabled if it doesn't have any bad effects?
Oh well, I will leave it enabled then. THANKS
Still works in vSphere 6.5!
Just in case, what is the command to undo the fake setting? would it be safe to assume that setting it to 0 as in "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 0" would reverse the change?
Yes, 0 = false and 1 = true
Thanks William for writing this articals, which will really helpful for us to play with VMC vmware ground.
recently we are using this ground for our product and recently I desided to use the VMC SDDC heavy node for nested purpose and distribute with my team to utilize the SDDC.
here I followed your Nested esx artical and I stuck at one point, while using the SDDC defined datastore.
VMC will not allow to RUN this command on there server. "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1"
So instead of that I created a Ubuntu NFS server in VMC Vcenter - Compute-Resource and utilized all the disk space available in SDDC.
and mounted NFS drive in to nested esx as a playground for nested setup.
Again Thank you for creating "VirtuallyGhetto".
Thank you so much. 🙂
This article is more than 9 years old, when ESXi 5.5 was state of the art and "VMFS-5 no longer used SCSI-2 reservations".
When deploying a Nested ESXi nowadays (November 2022) on top of vSAN 7 or even 8 - is this medieval trick with tweaking and faking SCSI-2 reservations really still needed? Was the LVM not updated to not require it anymore? Has the "faking" not being implemented by default in the meantime?
I'm asking because it's mentioned for the latest vSphere 8 nested setup script, and somehow can't believe this hangover has survived a decade...
thank you bro William, it work!
regards,
haikal s.