WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / Nested Virtualization / How to run Nested ESXi on top of a VSAN datastore?

How to run Nested ESXi on top of a VSAN datastore?

11.07.2013 by William Lam // 34 Comments

Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course peaked my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.

More from my site

  • OVF template for creating Nested ESXi 3 or 32 node VSAN Cluster
  • Restoring VSAN VM Storage Policy without vCenter Part 2: Using vSphere API
  • Restoring VSAN VM Storage Policies without vCenter Part 1: Using cmmds-tool
  • w00t! VMware Tools for Nested ESXi!
  • Quick Tip: New Hyper-V guestOS identifier in vSphere 5.5

Categories // Nested Virtualization, VSAN, vSphere 5.5 Tags // esxi 5.5, nested, nested virtualization, scsi reservation, Virtual SAN, VSAN, vSphere 5.5

Comments

  1. Jack Kuo says

    08/03/2014 at 8:42 am

    Hi ,William!
    Thank you for your great article.

    The command "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1" should be run on the ESXi Host(The really one),right?
    I have a question with it. If I run the command,does it make any negative consequence for the VSAN?I mean it sounds like to hack the ESXi, so I worry about that.

    Thank you very much.(Sorry for my English.)

    Reply
  2. Moises Sanchez says

    08/20/2014 at 6:00 pm

    Hi William!! great great article... so much useful... thanks and regards =)

    Reply
  3. Fletcher Cocquyt (@Cocquyt) says

    10/01/2014 at 7:12 pm

    Hi, I'm getting the same 1048320 error during the /usr/sbin/vmkfstools stage, but I'm not trying to do a nested config, I'm doing a bare metal install on a Cisco C240 with RAID6 4Tb drive config (36Tb total).
    We were able to install no problem on 2 other C240's (only difference was they had 2Tb drives - 18Tb total)
    Had an open VMware case for a while and not much to show - latest suggestion is to make the 36Tb volume smaller as a test.
    Any other ideas?

    Reply
    • Damon says

      01/10/2015 at 12:34 am

      Fletcher. Did you ever come up with a solution? I am having the same problem you were having.

      thanks brother

      Reply
  4. Bill says

    07/23/2015 at 10:41 pm

    Hey I was wondering if this has an performance implications on the vSan Datastore?

    Reply
  5. Glenn says

    09/08/2015 at 6:52 pm

    Just a note that this command needs to be run on every host in the Vsan cluster. Thought this would be fixed by vsan 6, but it's not 🙁

    Reply
  6. Samsonite801 says

    10/02/2016 at 12:49 pm

    I have a question about whether this will cause any side-effects on VSAN hosts which also run real LUNs.. Will faking reservations carry over to my real block storage which require it? Is the host smart enough to which one to fake it on and which one not to?

    Reply
  7. Samsonite801 says

    10/02/2016 at 8:16 pm

    I guess it doesn't seem to have any negative effect, as I enabled it and my LUNs are all still working. They do have hardware accelerated ATS locking enabled (don't use scsi-2 or 3 locking) so not sure if that is relevant or not, and at any rate the command from this article does have the path VSAN in it, as if it only changes it for VSAN..

    esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

    I wonder why they don't just default it to enabled if it doesn't have any bad effects?

    Oh well, I will leave it enabled then. THANKS

    Reply
  8. Matt Heldstab says

    12/04/2016 at 8:42 am

    Still works in vSphere 6.5!

    Reply
  9. Denis says

    01/25/2017 at 6:08 pm

    Just in case, what is the command to undo the fake setting? would it be safe to assume that setting it to 0 as in "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 0" would reverse the change?

    Reply
    • William Lam says

      01/25/2017 at 7:22 pm

      Yes, 0 = false and 1 = true

      Reply
  10. SachiN says

    05/02/2019 at 2:57 am

    Thanks William for writing this articals, which will really helpful for us to play with VMC vmware ground.

    recently we are using this ground for our product and recently I desided to use the VMC SDDC heavy node for nested purpose and distribute with my team to utilize the SDDC.

    here I followed your Nested esx artical and I stuck at one point, while using the SDDC defined datastore.

    VMC will not allow to RUN this command on there server. "esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1"

    So instead of that I created a Ubuntu NFS server in VMC Vcenter - Compute-Resource and utilized all the disk space available in SDDC.

    and mounted NFS drive in to nested esx as a playground for nested setup.

    Again Thank you for creating "VirtuallyGhetto".

    Reply
  11. Leo says

    11/08/2020 at 12:16 am

    Thank you so much. 🙂

    Reply
  12. dcf77 says

    11/10/2022 at 3:13 pm

    This article is more than 9 years old, when ESXi 5.5 was state of the art and "VMFS-5 no longer used SCSI-2 reservations".

    When deploying a Nested ESXi nowadays (November 2022) on top of vSAN 7 or even 8 - is this medieval trick with tweaking and faking SCSI-2 reservations really still needed? Was the LVM not updated to not require it anymore? Has the "faking" not being implemented by default in the meantime?

    I'm asking because it's mentioned for the latest vSphere 8 nested setup script, and somehow can't believe this hangover has survived a decade...

    Reply
  13. haikalshiddiq says

    01/20/2023 at 1:31 am

    thank you bro William, it work!

    regards,
    haikal s.

    Reply

Trackbacks

  1. Consolidated list of all Virtual SAN (VSAN) deep dive resources. | says:
    04/09/2014 at 1:16 am

    […] How to run Nested ESXi on top of a VSAN datastore? (William Lam) […]

    Reply
  2. Prepping for VCAP-DCA (VDCA510) | virtualfng says:
    05/17/2014 at 1:21 pm

    […] i see William Lam saves the day yet again.  The article that addresses this issue is posted at virtuallyGhetto .  You can read the explanation on William's site.  Quick fix is to add this command to […]

    Reply
  3. Interop Tokyo 2014 Software Defined Storage コーナー デモ環境構築TIPS | Japan Cloud Infrastructure Blog - VMware Blogs says:
    07/23/2014 at 11:10 am

    […] vSphere に発行しておく必要があります。(参考: How to run Nested ESXi on top of a VSAN datastore?) ※これをしておかないと […]

    Reply
  4. Link to post from virtuallyghetto.com | How to run Nested ESXi on top of a VSAN datastore? | VDC Ninja says:
    10/07/2014 at 10:56 am

    […] http://www.virtuallyghetto.com/2013/11/how-to-run-nested-esxi-on-top-of-vsan.html […]

    Reply
  5. Nested ESXi Problems running on vSAN – Stateless caching – Cannot apply the host configuration – ACM Computers says:
    02/08/2016 at 6:31 am

    […] How to run Nested ESXi on top of a VSAN datastore? […]

    Reply
  6. VSAN 6.2 (vSphere 6.0 Update 2) homelab on 6th Gen Intel NUC | virtuallyGhetto says:
    03/21/2016 at 8:48 am

    […] of your physical VSAN Cluster, be sure to add this configuration change outlined in this article here, else you may see some strangeness when trying to create VMFS […]

    Reply
  7. Creating a Single-Node VSAN | iThinkVirtual™ says:
    05/02/2016 at 7:03 am

    […] And last, but definitely not least, if you intend on running any Nested ESXi VMs on your newly created VSAN, be sure to run the following command to prevent any errors when trying to create SCSI disks for your ESXi VM.  This will enable an advanced ESXi setting that will "fake" SCSI reservations.  William Lam has a nice post about this here. […]

    Reply
  8. Running Nested ESXi on VSAN | RJ Approves This Message! says:
    05/17/2016 at 11:25 pm

    […] Turns out the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. You can read more here. […]

    Reply
  9. Couldn't format VMFS on nested ESXi (VSAN datastore) - vBrain.info says:
    09/14/2016 at 1:02 am

    […] Lam wrote a post in response to Tim Smith's blog post about the problem that he couldn't install an […]

    Reply
  10. Creating a Single-Node VSAN - 下一站 says:
    05/06/2018 at 8:31 am

    […] And last, but definitely not least, if you intend on running any Nested ESXi VMs on your newly created VSAN, be sure to run the following command to prevent any errors when trying to create SCSI disks for your ESXi VM.  This will enable an advanced ESXi setting that will “fake” SCSI reservations.  William Lam has a nice post about this here. […]

    Reply
  11. Nested ESXi on VMware Cloud on AWS (VMC) says:
    09/28/2018 at 8:06 am

    […] is VSAN. If you want to run Nested ESXi on top of a physical VSAN-based datastore, you need to enable an ESXi Advanced Setting that "fakes" SCSI Reservations which is required if you wish to create either a VMFS-based or VSAN Datastore for your Nested ESXi […]

    Reply
  12. [VMware] vSAN環境でネステッドESXiのインストールが失敗する – virtual hive-jp says:
    03/20/2019 at 7:49 am

    […] Lambさんのブログでした。 […]

    Reply
  13. Solution: Nested ESXi 6.7 install fails with the error 'tryFormatDevice' - vPirate says:
    07/01/2020 at 4:07 am

    […] then ran into an article by Willam Lam about another issue about running the Nested ESXi VMs on vSAN storage. The error though, is about the installer not being able to format a VMFS […]

    Reply
  14. Automated Nested Lab Deployment on SDDC Part 2: Azure VMware Solution says:
    10/28/2020 at 12:37 pm

    […] ESXi hosts is not available. This is required to enable an advanced ESXi setting to be able to run Nested vSAN on top of the physical vSAN. From a networking standpoint, customers do have full access to the NSX-T Manager instance and this […]

    Reply
  15. Create a nested ESXi template in vSphere – thecjtlab says:
    11/17/2020 at 5:19 am

    […] **Note** I had to run the below command on my physical hosts I am running vSAN in my lab) to get this to work as per the below link form William Lam here […]

    Reply
  16. Nested ESXi on vSAN – PowerCLI to the rescue – ryanmiv says:
    01/05/2021 at 7:43 pm

    […] I'm working on building out a nested lab environment on top of vSAN. I immediately thought of the last time I tried this, where I ran into all sorts of issues, luckily William Lam had already discovered a fix […]

    Reply
  17. Prepare The Nested Environment for VCF – Rick's Blog says:
    09/27/2021 at 10:09 pm

    […] How to run Nested ESXi on top of a VSAN datastore? […]

    Reply
  18. Nested ESXi on Top of a vSAN Datastore – vAndu says:
    10/03/2021 at 8:13 am

    […] How to run Nested ESXi on top of a VSAN datastore? […]

    Reply
  19. Homelab Revamp 2022 – Part 4 – Build, Configure, Learn – Z8N says:
    06/25/2022 at 3:16 pm

    […] nested ESXi needs tweaksFor my labs I am running nested ESXi. Thanks to William Lam's blog I now know that for the VSAN case I have to enable a setting to […]

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Self-Contained & Automated VMware Cloud Foundation (VCF) deployment using new VLC Holodeck Toolkit 03/29/2023
  • ESXi configstorecli enhancement in vSphere 8.0 Update 1 03/28/2023
  • ESXi on Intel NUC 13 Pro (Arena Canyon) 03/27/2023
  • Quick Tip - Enabling ESXi Coredumps to be stored on USB 03/26/2023
  • How to disable the Efficiency Cores (E-cores) on an Intel NUC? 03/24/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023