WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple
You are here: Home / ESXi / Updated VSAN 6.0 Nested ESXi OVF Templates for 64 Nodes, All-Flash Array & Fault Domain Testing

Updated VSAN 6.0 Nested ESXi OVF Templates for 64 Nodes, All-Flash Array & Fault Domain Testing

02.12.2015 by William Lam // 22 Comments

During the development of vSphere & VSAN 6.0, I built several new OVF templates to be able to help quickly deploy different configurations of VSAN to help test and provide feedback to Engineering. I figure I would share out these new templates as it may also benefit others in the community who wish to quickly setup VSAN 6.0 and give it spin in their own home lab or development environment using Nested ESXi.

Disclaimer: Nested ESXi is not officially supported by VMware, please use at your own risk.

One thing to be aware of when running VSAN 6.0 using Nested ESXi is that you will need a minimum of 6GB of memory for each ESXi VM. If you wish to run additional workloads on top or create larger VSAN Clusters, you may need to add additional memory on each ESXi VM. This is not really a problem for real physical hardware but can be a problem for smaller lab environments when running Nested ESXi.

Here are the four new VSAN 6.0 Nested ESXi OVF Templates:

  • Nested-ESXi-3-Node-VSAN-6.0-Template - This is a standard 3-Node VSAN OVF
  • Nested-ESXi-3-Node-VSAN-6.0-All-Flash-Template - This is a standard 3-Node VSAN with both disk2 & 3 set as an SSD for All-Flash configuration (Check out this blog article on steps of setting up an All-Flash VSAN environment)
  • Nested-ESXi-6-Node-VSAN-6.0-FD-Template - This is a 6-Node VSAN which can be useful when wanting to test out VSAN's new Fault Domain (aka Rack Aware) feature
  • Nested-ESXi-64-Node-VSAN-6.0-Template - This is mind-blowing 64-Node VSAN OVF! Can you handle all this awesomeness? If you can, would love to see you share some screenshots 🙂

Note: If you decide to build a 64-Node Cluster, you will need run the following ESXCLI command (reboot is required) to go beyond 32 nodes up to 64 node:

esxcli system settings advanced set -o /VSAN/goto11 -i 1

Here is a few screenshots of running VSAN 6.0 using the above OVF Templates:

vsan-6.0-nested-esxi
vsan-6.0-fault-domain

More from my site

  • How to run Qemu & KVM on ESXi?
  • Automating Cross vCenter vMotion (xVC-vMotion) between the same & different SSO Domain
  • Custom vSAN HCL JSON for VMware Cloud Foundation (VCF) 5.1 and vSAN ESA using Nested ESXi
  • Easier method to simulate custom ESXi SMBIOS hardware strings
  • How To Quickly Get Started With The New VMware Puppet Modules

Categories // ESXi, VSAN, vSphere 6.0

Comments

  1. *protectedHoward Marks (@DeepStorageNet) says

    02/12/2015 at 8:28 pm

    Downloading the 64 node OVF now. What resources will I need to host it?

    Reply
    • William Lam says

      02/12/2015 at 9:44 pm

      Hey Howard,

      Each Nested ESXi VM is configured as the following:
      2vCPU / 6GB memory (this is a minimum for VSAN 6.0)
      Disk1 = 2GB for ESXi Install (size can be change upon deployment as part of the flexible OVF I've created)
      Disk2 = 4GB for "Virtual SSD" (size can be change upon deployment as part of the flexible OVF I've created)
      Disk3 = 8GB for "MD" (size can be change upon deployment as part of the flexible OVF I've created)

      If you just want to stick with the defaults I've laid, then it would be 64 of these Nested ESXi VMs. On the disk side, I will assume you'll thin provision, so you definitely don't need all that storage capacity. Your limiting resource will be memory, so if you have several giant hosts you should be good to go. Look forwarding to see your setup. Let me know if you have more questions

      Reply
      • *protectedHoward Marks (@DeepStorageNet) says

        02/14/2015 at 6:11 pm

        Well that won't quite take all the capacity of the lab it will tie up my 3 biggest systems (96GB ea) and a couple of the 48GB ones.

        I am a bit confused about deployment though. I tried deploying in vSphere 5.5 and got "OVF is invalid, Operation not supported on the object" Clearly these are a type of OVF I haven't deployed before.

        Reply
        • William Lam says

          02/14/2015 at 10:34 pm

          Nice!

          Yes, these OVFs will need to be imported through an existing vSphere Web Client, as it contains settings that aren't preserved when importing through C# Client. Hopefully this won't be a problem and make sure to accept the advanced options which you'll be prompted for.

          Reply
  2. *protectedFabio says

    02/13/2015 at 7:25 am

    Hi William

    This idea is really fantastic!!

    I am testing the standard 3 node but at boot time every node is not able to find the boot device?

    Are DHCP and access to internet mandatory for the deployment? (I assume so as the OVF is really small 😉 )

    Thank you!!

    F.

    Reply
    • William Lam says

      02/13/2015 at 3:02 pm

      Fabio,

      Internet access is not required, these are just templates that you're downloading. You will still need to install ESXi as you normally would, but this saves you time from setting up the Nested ESXi VMs

      Reply
      • *protectedFabio says

        02/14/2015 at 5:43 pm

        Ok got it 😉

        I was being too lazy 😛

        Thank you!!

        F.

        Reply
  3. *protectedJason says

    02/14/2015 at 11:54 pm

    I would be interested in the technique you have used to create these files. That would be really great for me to destroy and re-deploy my (nested) test lab quickly and without much operational overhead. Can you elaborate a little bit how you did that? Is there maybe a blog post already that I have missed?

    Thanks!
    Jason

    Reply
    • William Lam says

      02/15/2015 at 1:02 am

      Jason,

      Take a look at these two articles for more details:

      http://www.virtuallyghetto.com/2013/12/did-you-know-ovf-supports-cool-feature.html
      http://www.virtuallyghetto.com/2013/12/deploy-truly-dynamic-ovf-using.html

      Reply
  4. *protectedDat Colin says

    03/17/2015 at 1:13 pm

    Hi Will,

    I'm building a nested lab for my DC and NSX lab prep.

    The host I have has Dual quad Intel procs, 256 Gig of RAM, Quad 1Gig NIC's and 2x10Gig Dual NIC's. 1 TB of SSD.

    My question is that will I be able to test all the features within a nested host or Do i need to go for individual 4 hosts.

    Cheers!
    Colin

    Reply
  5. *protectedgss3 says

    07/28/2015 at 7:55 pm

    Hi, I have deployed the 3 Node VSAN 6 OVF to my VCenter 6 environment. When the ESXi VMs boot up, they have no ESXi installed. Is this a manual step I must do? Apologies for the naivety!

    Reply
    • *protectedDavid says

      08/04/2015 at 8:37 pm

      From one of Williams previous answers - "You will still need to install ESXi as you normally would, but this saves you time from setting up the Nested ESXi VMs"

      Reply
  6. *protectedhongjunma says

    11/05/2015 at 1:15 pm

    Hi William,

    Thanks for the great blog as always.

    I tried to deploy the Nested ESXi VSAN vAPP template using VCSA6.0 web interface. The template was deployed ok but "vhv.enabled="enable" option is not there. I can manually enable it in web client and then it will show up in vmx file. After template deployed, I only have the option of featMask.vm.hv.capable = "Min:1" in VMX.

    any thoughts?

    Thanks
    Hongjun

    Reply
    • William Lam says

      11/05/2015 at 1:52 pm

      For VHV to be retained, you MUST import using the vSphere Web Client as these settings are not understood when using the C# Client.

      Reply
      • *protectedhongjunma says

        11/05/2015 at 2:57 pm

        I did use the web interface as I have read your notes in the blogs for persistent vhv settings. BTW, I did NOT see web interface confirmation checkbox to confirm advanced settings. Should I see this checkbox with 6.0 web interface? The web interface just say click next to confirm the advanced settings. also "featMast.vm.hv.capable" and other advanced settings in your OVF file did apply to the VMX file.

        Reply
      • *protectedhongjunma says

        11/05/2015 at 3:36 pm

        ok. I got it working but still have confusion here.

        One option got vhv.enabled option applied
        1a) web client->Deploy OVF Template...->select local download Vghetoo VSAN 3-node template
        this path I do see the checkbox of accepting advanced configuration

        Two options can't get vhv.enabled option applied
        2a) web client->New vApp->New vApp from Library
        2b) web client->content libaries->vGehetoo-Nested-Esxi-Content-Libary->Select Nested 3-node VSAN Template->New vApp from This Template

        These two paths have similar flow but one key screen is different, what I saw is below
        https://www.dropbox.com/s/yphglq8xv91fc4x/vsan-template-vhv-not-working.png?dl=0

        I'd rather to use 2b) flow for subscribing content library but then I got this "vhv.enabled" option issue.

        thanks
        Hongjun

        Reply
        • William Lam says

          11/05/2015 at 5:02 pm

          There's a known bug where VHV is not properly applied (even though the OVF has it) when deploying from Content Library, this will be fixed in a future update.

          Reply
          • *protectedhongjunma says

            11/05/2015 at 5:34 pm

            thanks for the info! Hopefully this can be fixed soon to fully automate your nice vESXi templates from content library.

            Hope all well with your family and baby.

  7. *protectedDanilo Noriega says

    05/25/2016 at 11:59 am

    Are fault domains a requirement when deployment an OVF? I'm running a POS and migrating my Horizon environment, built on 5.5 to 6.0u2. I've exported all the infrastructure servers to OVF but when attempting to deploy them in my 6 environment, I'm seeing errors related to needing 3 fault domains in the Webclient. Was curious if this is a hard requirement and if so, would it be best for me to present additional datastores in order to avoid the issue in the future.

    Reply
  8. *protectedRob Babb (@atlantauser) says

    07/08/2016 at 7:41 am

    Is there by chance a way to upgrade these templates to vm version 11?

    I noticed when trying to auto deploy pxe boot them to a vsphere 6.0 image it took a VERY long time at first. By upgrading to version 10 it went much faster. Version 11 was not available to upgrade and I'm guessing it has something to do with a configuration option selected in the template. Thinking if I could get it on VMXNET3 vnics perhaps it would go even faster?

    Reply
    • William Lam says

      07/08/2016 at 8:20 am

      Rob,

      I chose vHW10 to provide the wides level of compatibility for customers who may not be running the latest version of ESXi on their physical host. You're more than welcome to upgrade once you've deployed OR just tweak your version of the OVF to specify v11 as the default 🙂

      Reply
  9. *protectedDavid says

    10/20/2016 at 6:43 am

    Thanks for the article. I have a question. I have 2 servers both running ESXi 6.0 update 2 and 1 nested ESXi. I setup VSAN and added my disks. Each host has a yellow exclamation point and Host cannot communicate with all other nodes in the virtual VSAN enabled cluster. I believe it is the nested ESXi causing the issue. When I go to the other esxi servers - configuration - storage VSAN shows and status is Normal and it shows the correct size of 5.8TB. When I refresh it is still 5.8TB. On the nested ESXi it shows as normal but when I refresh it changes to 24.74GB. I then go back to either of the physical esxi hosts and refresh and it is back to the correct 5.8TB. Any ideas? I have the nested VSwitch set to promiscuous mode. Each standard VSwitch has a VSAN port.

    Reply

Thanks for the comment!Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...