During the development of vSphere & VSAN 6.0, I built several new OVF templates to be able to help quickly deploy different configurations of VSAN to help test and provide feedback to Engineering. I figure I would share out these new templates as it may also benefit others in the community who wish to quickly setup VSAN 6.0 and give it spin in their own home lab or development environment using Nested ESXi.
Disclaimer: Nested ESXi is not officially supported by VMware, please use at your own risk.
One thing to be aware of when running VSAN 6.0 using Nested ESXi is that you will need a minimum of 6GB of memory for each ESXi VM. If you wish to run additional workloads on top or create larger VSAN Clusters, you may need to add additional memory on each ESXi VM. This is not really a problem for real physical hardware but can be a problem for smaller lab environments when running Nested ESXi.
Here are the four new VSAN 6.0 Nested ESXi OVF Templates:
- Nested-ESXi-3-Node-VSAN-6.0-Template - This is a standard 3-Node VSAN OVF
- Nested-ESXi-3-Node-VSAN-6.0-All-Flash-Template - This is a standard 3-Node VSAN with both disk2 & 3 set as an SSD for All-Flash configuration (Check out this blog article on steps of setting up an All-Flash VSAN environment)
- Nested-ESXi-6-Node-VSAN-6.0-FD-Template - This is a 6-Node VSAN which can be useful when wanting to test out VSAN's new Fault Domain (aka Rack Aware) feature
- Nested-ESXi-64-Node-VSAN-6.0-Template - This is mind-blowing 64-Node VSAN OVF! Can you handle all this awesomeness? If you can, would love to see you share some screenshots 🙂
Note: If you decide to build a 64-Node Cluster, you will need run the following ESXCLI command (reboot is required) to go beyond 32 nodes up to 64 node:
esxcli system settings advanced set -o /VSAN/goto11 -i 1
Here is a few screenshots of running VSAN 6.0 using the above OVF Templates:
Downloading the 64 node OVF now. What resources will I need to host it?
Hey Howard,
Each Nested ESXi VM is configured as the following:
2vCPU / 6GB memory (this is a minimum for VSAN 6.0)
Disk1 = 2GB for ESXi Install (size can be change upon deployment as part of the flexible OVF I've created)
Disk2 = 4GB for "Virtual SSD" (size can be change upon deployment as part of the flexible OVF I've created)
Disk3 = 8GB for "MD" (size can be change upon deployment as part of the flexible OVF I've created)
If you just want to stick with the defaults I've laid, then it would be 64 of these Nested ESXi VMs. On the disk side, I will assume you'll thin provision, so you definitely don't need all that storage capacity. Your limiting resource will be memory, so if you have several giant hosts you should be good to go. Look forwarding to see your setup. Let me know if you have more questions
Well that won't quite take all the capacity of the lab it will tie up my 3 biggest systems (96GB ea) and a couple of the 48GB ones.
I am a bit confused about deployment though. I tried deploying in vSphere 5.5 and got "OVF is invalid, Operation not supported on the object" Clearly these are a type of OVF I haven't deployed before.
Nice!
Yes, these OVFs will need to be imported through an existing vSphere Web Client, as it contains settings that aren't preserved when importing through C# Client. Hopefully this won't be a problem and make sure to accept the advanced options which you'll be prompted for.
Hi William
This idea is really fantastic!!
I am testing the standard 3 node but at boot time every node is not able to find the boot device?
Are DHCP and access to internet mandatory for the deployment? (I assume so as the OVF is really small 😉 )
Thank you!!
F.
Fabio,
Internet access is not required, these are just templates that you're downloading. You will still need to install ESXi as you normally would, but this saves you time from setting up the Nested ESXi VMs
Ok got it 😉
I was being too lazy 😛
Thank you!!
F.
I would be interested in the technique you have used to create these files. That would be really great for me to destroy and re-deploy my (nested) test lab quickly and without much operational overhead. Can you elaborate a little bit how you did that? Is there maybe a blog post already that I have missed?
Thanks!
Jason
Jason,
Take a look at these two articles for more details:
http://www.virtuallyghetto.com/2013/12/did-you-know-ovf-supports-cool-feature.html
http://www.virtuallyghetto.com/2013/12/deploy-truly-dynamic-ovf-using.html
Hi Will,
I'm building a nested lab for my DC and NSX lab prep.
The host I have has Dual quad Intel procs, 256 Gig of RAM, Quad 1Gig NIC's and 2x10Gig Dual NIC's. 1 TB of SSD.
My question is that will I be able to test all the features within a nested host or Do i need to go for individual 4 hosts.
Cheers!
Colin
Hi, I have deployed the 3 Node VSAN 6 OVF to my VCenter 6 environment. When the ESXi VMs boot up, they have no ESXi installed. Is this a manual step I must do? Apologies for the naivety!
From one of Williams previous answers - "You will still need to install ESXi as you normally would, but this saves you time from setting up the Nested ESXi VMs"
Hi William,
Thanks for the great blog as always.
I tried to deploy the Nested ESXi VSAN vAPP template using VCSA6.0 web interface. The template was deployed ok but "vhv.enabled="enable" option is not there. I can manually enable it in web client and then it will show up in vmx file. After template deployed, I only have the option of featMask.vm.hv.capable = "Min:1" in VMX.
any thoughts?
Thanks
Hongjun
For VHV to be retained, you MUST import using the vSphere Web Client as these settings are not understood when using the C# Client.
I did use the web interface as I have read your notes in the blogs for persistent vhv settings. BTW, I did NOT see web interface confirmation checkbox to confirm advanced settings. Should I see this checkbox with 6.0 web interface? The web interface just say click next to confirm the advanced settings. also "featMast.vm.hv.capable" and other advanced settings in your OVF file did apply to the VMX file.
ok. I got it working but still have confusion here.
One option got vhv.enabled option applied
1a) web client->Deploy OVF Template...->select local download Vghetoo VSAN 3-node template
this path I do see the checkbox of accepting advanced configuration
Two options can't get vhv.enabled option applied
2a) web client->New vApp->New vApp from Library
2b) web client->content libaries->vGehetoo-Nested-Esxi-Content-Libary->Select Nested 3-node VSAN Template->New vApp from This Template
These two paths have similar flow but one key screen is different, what I saw is below
https://www.dropbox.com/s/yphglq8xv91fc4x/vsan-template-vhv-not-working.png?dl=0
I'd rather to use 2b) flow for subscribing content library but then I got this "vhv.enabled" option issue.
thanks
Hongjun
There's a known bug where VHV is not properly applied (even though the OVF has it) when deploying from Content Library, this will be fixed in a future update.
thanks for the info! Hopefully this can be fixed soon to fully automate your nice vESXi templates from content library.
Hope all well with your family and baby.
Are fault domains a requirement when deployment an OVF? I'm running a POS and migrating my Horizon environment, built on 5.5 to 6.0u2. I've exported all the infrastructure servers to OVF but when attempting to deploy them in my 6 environment, I'm seeing errors related to needing 3 fault domains in the Webclient. Was curious if this is a hard requirement and if so, would it be best for me to present additional datastores in order to avoid the issue in the future.
Is there by chance a way to upgrade these templates to vm version 11?
I noticed when trying to auto deploy pxe boot them to a vsphere 6.0 image it took a VERY long time at first. By upgrading to version 10 it went much faster. Version 11 was not available to upgrade and I'm guessing it has something to do with a configuration option selected in the template. Thinking if I could get it on VMXNET3 vnics perhaps it would go even faster?
Rob,
I chose vHW10 to provide the wides level of compatibility for customers who may not be running the latest version of ESXi on their physical host. You're more than welcome to upgrade once you've deployed OR just tweak your version of the OVF to specify v11 as the default 🙂
Thanks for the article. I have a question. I have 2 servers both running ESXi 6.0 update 2 and 1 nested ESXi. I setup VSAN and added my disks. Each host has a yellow exclamation point and Host cannot communicate with all other nodes in the virtual VSAN enabled cluster. I believe it is the nested ESXi causing the issue. When I go to the other esxi servers - configuration - storage VSAN shows and status is Normal and it shows the correct size of 5.8TB. When I refresh it is still 5.8TB. On the nested ESXi it shows as normal but when I refresh it changes to 24.74GB. I then go back to either of the physical esxi hosts and refresh and it is back to the correct 5.8TB. Any ideas? I have the nested VSwitch set to promiscuous mode. Each standard VSwitch has a VSAN port.