Last week at VMworld 2013, VMware announced the release of vSphere 5.5 which includes a variety of exciting new features. One of the most anticipated feature introduced in this release is VMware Virtual SAN (VSAN) which will be available initially as a public beta. One question that I heard repeatedly throughout the VMworld conference was whether it would be possible to test VSAN in a nested ESXi environment? The answer is absolutely! This is a great way to learn about VSAN and how works from a functional perspective before procuring the necessary hardware.
Disclaimer: Running VSAN in a nested ESXi environment is not officially supported nor is it a replacement for actual testing on actual physical hardware.
Before getting started, I would highly recommend you check out the following resources from my good friend Cormac Hogan which includes a detailed VSAN walk through as well what looks to be an awesome series of articles on how VSAN works:
Requirements:
- Environment running either vSphere 5.1 or 5.5 and access to the vSphere Web Client.
Configuration:
Nested ESXi VM configured with the minimal resources:
- 2 vCPU
- 5GB Memory (ESXi 5.5 now requires a minimum of 4GB vs 2GB as with previous releases but VSAN requires minimum of 5 with recommended 6)
- 2GB Disk for ESXi 5.5 installation
- 4GB Disk for an "Emulated" SSD
- 8GB Disk for HDD
Easy Method:
Instead of having you go through the process of building a Nested ESXi VM with all the prerequisites that includes steps from here and here. I have pre-built a VSAN Nested ESXi VM template (217Kb) that you can just download and import into your environment and being the installation process.
Download either:
- Single VSAN Nested ESXi VM Template
- 3-Node VSAN Nested ESXi VM Template
- 32-Node VSAN Nested ESXi VM Template
and connect to your vCenter Server 5.1 or 5.5 using the vSphere Web Client and import the OVF into your environment (do not use the vSphere C# Client as the import does not persist VHV configuration). Once you have imported the VM, you can then mount the ESXi 5.5 ISO and begin the installation. All three VMDKs have been thin provisioned and you can change the capacity during deployment.
Slightly Harder Method:
If you wish to build the Nested ESXi VM yourself, then you can follow these instructions:
Step 1 - Create a new VM and when you get to the compatibility screen, select either "ESXi 5.1 or greater" or "ESXi 5.5 or greater" depending on the version of vSphere you are running
Step 2 - For the GuestOS select "Other" and "Other (64-bit)"
Step 3 - We will need to customize the following virtual hardware configuration:
- Change vCPU to 2
- Click on CPU drop down and enable "Expose hardware assisted virtualization to the guest OS"
- Change Memory to 4GB
- Change the initial VMDK to 2GB or whatever value you wish to use for ESXi installation
- Add second VMDK with 4GB or whatever value you wish to use for "emulated" SSD
- Add third VMDK with 8GB of whatever value you wish to use for the HDD
- Click on the VM Options tab at the top and select the "Advanced" drop down box. We will need to add the following entry scsi0:1.virtualSSD = 1 For more details please refer to this article
Step 4 - Click okay to provision the VM and once it has been deployed you will need to re-configure the guestOS to "VMware ESXi 5.x" using the vSphere C# Client for vSphere 5.1 or vSphere Web Client for vSphere 5.5. At this point, you will have the same VM image as in the Easy Method and you are now ready to install ESXi 5.5
When you install ESXi 5.5, you should see the following three disks as shown in the screenshot below, ensure you install ESXi on the 2GB disk:
Prior to enabling VSAN on the particular vSphere Cluster, make sure you enable the new VSAN traffic type on one of your VMkernel interfaces for each of your ESXi hosts, this is required for VSAN communication.
If all the prerequisites have been met, you can now easily enable VSAN by simply checking the VSAN box when editing the vSphere Cluster. In just a few minutes you should see diskgroups automatically created (assuming you selected Automatic mode) consuming both the emulated SSD and HDD and the creation of the vsanDatastore which will be available on all ESXi hosts within that vSphere Cluster.
You can also use the same method for emulating an SSD running in a Nested ESXi to functional test the new VMware Flash Read Cache (vFRC) feature.
Isn't this essentially testing the GUI?
Not sure I follow your comment?
Sorry for not being clear. Without the hardware available to test the actual functionality of vSAN, running it in nested ESXI isn't really going to do much other than show you the configuration screens. I think in order to get the full benefit, this is a technology that needs to be deployed and actually tested in regards to performance, etc.
Just to note, if it's less than 4GB(I chose 1 for the lab), it won't work and it won't error out
Just a note that I was testing this in fusion and on SSD, a few things needed to be different:
1. The template doesnt import so you need to create the VM manually
2. As i was running the ESX VMs on SSD they automatically show as SSD without the change
3. You need to change the third disk to be not SSD by using scsi0:2.virtualSSD = 0
for some reason i do not get the nics in that position that all three hosts do see theier local drive's to bundle them as just one box of disks.
the summery of the SAN Cluster still say "Misconfiguration detected" although all three ESX5.5i boxes do have a vmkernel for vsan configured all on one vlan on just one fysical nic.
any idea how to solve this ??
Not exactly sure I follow what issues you're having.
I saw the same message,it was related to having a second vMkernel port group configured for vSAN. I removed the second portgroup and enabled vSAN on the primary management portgroup and fixed it right away.
I'm also running into the "Misconfiguration detected" vSAN network status error in my nested ESXi environment right after I turn on virtual SAN. The C# client says "Host cannot communicate with all other nodes in the VSAN enabled cluster" on the Summary tab. I'm able to ping across all three vSAN DHCP IPs from the nested ESXi hosts. I'm using vCenter 5.5.0 Build 1440532 and ESXi 5.5.0 Build 1439689 for both nested and physical ESXi. I'm using the template above. Anything else I can check to help understand what's failing? Any log files? All three nested ESXi virtual hosts are running on the same physical host. Can this still be a multicast issue? Thanks.
I narrowed the issue down to one specific item: subnet mask must not be 255.255.0.0. Subnet mask 255.255.255.0 works fine.
When using DHCP (as this WMware YouTube video shows) subnet mask 255.255.0.0 is used, and vSAN communication fails.
For those experiencing the "Misconfiguration detected" network status error on their VSAN setup running in a Nested ESXi environment, make sure you have Promiscuous Mode enabled (accepted) on the Port Group/vSwitch that the nested host VMs are attached to. This is normal for a Nested ESXi environment, but might have been missed during a new setup to test VSAN.
I also ran into this infamous problem. Turned out that my mistake was to convert a preconfigured ESXi-VM to a template and deployed the other VMs from that. Does *NOT* work with VSAN!
Now the template contains exerything but ESXi iteslf. Has to be installed after boot in every new VM.
I'm running into the issue that VSAN is unable to create LSOM file system for the data disk, i'm using a SSD and used the NO-SSD parameter, for the rest there are no error
Are you using my OVF template & is this 5.5u1?
I too get the "Unable to create LSOM file system". I created the disks myself and did not use the OVF yet.
How much memory did you allocate, I suspect you may not have enough memory which is why you're seeing this problem. You can always use the OVF template which will work OOTB
I have a two node cluster physical servers and I used your 3-Node OVF. So now I have a vApp with three VMs on which I have now installed ESXi 5.5. I am unclear of next steps -
"Prior to enabling VSAN on the particular vSphere Cluster, make sure you enable the new VSAN traffic type on one of your VMkernel interfaces for each of your ESXi hosts, this is required for VSAN communication"
I do not see an option of vSAN traffic type on my physical servers. Also, should I move the nested servers out from vApp into a new cluster so I can enable vSAN on that cluster?
The VSAN Traffic Type is not on your physical servers 🙂 This is in the VMkernel interface, you can do this using the vSphere Web Client by browsing to your ESXi host and selecting Networking and VMkernel Adapters and you can add "Virtual SAN" as a traffic type.
The other option is to do this via the command-line via ESXCLI using "esxcli vsan network ipv4 add -i"
I had missed the fundamental step of creating a new cluster and adding the new VMs, as hosts, to the cluster. All all settings I enabled vSAN and got a vsandatastore and diskgroups for individual hosts but none of the eligible disks were pulled in. Looking at tasks I see "A general system error occurred. Can not add disk: VSAN is not licensed on this host".
I am using the default 60 day trial license for VSAN but I thought that would work.
Hi William, I deployed the nested VM OVF, attached to vCenter, applied VSAN license and done the VSAN network creation, however for some reason none of the disk is eligible for VSAN from the web client.. Did I missed out anything?
Thanks
How did you import the OVF, remember it needs to be done using the vSphere Web Client as mentioned in the article. Else you will have issues
Hi William,
Yes, imported the Off via the Web client...
Cheers,
That's quite strange. Did you edit the Nested ESXi VMs such as disk size or memory? When ESXi is installed, do you see the other disks and are they marked appropriately?
William - Trying to setup a VSan and ran into an issue. I was hoping you can shed some light on my question.
3 Identical servers, each having a Fusion IO used for cache, 1 pair raid 1 SAS drives for VM Host, 1 Pair Raid 1 SSD for Disk Group 1 and 1 pair Raid 1 SAS HD for Disk Group 2. Can I share the Fusion IO cache for both Disk Groups? The system seems to not want to allow me to configure it like this.
Hi
I´m trying to deploy the Nested-ESXi-3-Node-VSAN-Template in an ESXi 5.1.0 but this message appears:
"The ovf package requires support for vApps with multiple VMs.
Details: Line 23: Unsupported element 'VirtualSystemCollection'"
Do i need to deploy it in a server with different capabilities or a diferent version of ESXi?
What type of server do i need to make it work?
Any advice please.
Thanks in advance.
Regards
You will need to deploy the OVF against a vCenter Server using the vSphere Web Client.
Thanks for your reply William, sorry for the delay.
I set this up like most others, on an ssd. Everything configured fine, got errors to go away with enabling igmp on my Asus home router but as soon as I put any mild load on it, like vmotion or install win on new VM, I get the host communication error and whatever I'm doing fails. Anyone seeing this? Did I miss configure something?
I did all of that, but it's complaining about upgrading the disks from 1 to 2 (I'm using esxi 6 & vSphere 6).
I'm unable to upgrade them.
If you could do a tutorial on doing VSAN 6, that would be great.
Thanks.
Thanks for these templates! I ended up creating my own manually for 6.5. One thing I'm seeing is an issue with NICs (may not be an issue but me). So i have 4 NICs in a Supermicro E200-8D server. I can't seem to utilize the 3 additional NICs on the nested hosts. Not sure if I'm missing something so for now I have vmotion, vsan & management all using the same vmkernel port. Not preferred but for the lab and in a pinch its working fine. Any ideas on how I can leverage the additional NICs?