WilliamLam.com

Re: Host is in a VSAN enabled cluster but does not have VSAN service enabled

by // Leave a Comment

I recently noticed a couple of people hitting a warning message when configuring VSAN and specifically when running VSAN in a Nested ESXi environment (which is not officially supported by VMware). The warning message is displayed on the summary page of the ESXi host which states the following:

Host is in a VSAN enabled cluster but does not have VSAN service enabled

vsan-minimal-memory
The reason you are seeing this issue is related to the amount of memory you have allocated for your Nested ESXi VM. During the VSAN Beta, the minimal amount of memory was 4GB but it looks like that has changed to 5GB with the GA of vSphere 5.5 Update 1 last week. I know we could have done better job with the error message and communicate the actual underlying issue (will ensure we have an FR filed for this).

However, the fix is quite simple, just shutdown your Nested ESXi VM and then change it to 5GB and this message will go away. It is also worth noting that as you increase the number of disks and disk groups in your ESXi hosts, there will be an increase in memory. I would highly recommend you take a look at the official VSAN Design & Sizing guide to properly size out your real VSAN environments.

For basic functional testing and education of VSAN (not including running additional VMs), running a Nested ESXi VM with 5GB will be sufficient. You can also take a look at the my VSAN Nested ESXi OVF template which can just download and install ESXi 5.5 Update 1 without any issues.

How to properly clone a Nested ESXi VM?

by // 52 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349 

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

  • The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
  • The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

Why is Promiscuous Mode & Forged Transmits required for Nested ESXi?

by // 28 Comments

Many of us who run Nested ESXi in our home labs for development/testing purposes are pretty familiar with the requirements to properly setup a Nested ESXi environment such as CPUs supporting both Intel-VT+EPT or AMD-V+RVI and enabling both Promiscuous Mode and Forged Transmits on the portgroup that your Nested ESXi VM is connected to. Though these requirements have become second nature to most of us, it may not always be obvious on why they are required, especially for new users of Nested ESXi.

UPDATE 09/01/2014 - Take a look at this article for an updated solution to the problem mentioned below.

I specifically wanted to focus on the networking requirements where both Promiscuous Mode and Forged Transmits are required to be enabled. At a high level, most of us have understood this as a prerequisite for proper network connectivity for the Nested Virtual Machines running inside of your Nested ESXi host, but why is that?

Promiscuous Mode:
Both VMware VSS (Virtual Standard Switch) and VDS (vSphere Distributed Switch) do not implement MAC Learning like a traditional network switch, since the vSphere platform already knows which MAC addresses are assigned to a particular Virtual Machine. This means that the virtual switch will only forward network packets to a Virtual Machine if the destination MAC Address matches the ESXi vmnic's (pNIC) MAC Address.

In a Nested ESXi environment where you can have Nested Virtual Machines, the destination MAC Address for network packets destined to those Virtual Machines will differ from the Nested ESXi vmnic's MAC Address. Due to this, the physical ESXi host's virtual switch will drop the packet if Promiscuous Mode is not enabled. Promiscuous Mode allows the underlying Nested ESXi VM vmnic to monitor all traffic of the virtual switch it is connected to and thus providing connectivity to the underlying Nested Virtual Machines.

An interesting observation was recently made by Anthony Spiteri with his article about Reduced Network Throughput with Promiscuous Mode PortGroups. Since Promiscuous Mode allows all traffic from the virtual switch to be visible on the configured portgroup, there is definitely going to be some amount of overhead when enabling this setting. If you drive a large amount of network traffic for your regular Virtual Machines, you may want to consider separating out your Nested ESXi environment.

Forged Transmits:
Chris Wahl has already written an excellent article on Forged Transmits and its implication with Nested ESXi. I highly recommend you check out this blog post for the details.

Additional Resources: