Many of us who run Nested ESXi in our home labs for development/testing purposes are pretty familiar with the requirements to properly setup a Nested ESXi environment such as CPUs supporting both Intel-VT+EPT or AMD-V+RVI and enabling both Promiscuous Mode and Forged Transmits on the portgroup that your Nested ESXi VM is connected to. Though these requirements have become second nature to most of us, it may not always be obvious on why they are required, especially for new users of Nested ESXi.
UPDATE 09/01/2014 - Take a look at this article for an updated solution to the problem mentioned below.
I specifically wanted to focus on the networking requirements where both Promiscuous Mode and Forged Transmits are required to be enabled. At a high level, most of us have understood this as a prerequisite for proper network connectivity for the Nested Virtual Machines running inside of your Nested ESXi host, but why is that?
Promiscuous Mode:
Both VMware VSS (Virtual Standard Switch) and VDS (vSphere Distributed Switch) do not implement MAC Learning like a traditional network switch, since the vSphere platform already knows which MAC addresses are assigned to a particular Virtual Machine. This means that the virtual switch will only forward network packets to a Virtual Machine if the destination MAC Address matches the ESXi vmnic's (pNIC) MAC Address.
In a Nested ESXi environment where you can have Nested Virtual Machines, the destination MAC Address for network packets destined to those Virtual Machines will differ from the Nested ESXi vmnic's MAC Address. Due to this, the physical ESXi host's virtual switch will drop the packet if Promiscuous Mode is not enabled. Promiscuous Mode allows the underlying Nested ESXi VM vmnic to monitor all traffic of the virtual switch it is connected to and thus providing connectivity to the underlying Nested Virtual Machines.
An interesting observation was recently made by Anthony Spiteri with his article about Reduced Network Throughput with Promiscuous Mode PortGroups. Since Promiscuous Mode allows all traffic from the virtual switch to be visible on the configured portgroup, there is definitely going to be some amount of overhead when enabling this setting. If you drive a large amount of network traffic for your regular Virtual Machines, you may want to consider separating out your Nested ESXi environment.
Forged Transmits:
Chris Wahl has already written an excellent article on Forged Transmits and its implication with Nested ESXi. I highly recommend you check out this blog post for the details.
Additional Resources:
Great information and quite timely for me. Article caught my eye immediately. Thanks.
I've just setup a new nested environment. I didn't want to enable promiscous mode on the production portgroups so I added 3 new vESXi portgroups for Mgmt/vMotion/Guests on the dvSwitch of the hosts the vESXi will be running on. Will this prevent the probleme of the reduced network performance?
Thanks!
The bit about VSS/vDS not doing mac learning was the missing piece for me to make sense of all of this.
If you have a two-level nested environment, make sure you only have one NIC enabled at the first level. I found that by even using a standby adapter, packets got duplicated at this level despite that the second vmnic was in standby mode. By removing the standby adapter, the duplication of packets disappeared. This was tested with ESX 5.1.
The reason for this may be related to the promiscuous mode on the physical level. With two adapters enabled at the first nested level, any packet arriving on nic1 will be forwarded to nic2 - despite that this adapter is listed as standby in the host of vESX-level1.
By the way, thanks to Lam for this great reference !
From a related conversation with VMW engineers on this topic: "By default when you install ESX and create the initial VMkernel interface for management traffic the MAC address of the management NIC takes the on the MAC address of the lowest ordered physical NIC of the host. In other words the management interface MAC and physical MAC are the same. You can apply this technique to any nested vVM. That is to say that you create a portgroup that has a single uplink via the underlying vESX host. You change the MAC address of the vVM to match that of of the vESX hosts’ uplink on the associated portgroup that the VM is connected to. By doing this, you are essentially faking the vVM into looking like the vESX hosts’ “physical” NIC so traffic will be sent to the VM.
Expanding on this technique, if you nest an L3 device within a vESX host and assign one of its NICs MAC address to be the same as the underlying vESX host then you can redirect all vVM traffic to this L3 device, out the vESX host and onto the network. In this case the L3 device serves to learn the MAC addresses of the vVMs.
Another technique would involve using NSX to create vxlan backed networks across vESX hosts and then use L2 bridging to get the traffic back onto unnested L2 segment."
I have 8 virtual ESXi hosts nested (for lab use), 4 VMs are on a 4-host physical VSAN cluster, and the other 4 VMs are on a 2-node physical VSAN cluster (w/ witness VM).
All of the 8 virtual ESXi hosts are network attached to an NSX (VxLAN vDS portgroup), and I was having the above issue described in this article. From a Windows VM 'jump server' connected to this same VxLAN portgroup I could ping the ESXi management VMK, but not a second VMK I created, nor could I get to any VMs running on the virtual host.
I had tried enabling Promiscuous mode on the virtual ESXi host switches (had been going off memory where I thought I read about the requirement to turn that on), but that didn't help of course.
I later figured out that it had to be something in the 'NSX' network but then found this article and realized that you make the above setting changes on the physical cluster, not the nested cluster (Duh, hehe).
So it was 'kind of' in the NSX but not really. I simply had to go into the VxLAN (NSX-created vDS portgroup) in vCenter, and enable promiscuous mode, and forged transmits only on the one portgroup and then the pings came right up.
THANKS