Using Nested Virtualization and specifically Nested ESXi, which is running ESXi inside of a VM, has become so automatic and second nature for me and many in the VMware community, that I sometimes forget we still have brand new users who are experiencing ESXi for the very first time.
Nested ESXi is an extremely powerful capability for so many different use cases from development, testing and general learning purposes. Before jumping in and deploying your Nested ESXi environment, which I highly recommend using my Nested ESXi Virtual Appliances, you need to make sure that you have proper networking setup or you will run into all sorts of strange issues. For all VMware Nested Virtualization resources, tips and tricks, you will definitely want to bookmark this page HERE.
Disclaimer: Nested Nested ESXi is not officially supported by VMware and Nested Virtualization is only supported under limited scenarios. For more details, please refer to this VMware KB 2009916.
While new users of ESXi may not be aware of the networking requirements, I have also come across issues reported from more experienced ESXi users and issue was resolved once I had reminded them of the networking requirements, which they knew but had totally forgotten about! It goes to show that even the most experienced users can also forget the basics and it certainly is something that can easily be missed, I admit I have done it a few times when installing or re-installing a new setup 🙂
I figured this was a good time to publish a refresher on the networking requirements, the reason for it and the different ways you can meet this requirement depending on the version of ESXi you are using in your setup.
Networking Requirement:
- Enable either Promiscuous Mode and Forged Transmit OR MAC Learning on the virtual switch of your the physical ESXi host. See this blog post HERE for the reason why this is needed and this blog post HERE to understand the performance impact to your environment when Promiscuous Mode is enabled specifically.
Note: If you have the ability to use the MAC Learning capability of the virtual switch of your the physical ESXi host, this will provide you with the best experience and no performance impact since the need for Promiscuous Mode is not required. If you must enable Promiscuous Mode, then take a look at the network recommendations below on ways to mitigate the performance impact.
Networking Configuration:
For a Virtual Standard Switch (VSS), you can find the Promiscuous Mode and Forged Transmit settings under the Security section of the vSwitch, both of which default to reject.
For a Virtual Distributed Switch (VDS), you can find the Promiscuous Mode and Forged Transmit settings under the Security section of a Distributed Portgroup both of which default to reject. If you are using vSphere 7.0 or later, you can also find the native MAC Learning configuration under this section, which is also disabled by default.
For NSX, you can find the MAC Learning configuration under an NSX Segment within the MAC Discovery Segment Profile. By default, MAC Learning is disabled and you will need to create a custom MAC Discovery Segment Profile that has MAC Learning enabled.
Click into the default MAC Discovery Segment Profile to create a new custom profile that contains the enablement of MAC Learning setting and apply that profile to the desired NSX Segment once you hare done.
Note: Enablement of Promiscuous Mode, Forged Transmit and MAC Learning can all be automated using either vSphere or NSX API, the examples above was to illustrate the quickest way to enable these settings for users who may only need to do this once.
Networking Recommendations:
- ESXi 5.0-5.5
- Virtual Standard Switch (VSS) - Use the ESXi dvFilter Fling (vmware-esx-dvfilter-maclearn-1.0.vib) to reduce the impact of enabling Promiscuous Mode
- ESXi 6.0-6.5
- Virtual Standard Switch (VSS) - Use the ESXi MAC Learning dvFilter Fling (esx-dvfilter-maclearn-6.5.0.vib) to reduce the impact of enabling Promiscuous Mode
- Distributed Virtual Switch (VDS) - Use ESXi Learnswitch Fling to reduce the impact of enabling Promiscuous Mode
- ESXi 6.7
- Virtual Standard Switch (VSS) - Performance is impacted by enabling Promiscuous Mode
- Distributed Virtual Switch (VDS) - Use Native MAC Learning on VDS portgroup via vSphere API
- ESXi 7.0
- Virtual Standard Switch (VSS) -Â Performance is impacted by enabling Promiscuous Mode
- Distributed Virtual Switch (VDS) - Use Native MAC Learning on VDS portgroup via vSphere UI/API
- NSX (C-VDS/N-VDS) - Enable MAC Learning on NSX Segment
- ESXi 8.0
- Virtual Standard Switch (VSS) -Â Performance is impacted by enabling Promiscuous Mode
- Distributed Virtual Switch (VDS) - Use Native MAC Learning on VDS portgroup via vSphere UI/API
- NSX (C-VDS/N-VDS) - Enable MAC Learning on NSX Segment
I've been doing nested virtualization of ESXi and NSX for years under VMware Workstation which works fine, but I was wondering is there any advantage in using ESXi instead Workstation as the virtualization host?
My work has always involved ESXi, so I’ve never used Workstation/Fusion in any serious manner, so I can’t comment. I know for many, they don’t have infinite amount of memory on this desktop which is always a constraint in lab setup. With modern hardware and larger setups being more common, I can see that being nice benefit where you have a single system and you get benefit of virtual networking built right in.
Maybe others can comment on their experience but I’m happy with running my vSphere setup not on my workstation as I used many features at the physical level and there’s unneeded complexity to do all of that Nested which can also burn resources, especially if you don’t have lot
Super helpful as always! I think it would also be helpful to add information about setting a larger MTU than default on the outer vCenter. When doing nested NSX-T this was the time killer for me - it took me days to figure it out and I'm still not sure I've done it properly. I ended up setting it on the vSwitch, the VMkernel adapter, and the physical switch as well.
Anytime you deal w/MTU, it must be consistent throughout the entire network path both Virtual and Physical (which folks sometimes forget). NSX has requirements that must be met if you intend to use, so those must also be translated when doing Nested. Depending on your goals/setup, you may or may not do full NSX setup, so it may depend but good call out for those looking to use NSX overlay networking
Hi William, the mac learning dv filter fling states that it should be working also for a legacy standard vswitch. Since R6.7 mac learning is part of ESXi and we should not have to load the fling anymore. However all new posts on mac learning are only talking about dvswitches. my question is can we use mac learning in ESXi8 on a standard vswitch? if yes how.
No, while improvements have been made to add native MAC Learning as you've mentioned, that is _only_ available when using VDS. As the Networking Recommendation already breaks down what you can and can NOT do, please have a read again if you still have questions
Hi William, I am experimenting with 8.1 hosts on top of ESXi 8.1 with a dvSwitch. I have found that MAC learning alone is not enough, once I have enabled forged transmits in addition to MAC learning I get a response now via icmp.
Another blog post mentions the same:
https://fojta.wordpress.com/2020/07/06/enable-mac-learning-as-default-on-vsphere-distributed-switch/
"Forged transmits might be needed to be set as well. Therefore this line needs to be added to the script."
Same here. Needed to flip on Forged transmits also
I'm using nested esxi on vCloud Director but the problem is that all networks are in the same subnet.
nested esxi ip manage ping to gateway vcloud = pass
nested esxi ip manage ping to guest on vcloud = pass
nested esxi ip manage ping to vm guest on nested esxi = pass
vm guest on nested esxi ping to nested esxi manage = pass
vm guest on nested esxi ping to another guest on nested esxi = pass
vm guest on nested esxi ping to guest on vcloud = failed
vm guest on nested esxi ping to gateway vcloud = failed
Is there a way to make vm guest on nested esxi access another guest on vcloud and access to gateway on vcloud?
This section I don't think is quite right. I've got 7.0 U3 vSphere setup and the MAC Learning options are not shown under the Security section of the Distributed Portgroup. I have another 8.0 U2 setup and it is shown in that location....my guess is the UI didn't get this until 8.x and not vSphere 7.0.
<<<<<>>>>>
William, thanks a lot! Turning these functions ON every time on nested stuff, always forget to turn on on primary ESXI host.... drives me nuts =D