Over the years, several solutions have been developed here and here to help reduce the impact of promiscuous mode, which is a requirement for running Nested ESXi as a workload. Although these solutions worked extremely well, it however did require users to install additional software to enable this functionality. The most recent solution was a new Learnswitch VMkernel module (released as a VMware Fling) that enables MAC learning capabilities on ESXi.
Today, I am pleased to announce that with the release of vSphere 6.7, the MAC Learning functionality is now available as a native feature of the VMware Distributed Virtual Switch (VDS) and as some of you may have guessed from the title, promiscuous mode is also no longer a requirement for running Nested ESXi! I wanted to take a moment and thank Subin, Jobin, Sriram, Rajeev & Samuel from our Network and Security Business Unit (NSBU) at VMware who worked tirelessly to get this integrated and productized into ESXi. Not only will this benefit Nested ESXi workloads but also other solutions and use cases that have historically required the use of promiscuous mode. For customers who are still running ESXi 6.0 or 6.5, you should continue to use the Learnswitch Fling until you fully upgrade to vSphere 6.7.
To use the new MAC Learning functionality, you will of course need to upgrade to vSphere 6.7 (both vCenter and ESXi) but also upgrade to the latest VDS version which is 6.6. MAC Learning can be enabled on a per Distributed Virtual Portgroup bases and today, it is only available when using the vSphere API. For those that have used the VDS API to manage their VDS, you will simply use the existing ReconfigureDVPortgroup_Task() method and in 6.7, there now a new macManagementPolicy property which allows you to enable and define your MAC Learning settings. This new MAC Management Policy will also be the new preferred method for managing security policies going forward for a DV Portgroup and the previous security policy settings should no longer be used.
Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.
To demonstrate the new MAC Learning APIs, I have created two small PowerCLI functions called Get-MacLearn and Set-MacLearn which you can download from here. You will need to make sure to download the latest PowerCLI 10.1.0 release which adds support for vSphere 6.7
The Get-MacLearn function can be used to retrieve the current MAC Learning configuration for a given DV Portgroup, simple run the following command which can accept a one or more DV Portgroup names:
Get-MacLearn -DVPortgroupName @("Nested-01-DVPG")
As we can see from the output, I currently do not have MAC Learning enabled on this DV Portgroup. We can also see new properties such the limit which defines maximum number of MAC Addresses that can be learned (4096 max) and limitPolicy which defines the switching policy (drop or accept) when exceeding the learned MAC Address limit. As mentioned earlier, the new Mac Management interface should be used to manage security policies and as part of the output, I have also include both the new and legacy security policy settings.
The Set-MacLearn function can be used to enable MAC Learning as well as specifying the security policies for a given DV Portgroup. For Nested ESXi usage, you will want to set the following:
- MAC Learning: true
- Promiscuous mode: False
- Forged Transmit: True
- MAC Changes: False
- Limit: 4096 (optional, default is provided in the function)
- Limit Policy: Drop (optional, default is provided in function)
Set-MacLearn -DVPortgroupName @("Nested-01-DVPG") -EnableMacLearn $true -EnablePromiscuous $false -EnableForgedTransmit $true -EnableMacChange $false
At this point, you are now ready to start deploying your Nested ESXi workloads to this DV Portgroup or if you performed this operation on one of your existing DVPortgroup, you have now disabled the need for promiscuous mode!
Lastly, I wanted to share one additional tool that can be useful get more information about the current learned MAC Addresses which is only available directly on the ESXi Shell. The utility is called netdebg and below are a few examples on how to use it.
Note: Please note, this tool is meant for debugging purposes and there are no guarantees this will continue to work the same way in future releases.
To list all switches both VSS and VDS, run the following command:
netdbg vswitch instance list
To check whether a given powered on VM's DV Port has MAC Learning enabled, you can run the following and specify the DVPortID as well as the name of your VDS (which you need to use esxcfg-vswitch -l or esxcli network vswitch dvs vmware list to retrieve):
netdbg vswitch mac-learning port get -p 10 --dvs-alias VDS 6.7
To retrieve all learned MAC Addresses on a given DV Port,you can run the following and specify the DVPortID as well as the name of your VDS (which you need to use esxcfg-vswitch -l or esxcli network vswitch dvs vmware list to retrieve):
netdbg vswitch mac-table port get -p 10 --dvs-alias VDS 6.7
In the screenshot above, the first address (d5:d6) is actually a VM running on top of my Nested ESXi VM and the second address (5c:98) is my Nested ESXi VM's vmnic0. MAC Address entries will age out automatically between 10-20 minutes and no additional steps are required to clear out old learned entries.