Over the years, several solutions have been developed here and here to help reduce the impact of promiscuous mode, which is a requirement for running Nested ESXi as a workload. Although these solutions worked extremely well, it however did require users to install additional software to enable this functionality. The most recent solution was a new Learnswitch VMkernel module (released as a VMware Fling) that enables MAC learning capabilities on ESXi.
Today, I am pleased to announce that with the release of vSphere 6.7, the MAC Learning functionality is now available as a native feature of the VMware Distributed Virtual Switch (VDS) and as some of you may have guessed from the title, promiscuous mode is also no longer a requirement for running Nested ESXi! I wanted to take a moment and thank Subin, Jobin, Sriram, Rajeev & Samuel from our Network and Security Business Unit (NSBU) at VMware who worked tirelessly to get this integrated and productized into ESXi. Not only will this benefit Nested ESXi workloads but also other solutions and use cases that have historically required the use of promiscuous mode. For customers who are still running ESXi 6.0 or 6.5, you should continue to use the Learnswitch Fling until you fully upgrade to vSphere 6.7.
To use the new MAC Learning functionality, you will of course need to upgrade to vSphere 6.7 (both vCenter and ESXi) but also upgrade to the latest VDS version which is 6.6. MAC Learning can be enabled on a per Distributed Virtual Portgroup bases and today, it is only available when using the vSphere API. For those that have used the VDS API to manage their VDS, you will simply use the existing ReconfigureDVPortgroup_Task() method and in 6.7, there now a new macManagementPolicy property which allows you to enable and define your MAC Learning settings. This new MAC Management Policy will also be the new preferred method for managing security policies going forward for a DV Portgroup and the previous security policy settings should no longer be used.
Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.
To demonstrate the new MAC Learning APIs, I have created two small PowerCLI functions called Get-MacLearn and Set-MacLearn which you can download from here. You will need to make sure to download the latest PowerCLI 10.1.0 release which adds support for vSphere 6.7
The Get-MacLearn function can be used to retrieve the current MAC Learning configuration for a given DV Portgroup, simple run the following command which can accept a one or more DV Portgroup names:
Get-MacLearn -DVPortgroupName @("Nested-01-DVPG")
As we can see from the output, I currently do not have MAC Learning enabled on this DV Portgroup. We can also see new properties such the limit which defines maximum number of MAC Addresses that can be learned (4096 max) and limitPolicy which defines the switching policy (drop or accept) when exceeding the learned MAC Address limit. As mentioned earlier, the new Mac Management interface should be used to manage security policies and as part of the output, I have also include both the new and legacy security policy settings.
The Set-MacLearn function can be used to enable MAC Learning as well as specifying the security policies for a given DV Portgroup. For Nested ESXi usage, you will want to set the following:
- MAC Learning: true
- Promiscuous mode: False
- Forged Transmit: True
- MAC Changes: False
- Limit: 4096 (optional, default is provided in the function)
- Limit Policy: Drop (optional, default is provided in function)
Set-MacLearn -DVPortgroupName @("Nested-01-DVPG") -EnableMacLearn $true -EnablePromiscuous $false -EnableForgedTransmit $true -EnableMacChange $false
Once the reconfiguration has completed, we can re-run the Get-MacLearn function to confirm our changes as shown in the screenshot below:
At this point, you are now ready to start deploying your Nested ESXi workloads to this DV Portgroup or if you performed this operation on one of your existing DVPortgroup, you have now disabled the need for promiscuous mode!
Lastly, I wanted to share one additional tool that can be useful get more information about the current learned MAC Addresses which is only available directly on the ESXi Shell. The utility is called netdebg and below are a few examples on how to use it.
Note: Please note, this tool is meant for debugging purposes and there are no guarantees this will continue to work the same way in future releases.
To list all switches both VSS and VDS, run the following command:
netdbg vswitch instance list
To check whether a given powered on VM's DV Port has MAC Learning enabled, you can run the following and specify the DVPortID as well as the name of your VDS (which you need to use esxcfg-vswitch -l or esxcli network vswitch dvs vmware list to retrieve):
netdbg vswitch mac-learning port get -p 10 --dvs-alias VDS 6.7
To retrieve all learned MAC Addresses on a given DV Port,you can run the following and specify the DVPortID as well as the name of your VDS (which you need to use esxcfg-vswitch -l or esxcli network vswitch dvs vmware list to retrieve):
netdbg vswitch mac-table port get -p 10 --dvs-alias VDS 6.7
In the screenshot above, the first address (d5:d6) is actually a VM running on top of my Nested ESXi VM and the second address (5c:98) is my Nested ESXi VM's vmnic0. MAC Address entries will age out automatically between 10-20 minutes and no additional steps are required to clear out old learned entries.
Thanks for the update William - this is great news. Looking forward to trying it out in my lab.
Do you know how MAC moves will be handled? If a guest VM VMotions from one nested ESXi to another, will the learned MAC be moved to the new port or will it exist in both places until the original one times out.
James,
Yes, here's what I received back from Engineering:
1) If the move is from one nested ESX VM to another on the same host, the mac will move from older vNIC port to the new vNIC port based on packets coming from new nested ESX VM.
2) If the move is from one nested ESX VM to another on a different host, the mac will move from vNIC port to uplink port on source outer ESX host based on packets coming from new nested ESX VM. It will also get learned on the vNIC of the destination outer ESX host.
A learned MAC (i.e. MAC address + VLAN) won’t exist in two ports at the same time for the same host switch.
Hello William,
I have deployed Nested vCenter and ESXi in Cloud Director organization. Underlay there is vDSwitch with organization porgroup. I have enabled MAC learning policy on organization vDS portgroups and communication is working as expected. However, when I perform vMotion of guest VM from nested ESXi to another nested ESXi, guest VM lose connection (even NSX Edge gateway cannot be pinged). I left this overnight and VM regained connection after 6 hours.
Is there a way to make sure connection is not lost during vMotion or at least to manually reset the MAC learning process ?
I resolved issue by additionally enabling on nested ESXI portgroups Forged transmits and Mac changes. Now vMotion between nested ESXis works as expected. Hope this helps someone 🙂
Thanks William - much appreciated.
Thanks William, Does the underlying Physical host need to be 6.7 as well as the Nested Lab or is it just the Nested Lab, If I leave my Underlying physical host as 6.5 can I make use of MacLearn ?
Just like past solutions, the settings are ONLY applied to the physical ESXi host 🙂 If you're still on 6.5, then you'll need to continue using the MAC Learn Fling
Can we still use the old macdvFilter with vSphere v6.7 if we're not using a vDS?
I did a --dry-run on upgrading 6.5 to 6.7 and it failed with a dependency error on maclearn. I want to keep using it because I run an OpenVPN server as a VM, and bridge mode requires promiscuous mode. I have the essentials bundle, so I can't use a VDS. I'm debating on whether or not to actually force the upgrade (-f option) to see if it will actually work on 6.7.
William, can you comment on this?
Novice question: Do all ESXi hosts, both inner (nested) and outer (real) need to be added to this dvswitch? Or is it sufficient to have only the outer host(s) joined to this dvswitch while the inner hosts are simply nodes on said switch? Hope that made sense...
On which VDS portgroup(s) the feature has to be enabled, inside/outside/all?
I am getting following error message running Set-MacLearn function:
PS C:\Users\xxyy> Set-MacLearn -DVPortgroupName HomeDSwitch2-DVUplinks-833 -EnableMacLearn $true -EnablePromiscuous $false -EnableForgedTransmit $true -EnableMacChange $false
Enabling MAC Learning on DVPortgroup: HomeDSwitch2-DVUplinks-833 ...
Wait-Task : 01.09.2018 16:17:42 Wait-Task The operation for the entity "HomeDSwitch2-DVUplinks-833" failed with the following message: "A specified parameter was not correct:
macManagementPolicy". com.vmware.vim.vpxd.dvs.NotPermittedOnUplinkPortgroup.label
In D:\Private\VMWareBakup\MacLearn.ps1:237 Zeichen:26
+ $task1 | Wait-Task | Out-Null
+ ~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Wait-Task], InvalidArgument
+ FullyQualifiedErrorId : Client20_TaskServiceImpl_CheckServerSideTaskUpdates_OperationFailed,VMware.VimAutomation.ViCore.Cmdlets.Commands.WaitTask
Hi William
when i set
LegacyAllowPromiscuous to false
LegacyForgedTransmits to false
it is not working ?
Means that that the Classic Promiscuous and ForgedTransmits muststay enabled ?
Same question here! Do I need to enable LegacyAllowPromiscuous and LegacyForgedTransmits in order to get MAC learning working?
Thanks!
Same Question here!
Must LegacyAllowPromiscuous and LegacyForgedTransmits be enabled ?
No, they're not applicable when using MAC Learning 🙂
Hi William,
So how would we do this on a VSS (Free edition ESXi 6.7)?
I have encountered the same problem. It seems that the documentation does not reflect how esxi 6.7 should enable mac learning in VSS.
Do you have any new news?
I need to enable mac learning in VSS.
Hi,
I have seen strange behavior. If I enable MAC learning on portgroup where vmkernel NIC (on nested) ESXi level is connected, I can no longer access suh vmkernel interface.
BUT if there is a VM aswell (succesfully learned MAC address) then even vmkernel NIC is avaliable
[root@ns3016814:~] netdbg vswitch mac-table port get -p 102 --dvs-alias vDS-LAB
MAC: 00:0c:29:a3:c2:00 vid: 0 vni: 0 type: learned aging: yes elapsed: 2
MAC: 00:0c:29:ed:a9:37 vid: 0 vni: 0 type: static aging: yes elapsed: 0
00:0c:29:a3:c2:00 is a VM running on the top of nested ESXi
[root@ns3016814:~] netdbg vswitch mac-table port get -p 101 --dvs-alias vDS-LAB
MAC: 00:0c:29:ed:a9:2d vid: 0 vni: 0 type: static aging: yes elapsed: 0
both ports (101 and 102) are connected to vswith0 on the nested ESXi and vmk0 with IP address is configured (both uplinks are active)
if I reconfigure vswitch0 (on nested esxi) so only uplink connected to port 102 is avaliable, then the vmekrnel nic IP address starts to respond (although it won't be seen in netdbb vswitch mac-table). If I choose to use port 101 and 102 is disconnected, no traffic pass.
Any advice here?
Hello Martinb,
same problem here.. have you been able to solve ?
Thanks!
Solved it.
The MAC address of the vmk management interface of the nested ESXi will be equal to the "physical" MAC address of the VMXNET3 interface. This, for some reason, does not work.
Edit the /etc/vmware/esx.conf file with vi and change the vmk management interface MAC to something else and it will work.
Props on the solution. I was getting irked that I couldn't get my nested ESXi hosts converted over to a dvSwitch or use redundant uplinks for even a standard vSwitch (in my use case I'm building a hands on lab and trying to very closely mimic the hardware solution the lab is training for). Editing the MAC address of vmk0 to break the link to the vmnic0 mac address did the trick.
Hello Christopher, I may be in a similar situation. I am having problem moving my vmk0 (default is on VSS 0) on my nested ESX over to vds. My nested ESX are connect to a trunk portgroup on the physical ESX vds with MacLearn enable. I create a brand new vds with three portgroups - mgmt, trunk, and VM within my nested ESX environment. There are no vlan tagging on mgmt, but is set to trunk on the trunk portgroup. The error I am getting is "throwable .proxy", not sure what that mean. I am trying to get a better understand as to which MAC address are you editing. Would you elaborate? Are you editing the MAC address on vmnic0 within the nested ESX as I was not aware there is a MAC address you can set specifically for vmk0. Also, what are you setting the MAC to?
This is awesome!!!!! Great article!! I'm new to powershell so it took me couple day to figure it out.
After enabling mac-learning, will the mac addresses learned from nested devices (VMs) also show up on my Northbound switches (mac/arp tables)?
I meant will the learned mac addresses show up on northbound physical switches that are used for the VDS uplinks?
I've enabled MAC Learning and worked... for a minute or two. Now even after disabling it the nested ESXi connected to the Distributed Portgroup won't work at all. I have a DVS 6.6 with a Trunk Portgroup that had Promiscuous mode enabled. After enabling MacLearning it worked for a minute or two, I even saw some errors fixed (no duplicate packets now), but after a while everything disconnected and it just wont work :S
I even created a new Portgroup and migrated a host there to try with Promiscuous but without MacLearning (as before) and it doesn't work anymore!
I don't know what to do...
Hey thanks, I really appreciate your time in creating the maclearn.ps1 function, a real time saver in getting this going.
I have an NSX-V question related to the maclearn feature... Do you know if we ONLY need to enable the VXLAN portgroups with the maclearn function, or do we ALSO need to enable maclearn on the VTEP portgroup as well? My thought is the VTEP portgroup doesn't matter, because the maclearn is done on the VXLAN portgroup level. I am only using single VTEP per host with single uplink4, and other uplinks are all standby...
Other than that question, I feel like it is functioning fine on 200 VXLAN portgroups (each having 4 virtual ESXi hosts as labs on them)... Right when I ran your script to enable, the hosts started responding to ping without promiscuous mode, so it looks good.
Does not work in my case from Ubuntu Server 18.04 with PowerShell Core and PowerCLI Module installed. I can execute the script without errors but it does not give any output. Simply nothing. Double checked the portgroup name.
Hi William,
I am using your Automated Lab Deployment PowerShell script with your provided ESXi.6.7 OVA's and I am wondering if you could advise how to amend the deployment (I am guessing in the OVA post deploy script) so that the vmnic0 MAC address is NOT inherited by VMK0. Can you randomise the VMK0 MAC address as part of the ESXi host finalisation?
Thanks,
Pin
Hi William,
Thanks for the great article - I work in a vCD Lab based role so have always followed your Nested Lab posts with interest.
We have a dvs that uses vxlan and is just used for Nested ESXi's for labs - before 6.7 we use to set the default Port Security policies to Accept that way any new port group (virtual wire) created always has security set to Accept and promiscuous enabled - we used the Set-SecurityPolicy
is it possible in vxlan style enviroment to set a default so all new virtual wires would have the mac learn feature enabled by default already.
We use to set non standard defaults using the snippet below that was online - I suspect there might be a
DefaultPortConfig.SecurityPolicy.MacLearn or something similar - I'll test when i get a chance and let you know
$dvSwitch = Get-VDSwitch $switchname
$spec = New-Object VMware.Vim.VMwareDVSConfigSpec
$spec.configVersion = $dvswitch.ExtensionData.Config.ConfigVersion
$spec.DefaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting
$spec.DefaultPortConfig.SecurityPolicy = New-Object VMware.Vim.DVSSecurityPolicy
$spec.DefaultPortConfig.SecurityPolicy.AllowPromiscuous = New-Object VMware.Vim.BoolPolicy
$spec.DefaultPortConfig.SecurityPolicy.AllowPromiscuous.Value = $promiscuous
$spec.DefaultPortConfig.SecurityPolicy.MacChanges = New-Object VMware.Vim.BoolPolicy
$spec.DefaultPortConfig.SecurityPolicy.MacChanges.Value = $macChanges
$spec.DefaultPortConfig.SecurityPolicy.ForgedTransmits = New-Object VMware.Vim.BoolPolicy
$spec.DefaultPortConfig.SecurityPolicy.ForgedTransmits.Value = $forgedTransmits
$dvswitch.ExtensionData.ReconfigureDvs_Task($spec)
Doesn't seem like a big deal to me to implement MAC learning into vSphere client GUI when command line function exists. So what's the abvious reason this still was not implemented into vSphere 7? Nested VMs are definitive used in production environments.
It's not working anymore with vSphere 7:
"Unable to find DVPortgroup or VDS is not running 6.6.0"
Unfortunately this does no longer work on ESXi 7.0:
netdbg vswitch mac-learning get -dvs DSwitch-NSXt
Traceback (most recent call last):
File "/bin/netdbg", line 32, in
RootCommandGroup()
File "/lib64/python3.5/click/core.py", line 611, in __call__
return self.main(*args, **kwargs)
File "/lib64/python3.5/click/core.py", line 591, in main
rv = self.invoke(ctx)
File "/lib64/python3.5/click/core.py", line 938, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/lib64/python3.5/click/core.py", line 938, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/lib64/python3.5/click/core.py", line 938, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/lib64/python3.5/click/core.py", line 784, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/lib64/python3.5/click/core.py", line 417, in invoke
return callback(*args, **kwargs)
File "/build/mts/release/bora-15843807/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/netdbg/vswitch/mac_learning.py", line 120, in MACLearningGetCommand
File "/lib64/python3.5/site-packages/net/lib/libvswitch.py", line 5251, in GetMACLearning
raise DvsFailure('Get global MAC learning config', dvs_alias, status)
net.lib.exceptions.DvsFailure: DVS Failure DSwitch-NSXt:195887107::Get global MAC learning config.
Seems like you have to still run Promiscuous mode with ESXi 7.0 OR deploy NSX-T and then configure the Segment profiles like William mentions.
I am trying to deploy your ESX6.7 on a single ESX host with its own VCSA and VDS switch. Regarding to enable MacLearn on the portgroup with your "Get-MacLearn and Set-MacLearn" ps1, do I set this on the physical VDS switch (the physical ESX host), or nested / virtual VDS switch (within the nested ESX environment)?
After enabling this on a vSphere 7.0U1 host my nested VMs become inaccessible. If I recreate the portgroups, with the older settings they come back online. Any ideas?
https://communities.vmware.com/t5/Nested-Virtualization/vSphere-7-0U1-Nested-with-Mac-Learn/m-p/2813860#M3073
Hi
i tried to run script, but i receive an error:
New-Object : Cannot find type [VMware.Vim.DVSMacManagementPolicy]: verify that
the assembly containing this type is loaded.
At line:50 char:31
+ ... $macMmgtSetting = New-Object VMware.Vim.DVSMacManagementPolicy
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidType: (:) [New-Object], PSArgumentExcepti
on
+ FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewOb
jectCommand
this is the version that i installed with command:
Install-Module -Name VMware.PowerCLI -RequiredVersion 10.1.0.8403314
Can you help me please?
thanks
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 6.5.4.7... VMware.VimAutomation.HA Get-DrmInfo
Directory: C:\Program Files\WindowsPowerShell\Modules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 12.2.0.... VMware.CloudServices {Connect-Vcs, Disconnect-Vcs, Get-VcsOrganizationRole, Get..
Script 7.0.0.1... VMware.DeployAutomation {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-D..
Script 7.0.0.1... VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-Esx..
Manifest 10.1.0.... VMware.PowerCLI
Script 7.0.1.1... VMware.Vim
Script 12.2.0.... VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Script 12.0.0.... VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, G..
Script 12.2.0.... VMware.VimAutomation.Common {Get-PowerCLIContext, Get-Task, New-OAuthSecurityContext, ..
Script 12.2.0.... VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAd..
Script 7.9.0.1... VMware.VimAutomation.HorizonView {Connect-HVServer, Disconnect-HVServer}
Script 12.0.0.... VMware.VimAutomation.License Get-LicenseDataManager
Script 12.2.0.... VMware.VimAutomation.Nsxt {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtGlobal..
Script 10.0.0.... VMware.VimAutomation.PCloud {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInsta..
Script 12.2.0.... VMware.VimAutomation.Sdk Get-ErrorReport
Script 12.2.0.... VMware.VimAutomation.Srm {Connect-SrmServer, Disconnect-SrmServer}
Script 12.2.0.... VMware.VimAutomation.Storage {Add-EntityDefaultKeyProvider, Add-KeyManagementServer, Ad..
Script 1.6.0.0 VMware.VimAutomation.StorageUtility Update-VmfsDatastore
Script 12.2.0.... VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, E..
Script 12.2.0.... VMware.VimAutomation.Vmc {Add-VmcSddcHost, Connect-Vmc, Disconnect-Vmc, Get-AwsAcco..
Script 12.2.0.... VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-O..
Script 6.5.1.7... VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Complia..
Hi William,
I tried to run the two script (Get and Set MacLearn) without success....
I've VCSA and ESXi 6.7 U3. I created a VDS 6.6 with name DS-VMW-ESX and a portgroup with name "MGMT-vMotion" but when run Get-MacLear I don't receive any output...why?
PS E:\> .\Get-MacLearn.ps1 -DVPortgroupName @("MGMT-vMotion")
PS E:\>
If run cmdlet Get-VDPortGroup its works fine and name is correct.
Can you help me..please ?
Thanks
If I define the $ DVPortgroupName variable and just do the foreach, it works and returns me the values.
I believe the problem is in the definition of the $ DVPortgroupName variable not being passed to the function correctly.
For others it works ... so what's wrong?