mac learning

Native MAC Learning in vSphere 6.7 removes the need for Promiscuous mode for Nested ESXi

04.25.2018 by William Lam // 41 Comments

Over the years, several solutions have been developed here and here to help reduce the impact of promiscuous mode, which is a requirement for running Nested ESXi as a workload. Although these solutions worked extremely well, it however did require users to install additional software to enable this functionality. The most recent solution was a new Learnswitch VMkernel module (released as a VMware Fling) that enables MAC learning capabilities on ESXi.

Today, I am pleased to announce that with the release of vSphere 6.7, the MAC Learning functionality is now available as a native feature of the VMware Distributed Virtual Switch (VDS) and as some of you may have guessed from the title, promiscuous mode is also no longer a requirement for running Nested ESXi! I wanted to take a moment and thank Subin, Jobin, Sriram, Rajeev & Samuel from our Network and Security Business Unit (NSBU) at VMware who worked tirelessly to get this integrated and productized into ESXi. Not only will this benefit Nested ESXi workloads but also other solutions and use cases that have historically required the use of promiscuous mode. For customers who are still running ESXi 6.0 or 6.5, you should continue to use the Learnswitch Fling until you fully upgrade to vSphere 6.7.

To use the new MAC Learning functionality, you will of course need to upgrade to vSphere 6.7 (both vCenter and ESXi) but also upgrade to the latest VDS version which is 6.6. MAC Learning can be enabled on a per Distributed Virtual Portgroup bases and today, it is only available when using the vSphere API. For those that have used the VDS API to manage their VDS, you will simply use the existing ReconfigureDVPortgroup_Task() method and in 6.7, there now a new macManagementPolicy property which allows you to enable and define your MAC Learning settings. This new MAC Management Policy will also be the new preferred method for managing security policies going forward for a DV Portgroup and the previous security policy settings should no longer be used.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk. [Read more...]

ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

04.24.2017 by William Lam // 22 Comments

The ESXi MAC Learn dvFilter Fling was released a little over two years ago and it has become a must have when it comes to running our ESXi Hypervisor within a VM, also referred to as Nested ESXi. The reason this Fling has become such a popular hit amongst our customers and partners is that it greatly improves the performance when “Promiscuous Mode” is enabled on a Virtual or Distributed Virtual Portgroup, which is a requirement for using Nested ESXi. Although this Fling works great, there are a couple of limitations with this solution today. The first of which is called out in the original Fling release notes, that once a MAC Address has been learned, it never ages out which is not ideal for long running Nested ESXi environments that generates a large amount of new MAC Addresses. The second, is the lack of vMotion support where the learned MAC Address table is not transfered to the destination ESXi host and must be re-learned.

To help address both of these limitations, the folks over in the Network and Security Business Unit (NSBU) have been working hard to improve upon the existing solution and have developed a new native MAC Learning VMkernel module called the Learnswitch. This new Learnswitch not only helps improves Nested ESXi workloads but it can also potentially benefit other workloads such as Nested Containers or other 3rd Party network inspection software. One immediate difference from the previous MAC Learn dvFilter solution is that rather than operating on the Network IO Chain, the filtering is now performed within the outer virtual switch layer itself which will provide some additional performance gains. The other added benefit from an internal VMware standpoint is that the Learnswitch is now vmkapi compatible, which means we will have a better backwards compatible story for supporting old releases of ESXi. One downside to this new solution compared to the previous one is that because the dvFilter operated below the virtual switch layer, it could support both a Virtual Standard Switch as well as the Distributed Virtual Switch. With the new Learnswitch, a Distributed Virtual Switch will be required. If you currently do not meet the requirements of the new Learnswitch, you can continue using the dvFilter, but it is recommended that you do not mix both on a single system but you can definitely make use of both solutions across different ESXi hosts depending on the constraints of your environment.

Here are some of the new capabilities provided by the new Learnswitch module:

Overlay Network based that learning and filtering are done in Etherswitch forwarding check
MAC Address learning is based on VLAN ID or VXLAN ID on uplink and leaf port
Packet is filtered on uplink and leaf port if the MAC is learned on a different port
MAC Address table size is 32k per system
MAC Address aging support with default aging time of 5 minutes and configurable
Unknown unicast packet is flooded by default and configurable to drop
vMotion support that the MAC table learned on the port is transferred to destination host and RARP packet is sent
Standalone VMkernel module available as a VIB
net-learnswitch CLI to display MAC Address table, configuration and stats

[Read more...]

Does the ESXi Mac Learn dvFilter work with Nested ESXi on NSX VXLAN's?

09.19.2014 by William Lam // 3 Comments

After publishing my article on the new ESXi Mac Learn dvFilter which helps improve CPU/Network performance when using promiscuous mode with Nested ESXi, I received a couple of questions asking whether the dvFilter would work with NSX VXLAN's? At the time, I had only tested the Mac Learn dvFilter using standard VSS/VDS and not with any VXLAN based networks. I had reached out to a couple of folks asking whether this would work and to my surprise, I actually got back a mix set of answers to it will not work to it could work. One of the reasons that was given to me on why this may not work is that NSX-v (NSX for vSphere) leverages a different "virtual switch" than VSS/VDS and hence the Mac Learn dvFilter would not properly function. This actually would make sense, but because I received other responses negating that fact, I figured I probably should just test it for myself and see.

NSX 6.1 was recently released and I figured this would be a great opportunity for me to learn a bit more about NSX, as I have never played with it before and also test whether Mac Learn dvFilter would in fact work with NSX VXLAN's. In my lab environment I have deployed NSX and I have 3 physical ESXi hosts running VSAN (go SDS!). I deployed both an NSX ESR (Edge Service Router) hosting 2 Logical Networks (aka VXLAN segments) and an NSX DLR (Distributed Logical Router) hosting another 2 Logical Networks.

Here is a screenshot of the 4 Logical Networks, the first two on NSX ESR and the last two on NSX DLR:

Here is a screenshot of both the NSX ESR and DLR:

Note: If you would like to learn more about NSX ESR and DLR, check out this great article by Brad Hedlund who goes into more detail.

For my test, I first enabled Promiscuous Mode and Forged Transmit on the respective Logical Switches which is just a dvPortgroup on the VDS for my NSX ESR setup. I then had 2 Nested ESXi VMs running (without the Mac Learn dvFilter), a Windows "Jump Box" VM and vMA all connected to the same VLXAN network.

I then transfer an ISO from the Windows VM to vMA while running ESXTOP on the physical ESXi host which is hosting these four VMs. As I expected, both the Nested ESXi VMs and vMA were receiving network packets. Next, I installed the Mac Learn dvFilter VIB on the physical ESXi host and added the required VM Advanced Settings to both the Nested ESXi VMs and then re-ran the test. To my surprise, both the Nested ESXi VMs were no longer receiving the erroneous packets! So it seems that using VLXAN with NSX ESR, the Mac Learn dvFilter is working as expected.

To be thorough, I also ran through same test but now for the VXLAN segments backed by NSX DLR. This time, I was really surprised by the results. The test was prior to installing the Mac Learn dvFilter and my expectation was that the two Nested ESXi VMs would be seeing the duplicated network packets from the VDS, but to my surprise, they did not! Both the Nested ESXi VMs were pretty much idling at 0 packets as nothing was being sent to them. I am not exactly sure why I was seeing this behavior, perhaps there is some type of optimization in the DLR? This is something I hope to get an answer from someone in Engineering on why I might be seeing this positive behavior.

To summarize, this myth has been busted and the Mac Learn dvFilter does in fact work with VXLAN networks. If you are using NSX ESR for your VXLAN setup, then you will need to install the dvFilter and if you are using NSX DLR, it seems like you do not need to make any additional changes. After briefly speaking with Christian Dickmann, the creator of the dvFilter as I wanted to share the results with him, I also learned about some interesting tidbits. Christian was not surprised by the results actually, the reason for this is that the VMkernel networking stack was architected and designed to be modular. This meant that, one could switch out the "virtual switch" with other implementations and the underlying dvFilter framework would still continue to work regardless of the "virtual switch" being used.

Additional Note:

I did not get a chance to test with vCNS and VXLAN, but I believe it should work given NSX-v is functional. If you are able to test this, feel free to leave a comment on whether the expected behavior is seen with the Mac Learn dvFilter.
I did not get a chance to test this with vCloud Director with VXLAN based networks, but as I mentioned, this should work. Please leave a comment if you can confirm
I also noticed when creating the Logical Switches, there is a Mac Learning capability, but from my testing, I found it did not benefited Nested ESXi and the Mac Learn dvFilter was still required.