WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

New VMware Fling to improve Network/CPU performance when using Promiscuous Mode for Nested ESXi

08.28.2014 by William Lam // 44 Comments

I wrote an article awhile back Why is Promiscuous Mode & Forged Transmits required for Nested ESXi? and the primary motivation behind the article was in regards to an observation a customer made while using Nested ESXi. The customer was performing some networking benchmarks on their physical ESXi hosts which happened to be hosting a couple of Nested ESXi VMs as well as regular VMs. The customer concluded in his blog that running Nested ESXi VMs on their physical ESXi hosts actually reduced overall network throughput.

UPDATE (04/24/17) - Please have a look at the new ESXi Learnswitch which is an enhancement to the existing ESXi dvFilter MAC Learn module.

UPDATE (11/30/16) - A new version of the ESXi MAC Learning dvFilter has just been released to support ESXi 6.5, please download v2 for that ESXi release. If you have ESXi 5.x or 6.0, you will need to use the v1 version of the Fling as it is not backwards compat. You can all the details on the Fling page here.

This initially did not click until I started to think about this a bit more and the implications when enabling Promiscuous Mode which I think is something that not many of us are not aware of. At a very high level, Promiscuous Mode allows for proper networking connectivity for our Nested VMs running on top of a Nested ESXi VMs (For the full details, please refer to the blog article above). So why is this a problem and how does this lead to reduced network performance as well as increased CPU load?

The diagram below will hopefully help explain why. Here, I have a single physical ESXi host that is connected to either a VSS (Virtual Standard Switch) or VDS (vSphere Distributed Switch) and I have a portgroup which has Promiscuous Mode enabled and it contains both Nested ESXi VMs as well as regular VMs. Lets say we have 1000 Network Packets destined for our regular VM (highlighted in blue), one would expect that the red boxes (representing the packets) will be forwarded to our regular VM right?

nested-esxi-prom-new-01
What actually happens is shown in the next diagram below where every Nested ESXi VM as well as other regular VMs within the portgroup that has Promiscuous Mode enabled will receive a copy of those 1000 Network Packets on each of their vNICs even though they were not originally intended for them. This process of performing the shadow copies of the network packets and forwarding them down to the VMs is a very expensive operation. This is why the customer was seeing reduced network performance as well as increased CPU utilization to process all these additional packets that would eventually be discarded by the Nested ESXi VMs.

nested-esxi-prom-new-02
This really solidified in my head when I logged into my own home lab system which I run anywhere from 15-20 Nested ESXi VMs at any given time in addition to several dozen regular VMs just like any home/development/test lab would. I launched esxtop and set the refresh cycle to 2seconds and switched to the networking view. At the time I was transferring a couple of ESXi ISO’s for my kicskstart server and realized that ALL my Nested ESXi VMs got a copy of those packets.

nested-esxi-mac-learning-dvfilter-0
As you can see from the screenshot above, every single one of my Nested ESXi VMs was receiving ALL traffic from the virtual switch, this definitely adds up to a lot of resources being wasted on my physical ESXi host which could be used for running other workloads.

I decided at this point to reach out to engineering to see if there was anything we could do to help reduce this impact. I initially thought about using NIOC but then realized it was primarily designed for managing outbound traffic where as the Promiscuous Mode traffic is all inbound and it would not actually get rid of the traffic. After speaking to a couple of Engineers, it turns out this issue had been seen in our R&D Cloud (Nimbus) which provides IaaS capabilities to the R&D Organization for quickly spinning up both Virtual/Physical instances for development and testing.

Christian Dickmann was my go to guy for Nimbus and it turns out this particular issue has been seen before. Not only has he seen this behavior, he also had a nice solution to fix the problem in the form of an ESXi dvFilter that implemented MAC Learning! As many of you know our VSS/VDS does not implement MAC Learning as we already know which MAC Addresses are assigned to a particular VM.

I got in touch with Christian and was able to validate his solution in my home lab using the latest ESXi 5.5 release. At this point, I knew I had to get this out to the larger VMware Community and started to work with Christian and our VMware Flings team to see how we can get this released as a Fling.

Today, I am excited to announce the ESXi Mac Learning dvFilter Fling which is distributed as an installable VIB for your physical ESXi host and it provides support for ESXi 5.x & ESXi 6.x

esxi-mac-learn-dvfilter-fling-logo
Note: You will need to enable Promiscuous Mode either on the VSS/VDS or specific portgroup/distributed portgroup for this solution to work.

You can download the MAC Learning dvFilter VIB here or you can install directly from the URL shown below:

To install the VIB, run the following ESXCLI command if you have VIB uploaded to your ESXi datastore:

esxcli software vib install -v /vmfs/volumes/<DATASTORE>/vmware-esx-dvfilter-maclearn-0.1-ESX-5.0.vib -f

To install the VIB from the URL directly, run the following ESXCLI command:

esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi-mac-learning-dvfilter/vmware-esx-dvfilter-maclearn-1.0.vib -f

A system reboot is not necessary and you can confirm the dvFilter was successfully installed by running the following command:

/sbin/summarize-dvfilter

You should be able see the new MAC Learning dvFilter listed at the very top of the output.

nested-esxi-mac-learning-dvfilter-2
For the new dvFilter to work, you will need to add two Advanced Virtual Machine Settings to each of your Nested ESXi VMs and this is on a per vNIC basis, which means you will need to add N-entries if you have N-vNICs on your Nested ESXi VM.

    ethernet#.filter4.name = dvfilter-maclearn
    ethernet#.filter4.onFailure = failOpen

This can be done online without rebooting the Nested ESXi VMs if you leverage the vSphere API. Another way to add this is to shutdown your Nested ESXi VM and use either the “legacy” vSphere C# Client or vSphere Web Client or for those that know how to append and reload the .VMX file as that’s where the configuration file is persisted
on disk.

nested-esxi-mac-learning-dvfilter-3
I normally provision my Nested ESXi VMs with 4 vNICs, so I have four corresponding entries. To confirm the settings are loaded, we can re-run the summarize-dvfilter command and we should now see our Virtual Machine listed in the output along with each vNIC instance.

nested-esxi-mac-learning-dvfilter-4
Once I started to apply this changed across all my Nested ESXi VMs using a script I had written for setting Advanced VM Settings, I immediately saw the decrease of network traffic on ALL my Nested ESXi VMs. For those of you who wish to automate this configuration change, you can take a look at this blog article which includes both a PowerCLI & vSphere SDK for Perl script that can help.

I highly recommend anyone that uses Nested ESXi to ensure you have this VIB installed on all your ESXi hosts! As a best practice you should also ensure that you isolate your other workloads from your Nested ESXi VMs and this will allow you to limit which portgroups must be enabled with Promiscuous Mode.

Categories // ESXi, Home Lab, Nested Virtualization, vSphere, vSphere 6.0 Tags // dvFilter, ESXi, fling, mac learning, nested, nested virtualization, promiscuous mode, vib

Quick Tip - Listing Image Profiles From an ESXi Patch Using ESXCLI

06.21.2013 by William Lam // 6 Comments

I was cleaning out a few of my to-do items (list just keeps getting longer everyday) this morning and there was a question that I received a few weeks back asking how to retrieve the list of Image Profiles for a given ESXi patch. This is actually quite easy and you will want to use ESXCLI.

Note: The examples shown below is using ESXCLI on the ESXi Shell, but these commands can be execute remotely as well using ESXCLI or through PowerCLI with Get-EsxCli cmdlet.

To list the available Image Profiles for an ESXi patch, run the following command (ensure you substitute the full path to your ESXi patch):

esxcli software sources profile list -d /vmfs/volumes/datastore1/ESXi510-201212001.zip

To get more details on a particular Image Profile, run the following command and specify the -p for the specific Image Profile:

esxcli software sources profile get -d /vmfs/volumes/datastore1/ESXi510-201212001.zip -p ESXi-5.1.0-20121204001-no-tools

To install/update a specific Image Profile, run the following command with the Image Profile name:

esxcli software profile update -d /vmfs/volumes/datastore1/ESXi510-201212001.zip -p ESXi-5.1.0-20121204001-no-tools

If you just want to install the ESXi patch, run the following command which will install the esx-base Image Profile by default which will include everything:

esxcli software vib update -d /vmfs/volumes/datastore1/ESXi510-201212001.zip

To check for the Image Profile you have installed on your ESXi host, run the following command:

esxcli software profile get

Here are some additional resources for ESXi patch management that may also be useful:

  • Quickest Way to Patch an ESX/ESXi Using the Command-line
  • Update a Host with Image Profiles
  • Understanding ESXi Patches - Size & Patch Bundles
  • A Pretty Cool Method of Upgrading to ESXi 5.1

Categories // Uncategorized Tags // esxcli, ESXi, image profile, vib

ESXi Google Authenticator Is Now A VMware Fling!

06.02.2013 by William Lam // 6 Comments

Earlier this year I wrote an article about using Google's Authenticator application to provide 2-Factor Authentication for connecting to ESXi using either the ESXi Shell locally or remotely over SSH. I also documented the process for compiling and building your own custom ESXi VIB with the help of two VMware engineers (Hongkun Xi & Jian Ouyang). Though the process was not terribly difficult, it did require minor source code modification and building a custom ESXi VIB. This also meant that you were required to lower the security acceptance of your ESXi host to community supported which is not a recommended practice. In addition, the custom ESXi VIB only supported a single administrator account which was root and additional work was required to support multiple administrators.

Well it turns out that both Hongkun and Jian have been quite busy enhancing this project in their spare time and have just released an ESXi Google Authenticator Fling! The Fling is distributed as a custom ESXi VIB which is signed by VMware, so you no longer have to lower the security of your ESXi host. It supports both ESXi 5.0 and 5.1 and it allows for multiple administrators to login using Google Authenticator.

Here is a list of the features that are supported:

  • Two-Factor Authentication for ESXi Shell and SSH access
  • Supports multiple administrators login on esx5.1, and single admin (root) on esx5.0
  • Support for 30-second TOTP codes
  • Support for emergency scratch codes
  • Protection against replay attacks

To learn more about the Fling and instructions on setting up the ESXi Google Authenticator, be sure to visit the VMware Lab's site.

If you have any feedback or questions, be sure to leave a comment on the Fling's web page here.

Categories // Uncategorized Tags // 2FA, ESXi, google authenticator, notsupported, pam, ssh, two factor, vib

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...