Fling – Page 10

New VMware Fling to improve Network/CPU performance when using Promiscuous Mode for Nested ESXi

08.28.2014 by William Lam // 44 Comments

I wrote an article awhile back Why is Promiscuous Mode & Forged Transmits required for Nested ESXi? and the primary motivation behind the article was in regards to an observation a customer made while using Nested ESXi. The customer was performing some networking benchmarks on their physical ESXi hosts which happened to be hosting a couple of Nested ESXi VMs as well as regular VMs. The customer concluded in his blog that running Nested ESXi VMs on their physical ESXi hosts actually reduced overall network throughput.

UPDATE (04/24/17) - Please have a look at the new ESXi Learnswitch which is an enhancement to the existing ESXi dvFilter MAC Learn module.

UPDATE (11/30/16) - A new version of the ESXi MAC Learning dvFilter has just been released to support ESXi 6.5, please download v2 for that ESXi release. If you have ESXi 5.x or 6.0, you will need to use the v1 version of the Fling as it is not backwards compat. You can all the details on the Fling page here.

This initially did not click until I started to think about this a bit more and the implications when enabling Promiscuous Mode which I think is something that not many of us are not aware of. At a very high level, Promiscuous Mode allows for proper networking connectivity for our Nested VMs running on top of a Nested ESXi VMs (For the full details, please refer to the blog article above). So why is this a problem and how does this lead to reduced network performance as well as increased CPU load?

The diagram below will hopefully help explain why. Here, I have a single physical ESXi host that is connected to either a VSS (Virtual Standard Switch) or VDS (vSphere Distributed Switch) and I have a portgroup which has Promiscuous Mode enabled and it contains both Nested ESXi VMs as well as regular VMs. Lets say we have 1000 Network Packets destined for our regular VM (highlighted in blue), one would expect that the red boxes (representing the packets) will be forwarded to our regular VM right?

What actually happens is shown in the next diagram below where every Nested ESXi VM as well as other regular VMs within the portgroup that has Promiscuous Mode enabled will receive a copy of those 1000 Network Packets on each of their vNICs even though they were not originally intended for them. This process of performing the shadow copies of the network packets and forwarding them down to the VMs is a very expensive operation. This is why the customer was seeing reduced network performance as well as increased CPU utilization to process all these additional packets that would eventually be discarded by the Nested ESXi VMs.

This really solidified in my head when I logged into my own home lab system which I run anywhere from 15-20 Nested ESXi VMs at any given time in addition to several dozen regular VMs just like any home/development/test lab would. I launched esxtop and set the refresh cycle to 2seconds and switched to the networking view. At the time I was transferring a couple of ESXi ISO’s for my kicskstart server and realized that ALL my Nested ESXi VMs got a copy of those packets.

As you can see from the screenshot above, every single one of my Nested ESXi VMs was receiving ALL traffic from the virtual switch, this definitely adds up to a lot of resources being wasted on my physical ESXi host which could be used for running other workloads.

I decided at this point to reach out to engineering to see if there was anything we could do to help reduce this impact. I initially thought about using NIOC but then realized it was primarily designed for managing outbound traffic where as the Promiscuous Mode traffic is all inbound and it would not actually get rid of the traffic. After speaking to a couple of Engineers, it turns out this issue had been seen in our R&D Cloud (Nimbus) which provides IaaS capabilities to the R&D Organization for quickly spinning up both Virtual/Physical instances for development and testing.

Christian Dickmann was my go to guy for Nimbus and it turns out this particular issue has been seen before. Not only has he seen this behavior, he also had a nice solution to fix the problem in the form of an ESXi dvFilter that implemented MAC Learning! As many of you know our VSS/VDS does not implement MAC Learning as we already know which MAC Addresses are assigned to a particular VM.

I got in touch with Christian and was able to validate his solution in my home lab using the latest ESXi 5.5 release. At this point, I knew I had to get this out to the larger VMware Community and started to work with Christian and our VMware Flings team to see how we can get this released as a Fling.

Today, I am excited to announce the ESXi Mac Learning dvFilter Fling which is distributed as an installable VIB for your physical ESXi host and it provides support for ESXi 5.x & ESXi 6.x

Note: You will need to enable Promiscuous Mode either on the VSS/VDS or specific portgroup/distributed portgroup for this solution to work.

You can download the MAC Learning dvFilter VIB here or you can install directly from the URL shown below:

To install the VIB, run the following ESXCLI command if you have VIB uploaded to your ESXi datastore:

esxcli software vib install -v /vmfs/volumes/<DATASTORE>/vmware-esx-dvfilter-maclearn-0.1-ESX-5.0.vib -f

To install the VIB from the URL directly, run the following ESXCLI command:

esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi-mac-learning-dvfilter/vmware-esx-dvfilter-maclearn-1.0.vib -f

A system reboot is not necessary and you can confirm the dvFilter was successfully installed by running the following command:

/sbin/summarize-dvfilter

You should be able see the new MAC Learning dvFilter listed at the very top of the output.

For the new dvFilter to work, you will need to add two Advanced Virtual Machine Settings to each of your Nested ESXi VMs and this is on a per vNIC basis, which means you will need to add N-entries if you have N-vNICs on your Nested ESXi VM.

ethernet#.filter4.name = dvfilter-maclearn
ethernet#.filter4.onFailure = failOpen

This can be done online without rebooting the Nested ESXi VMs if you leverage the vSphere API. Another way to add this is to shutdown your Nested ESXi VM and use either the “legacy” vSphere C# Client or vSphere Web Client or for those that know how to append and reload the .VMX file as that’s where the configuration file is persisted
on disk.

I normally provision my Nested ESXi VMs with 4 vNICs, so I have four corresponding entries. To confirm the settings are loaded, we can re-run the summarize-dvfilter command and we should now see our Virtual Machine listed in the output along with each vNIC instance.

Once I started to apply this changed across all my Nested ESXi VMs using a script I had written for setting Advanced VM Settings, I immediately saw the decrease of network traffic on ALL my Nested ESXi VMs. For those of you who wish to automate this configuration change, you can take a look at this blog article which includes both a PowerCLI & vSphere SDK for Perl script that can help.

I highly recommend anyone that uses Nested ESXi to ensure you have this VIB installed on all your ESXi hosts! As a best practice you should also ensure that you isolate your other workloads from your Nested ESXi VMs and this will allow you to limit which portgroups must be enabled with Promiscuous Mode.

pyvmomi (vSphere SDK for Python) 5.5.0-2014.1 released!

08.15.2014 by William Lam // 1 Comment

The 5.5.0-2014.1 release of @pyvmomi is now available https://t.co/deHgZviLN1

— Shawn Hartsock ☁️ (@hartsock) August 15, 2014

I just saw an awesome update from Shawn Hartsock, a fellow VMware colleague. For those of you who do not know him, Shawn works in our Ecosystems and Solutions Engineering (EASE) organization and is the primary maintainer of VMware's pyvmomi (vSphere SDK for Python) open-source project. The pyvmomi project was open sourced since last December which I had written about here, it has received over 3K+ downloads and has a very active community. Much of this success has been due to the hard from Shawn fostering an active community around pyvmomi.

The announcement today from Shawn is a new release of pyvmomi at version 5.5.0-2014.1:

As mentioned earlier, the pyvmomi project is a very active project and Shawn is constantly engaging with users looking for feedback, suggestions or requests for new samples to build. If you are interested in vSphere Automation and would like to leverage Python, be sure to check out the pyvmomi Github repository! Lastly, if you have written some cool scripts/applications or would like to request specific sample scripts, be sure to send a pull request to Shawn as we would love to see more contributions and collaborations from the community!

How to bootstrap Horizon View 5.3.1 onto a VSAN Datastore using VCT

03.14.2014 by William Lam // 2 Comments

As some of you may have heard, vSphere 5.5 Update 1 including the much anticipated VMware Virtual SAN (VSAN) was released earlier this week. To take advantage of this new vSphere release, other VMware solutions were also updated including the latest Horizon View 5.3.1 which now supports VSAN. Having spent some time playing around with the recent VMware Fling VCT which I have written about here and here, I thought why not give the latest version a try? I really like the simplicity of the VCT appliance which allows you to easily deploy an entire Horizon View environment from scratch and it just requires a stand-alone ESXi host not managed by vCenter Server and a Window 2008 ISO. Out of the box, VCT only supports vSphere 5.5 and Horizon View 5.3, but you can easily tweak either the HTML pages or modify the POST request to deploy the latest version of VCSA and Horizon View/Composer.

Since Horizon View 5.3.1 supports VSAN, I thought it would be neat to be able to "bootstrap" the entire Horizon View environment onto a VSAN datastore which would allow you to consume all the underlying physical disks without having to resort to a "temporary" VMFS volume to deploy the additional infrastructure. Now of course, this assumes you have no existing infrastructure or if you want to quickly spin up a POC or testing environment for View. The diagram above illustrates what the environment will look like at the end.

Below are the instructions for leveraging VCT to automate the deployment of vCenter Server Appliance 5.5 Update 1 and Horizon View 5.3.1

Step 1 - Install ESXi 5.5 Update 1 on your physical ESXi host or even virtual (which is how I tested this configuration)

Step 2 - Configure VSAN datastore on your single ESXi node using the "bootstrap" instructions here

Step 2 - Download VMware VCT & Studio and follow the optimized deployment here and deploy onto the VSAN datastore in previous step

Step 3 - Download VCSA 5.5 Update 1 & Horizon View 5.3.1, download links can be found here

Step 4 - Upload the three files to the VCT appliance via SCP and store them in /installers directory

Step 5 - If you plan on automating the Horizon View deployment from the command line, you will need the automateVCT.sh script and modify the following variables so they look like the following:

VIEW_SERVER_BIN=VMware-viewconnectionserver-x86_64-5.3.1-1634134.exe
VIEW_COMPOSER_BIN=VMware-viewcomposer-5.3.1-1634135.exe
VCSA_OVA=VMware-vCenter-Server-Appliance-5.5.0.10000-1624811_OVF10.ova

If you plan on using the VCT UI to deploy your Horizon View environment, then you will need to edit the following HTML files and either append or replace the VCSA OVA / View filenames

/apache-tomcat-7.0.27/webapps/vct/existing.html
/apache-tomcat-7.0.27/webapps/vct/new.html

Once the files have been uploaded, you will then be able to run through VCT as you normally would and in an hour or so, you should have a fully deployed VCSA 5.5 Update 1 and Horizon View 5.3.1 environment up and running on top of a VSAN datastore.

Once your initial ESXi host is configured, you can then deploy your other ESXi hosts and add them to your VSAN Cluster. Remember, you should have at least a minimum of 3 nodes with a recommendation of 4 as pointed out by Duncan Epping in his blog article here. From what I can tell, VCT had no issues provisioning VCSA 5.5 Update 1, there were no major changes and the same goes for Horizon View 5.3.1. I was even able to add in my vCenter Server to ensure basic View functionality was working.

I think this is a great way if you want to quickly setup a vSphere 5.5 Update 1 environment and evaluate the latest Horizon View 5.3.1 release. I also came across this Horizon View 5.3.1 on VMware Virtual SAN - Quick Start Guide KB that will also come in handy if you are looking to use Horizon View with VSAN.