William Lam

Does the ESXi Mac Learn dvFilter work with Nested ESXi on NSX VXLAN's?

09.19.2014 by William Lam // 3 Comments

After publishing my article on the new ESXi Mac Learn dvFilter which helps improve CPU/Network performance when using promiscuous mode with Nested ESXi, I received a couple of questions asking whether the dvFilter would work with NSX VXLAN's? At the time, I had only tested the Mac Learn dvFilter using standard VSS/VDS and not with any VXLAN based networks. I had reached out to a couple of folks asking whether this would work and to my surprise, I actually got back a mix set of answers to it will not work to it could work. One of the reasons that was given to me on why this may not work is that NSX-v (NSX for vSphere) leverages a different "virtual switch" than VSS/VDS and hence the Mac Learn dvFilter would not properly function. This actually would make sense, but because I received other responses negating that fact, I figured I probably should just test it for myself and see.

NSX 6.1 was recently released and I figured this would be a great opportunity for me to learn a bit more about NSX, as I have never played with it before and also test whether Mac Learn dvFilter would in fact work with NSX VXLAN's. In my lab environment I have deployed NSX and I have 3 physical ESXi hosts running VSAN (go SDS!). I deployed both an NSX ESR (Edge Service Router) hosting 2 Logical Networks (aka VXLAN segments) and an NSX DLR (Distributed Logical Router) hosting another 2 Logical Networks.

Here is a screenshot of the 4 Logical Networks, the first two on NSX ESR and the last two on NSX DLR:

Here is a screenshot of both the NSX ESR and DLR:

Note: If you would like to learn more about NSX ESR and DLR, check out this great article by Brad Hedlund who goes into more detail.

For my test, I first enabled Promiscuous Mode and Forged Transmit on the respective Logical Switches which is just a dvPortgroup on the VDS for my NSX ESR setup. I then had 2 Nested ESXi VMs running (without the Mac Learn dvFilter), a Windows "Jump Box" VM and vMA all connected to the same VLXAN network.

I then transfer an ISO from the Windows VM to vMA while running ESXTOP on the physical ESXi host which is hosting these four VMs. As I expected, both the Nested ESXi VMs and vMA were receiving network packets. Next, I installed the Mac Learn dvFilter VIB on the physical ESXi host and added the required VM Advanced Settings to both the Nested ESXi VMs and then re-ran the test. To my surprise, both the Nested ESXi VMs were no longer receiving the erroneous packets! So it seems that using VLXAN with NSX ESR, the Mac Learn dvFilter is working as expected.

To be thorough, I also ran through same test but now for the VXLAN segments backed by NSX DLR. This time, I was really surprised by the results. The test was prior to installing the Mac Learn dvFilter and my expectation was that the two Nested ESXi VMs would be seeing the duplicated network packets from the VDS, but to my surprise, they did not! Both the Nested ESXi VMs were pretty much idling at 0 packets as nothing was being sent to them. I am not exactly sure why I was seeing this behavior, perhaps there is some type of optimization in the DLR? This is something I hope to get an answer from someone in Engineering on why I might be seeing this positive behavior.

To summarize, this myth has been busted and the Mac Learn dvFilter does in fact work with VXLAN networks. If you are using NSX ESR for your VXLAN setup, then you will need to install the dvFilter and if you are using NSX DLR, it seems like you do not need to make any additional changes. After briefly speaking with Christian Dickmann, the creator of the dvFilter as I wanted to share the results with him, I also learned about some interesting tidbits. Christian was not surprised by the results actually, the reason for this is that the VMkernel networking stack was architected and designed to be modular. This meant that, one could switch out the "virtual switch" with other implementations and the underlying dvFilter framework would still continue to work regardless of the "virtual switch" being used.

Additional Note:

I did not get a chance to test with vCNS and VXLAN, but I believe it should work given NSX-v is functional. If you are able to test this, feel free to leave a comment on whether the expected behavior is seen with the Mac Learn dvFilter.
I did not get a chance to test this with vCloud Director with VXLAN based networks, but as I mentioned, this should work. Please leave a comment if you can confirm
I also noticed when creating the Logical Switches, there is a Mac Learning capability, but from my testing, I found it did not benefited Nested ESXi and the Mac Learn dvFilter was still required.

Community stories of VMware & Apple OS X in Production: Part 7

09.18.2014 by William Lam // Leave a Comment

Company: Fortune 150 (Retail)
Software: vSphere + vSphere Replication
Hardware: Apple Mac Pro

[William] - Hi Vitaliy, thank you for reaching out and wanting to share your experiences with the community on managing a VMware and Apple OS X infrastructure. Can you tell us a little bit about yourself and what you currently do?

[Vitaliy] - I am a Senior Systems Analyst for a Fortune 150 company that wishes to remain anonymous (aka I do not have legal clearance to use the company name). I am part of a team that is responsible for providing IT infrastructure for many creative and marketing applications -- think pre-press and advertising.

[William] - Can you provide us some details about the VMware and OS X infrastructure that you’re supporting? Software/Hardware specs that you decided to go with and the workload characteristics?

[Vitaliy] - Prior to virtualization we were running two dozen Xserves with OS X 10.6 running a wide range of applications from Open Directory to custom in-house scripts. We have virtualized the whole environment with just 4 Mac Pro machines, each machine has 12 cores and 64GB of memory giving us a total of about 128GHz and 256GB of memory.

We have exhausted all the PCI-X slots on the Mac Pro's by adding two dual port network cards and a dual port HBA. As a result we have two redundant management, data, and vMotion ports on each machine. Oh, one thing worth mentioning is that VMware officially only supports 32GB of memory per Mac Pro but we have been running 64GB with no issues. For the past year we have been running vSphere 5.1 and just upgraded to 5.5 last week.

We have been using HP 3PAR SAN for our storage back-end and over the last couple of weeks we have migrated to an Oracle SAN. The whole process was completely seamless and transparent to the users thanks to VMware.

Here is a picture of the Mac Pro setup courtesy of Vitaliy:

[William] - Wow, that’s great to hear you’ve been able to really push the Mac Pro’s. You must have been happy to be able to consolidate all those Xserves! What was your approach for virtualizing OS X from the physical Xserve to Mac Pro? Did you rebuild or leverage some type of V2V?

[Vitaliy] - We decided to rebuild from scratch. We were running an outdated version of OS X 10.6 and all the applications running on top of that were just as old.

[William] - Can you talk to how you provision your OS X Virtual Machines and Applications and how it gets to the end users? Do users get their own systems or is this a shared infrastructure?

[Vitaliy] - It's a shared infrastructure, generally a VM is dedicated to a particular application. We created a "base VM" that has basic settings like power/energy saver settings, local accounts, monitoring software, etc. preconfigured and whenever we need a new virtual machine we simply clone it and change the hostname and IP address on the new VM. Perhaps a template would've been a cleaner solution but this is what we do. We are currently looking into automating configuration with either Puppet or Casper.

When we initially rolled out a couple of OS X virtual machines we noticed that CPU usage on the VMware cluster spiked up to almost a 100% while the virtual machines were idle. It turned out that the default OS X screensaver uses GPU power to generate that flare effect and because not enough GPU memory was available it resorted to using up all the CPU. Disabling the screensaver or switching to a text based one quickly fixed that issue ...

[William] - Thanks for the excellent tip on OS X screensaver, this is a handy one to know about! How do you go about monitoring the Mac Pro infrastructure? What’s the process for replacing failed hardware components and have you had any challenges with this?

[Vitaliy] - We treat it the same way as the rest of our environment -- each vSphere node and virtual machine is monitored via Nagios. We have this cluster running for little over a year now and luckily we have not had to deal with any hardware failure.

[William] - For your OS X Virtual Machines, do you have a need for backups or a DR strategy? If so, could you share some details on what you are currently using?

[Vitaliy] - We have a replica of our production environment at a remote disaster recovery site and we use vSphere Replication to copy all the VMs nightly. We also heavily rely on the snapshot feature prior to making any operating system or application changes, it has been a lifesaver so far.

[William] - Vitaliy, I want to say thank you very much for taking some time out of your super busy schedule to have a chat. Before I let you go, do you have any words of wisdom for others looking to manage a similar infrastructure? Anything you would do differently and any resources you have found useful in aiding you to support a VMware / OS X infrastructure?

[Vitaliy] - Speak to your manager, legal department, or whoever is in charge about interpreting Apple EULA. I have heard of at least three different interpretations and all have legal implications. I am very happy with our environment and would not change a thing if I had to build it again. Your blog, virtuallyGhetto, has been a great resource as you are the only one talking about VMware products running on Apple hardware.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Want to issue a VAAI UNMAP operation using the vSphere Web Client?

09.18.2014 by William Lam // 3 Comments

Recently, I have seen several requests from both customers and partners wanting to be able to run the VAAI UNMAP operation from within the vSphere Web Client. For those of you not familiar with the VAAI UNMAP operation, I recommend you check out this blog post by my colleague Cormac Hogan. Today, the only way to issue the UNMAP operation is by using ESXCLI either remotely or in the ESXi Shell. There is currently not a vSphere API for this operation and therefore it would be difficult to build a native vSphere Web Client Plugin to provide this functionality.

Having said that, one way to provide this capability is through the use of a vCenter Orchestrator (vCO) Workflow which can remotely execute an ESXCLI command whether that is going through ESXCLI using a Linux jump box or through PowerCLI using a Windows jump box. Starting with vSphere 5.5, you can now extend the vSphere Web Client and attach a vCO Workflow to a vSphere Object and be able to execute the workflow right from the vSphere Web Client. This is great if you are already using vCO, but for those that are not, it can be somewhat complex to setup along with a steep learning curve depending on your experience.

Today, there was an exciting announcement from my Automation buddy, Alan Renouf for a new VMware Fling called PowerActions for the vSphere Web Client. This new Fling allows you to easily extend the vSphere Web Client in the following ways:

Access a PowerCLI console directly in the vSphere Web Client
Ability to run a context aware PowerCLI script directly from the vSphere Web Client

The prerequisite for setting up PowerActions is no different than vCO calling a PowerCLI script, you just need a Windows "jump-box" that has PowerCLI installed along with PowerActions. The added benefit, is that you do need to setup another piece of infrastructure like vCO if you are not already using it. This made setting up PowerActions extremely easy to setup and even I was able get it up and running in under 5minutes (minus a quick RTFM moment :)).

Given the number of inquiries regarding VAAI UNMAP operation via the vSphere Web Client, I thought that would be a great use case for my first PowerActions script! Below are the instructions on creating the VAAI UNMAP script for PowerActions:

Step 1 - Click on the PowerCLI Scripts option on the left hand side of the Object Navigator and then click on the "New Script" Icon. Select Datastore as the context aware object for the script.

Step 2 - Provide a name and description for the script. Also make sure to select "Action".

Step 3 - Copy and paste the following script from https://github.com/lamw/vghetto-scripts/blob/master/powershell/unmap-poweraction.ps1 inside the script window and then save the script. What the script does is takes the Datastore object and retrieves a list of ESXi hosts that has access to the Datastore and then randomly selects one of the host. This is required because ESXCLI operations on a per host level and we use that information to pass into Get-EsxCli cmdlet to issue the VAAI UNMAP operation.

Step 4 - To test the script, you just need to right click on a VMFS Datastore and click on PowerCLI->Execute a Script

Note: Please be aware of the impact when running a UNMAP operation, you may want to run this on a non-production datastore for testing purposes or during off hours when your workload may not be as busy.

Step 5 - Select the VAAI UNMAP script you just created and once selected and you will be prompted to specify the number of VMFS blocks to unmap per iteration which is exactly the same input when manually ESXCLI.

At this point, if everything was successful the VAAI UNMAP operation should begin and you can tail /var/log/hostd.log to see the UNAMP operation. Once completed, you should see the prompt return true.

As you can see, it was extremely easy to create my own PowerAction script that expose new functionality and making it available within the vSphere Web Client. I think this is going to be a pretty popular Fling and remember if this is something you would like to see officially in the product, be sure to leave a comment on the PowerAction for vSphere Web Client Fling page, product managers are listening! The only feedback I have is that I would love to see this get extended beyond just PowerCLI and into a generic script extension framework, just imagine the possibilities!