WilliamLam.com

A gift for my Waves to Wine 2014 Supporters!

09.22.2014 by William Lam // Leave a Comment

This weekend I took part in the Waves to Wine 2014 ride for the National MS Society. I took the century option (104 miles) starting from Cow Palace in San Francisco and finished at Ronert Park in Sonoma County. Not only was this an amazing experience and a personal goal of mine to complete a century ride, but I was also able to ride for an awesome cause.

I ended up riding 109.1 miles in total due to slight detour that I took (luckily it was only a couple of miles). In addition to a couple of rest stops, I also had take an unexpected stop as my GoPro was vibrating so much, it broke off its connector while I was riding. Luckily, I was able to re-connect the GoPro directly to the bike mount instead of the intermediate adapter (whew). The ride took a total of 7hrs and 41minutes to complete, I think I probably could have finished it in 7.5hrs flat if it was not for the detour!

Lastly, I would like to say thank you to all my fellow VMware colleagues and Virtualization community members (listed below) for your support and donations. With your help, I was able to fund-raise a grand total of $6,206 for the National MS Society! As a small token of my appreciation, I have put together a short video highlighting my 7hr ride into a couple of minutes (make sure to watch the entire video :)) and I hope you enjoy it!

BikeMS: Waves To Wine 2014 from lamw on Vimeo.

Thank you very much to the following folks who donated:

Aaron Blasius
Balaji Parimi
Ben Lin
Bill Call
Chas Setchell
Chris Wolf
Coho Data
DataGravity
Dexter Arver
Dmitriy Sandler
Doug Baer
Duncan Epping
Ed Swindelles
El-Zein Family
Erik Bussink
Evan Bills
Frank Buechsel
Gary Blake
Greg Mulholland
Henry Robinson
Howard Marks
Irfan Ahmad
Jackie Rees
Jeff Hu
Jehad Affoneh
Jeremy Rickard
Joerg Lew
John W
Judy Snow
Kaushik Banerjee
Kris Inglis
Linda Jones
Mark Chuang
Matt Dreyer
Max Daneri
Michael Emery
Michael White
Mike Nisk
Minh Dao
Nambisan family
Noel Nguyen
Paul Meehan
Phillip Jordan
Platform9 Systems
Raj Yavatkar
Ravi Soundararajan
Robert Bosch
Ron Totah
Ronald Chenoweth
Skyera Inc.
Steve Kaplan
Susan Gudenkauf and Family
The Delgado Family
The Murray Family
Tina Walsworth
vmkdaily

Does the ESXi Mac Learn dvFilter work with Nested ESXi on NSX VXLAN's?

09.19.2014 by William Lam // 3 Comments

After publishing my article on the new ESXi Mac Learn dvFilter which helps improve CPU/Network performance when using promiscuous mode with Nested ESXi, I received a couple of questions asking whether the dvFilter would work with NSX VXLAN's? At the time, I had only tested the Mac Learn dvFilter using standard VSS/VDS and not with any VXLAN based networks. I had reached out to a couple of folks asking whether this would work and to my surprise, I actually got back a mix set of answers to it will not work to it could work. One of the reasons that was given to me on why this may not work is that NSX-v (NSX for vSphere) leverages a different "virtual switch" than VSS/VDS and hence the Mac Learn dvFilter would not properly function. This actually would make sense, but because I received other responses negating that fact, I figured I probably should just test it for myself and see.

NSX 6.1 was recently released and I figured this would be a great opportunity for me to learn a bit more about NSX, as I have never played with it before and also test whether Mac Learn dvFilter would in fact work with NSX VXLAN's. In my lab environment I have deployed NSX and I have 3 physical ESXi hosts running VSAN (go SDS!). I deployed both an NSX ESR (Edge Service Router) hosting 2 Logical Networks (aka VXLAN segments) and an NSX DLR (Distributed Logical Router) hosting another 2 Logical Networks.

Here is a screenshot of the 4 Logical Networks, the first two on NSX ESR and the last two on NSX DLR:

Here is a screenshot of both the NSX ESR and DLR:

Note: If you would like to learn more about NSX ESR and DLR, check out this great article by Brad Hedlund who goes into more detail.

For my test, I first enabled Promiscuous Mode and Forged Transmit on the respective Logical Switches which is just a dvPortgroup on the VDS for my NSX ESR setup. I then had 2 Nested ESXi VMs running (without the Mac Learn dvFilter), a Windows "Jump Box" VM and vMA all connected to the same VLXAN network.

I then transfer an ISO from the Windows VM to vMA while running ESXTOP on the physical ESXi host which is hosting these four VMs. As I expected, both the Nested ESXi VMs and vMA were receiving network packets. Next, I installed the Mac Learn dvFilter VIB on the physical ESXi host and added the required VM Advanced Settings to both the Nested ESXi VMs and then re-ran the test. To my surprise, both the Nested ESXi VMs were no longer receiving the erroneous packets! So it seems that using VLXAN with NSX ESR, the Mac Learn dvFilter is working as expected.

To be thorough, I also ran through same test but now for the VXLAN segments backed by NSX DLR. This time, I was really surprised by the results. The test was prior to installing the Mac Learn dvFilter and my expectation was that the two Nested ESXi VMs would be seeing the duplicated network packets from the VDS, but to my surprise, they did not! Both the Nested ESXi VMs were pretty much idling at 0 packets as nothing was being sent to them. I am not exactly sure why I was seeing this behavior, perhaps there is some type of optimization in the DLR? This is something I hope to get an answer from someone in Engineering on why I might be seeing this positive behavior.

To summarize, this myth has been busted and the Mac Learn dvFilter does in fact work with VXLAN networks. If you are using NSX ESR for your VXLAN setup, then you will need to install the dvFilter and if you are using NSX DLR, it seems like you do not need to make any additional changes. After briefly speaking with Christian Dickmann, the creator of the dvFilter as I wanted to share the results with him, I also learned about some interesting tidbits. Christian was not surprised by the results actually, the reason for this is that the VMkernel networking stack was architected and designed to be modular. This meant that, one could switch out the "virtual switch" with other implementations and the underlying dvFilter framework would still continue to work regardless of the "virtual switch" being used.

Additional Note:

I did not get a chance to test with vCNS and VXLAN, but I believe it should work given NSX-v is functional. If you are able to test this, feel free to leave a comment on whether the expected behavior is seen with the Mac Learn dvFilter.
I did not get a chance to test this with vCloud Director with VXLAN based networks, but as I mentioned, this should work. Please leave a comment if you can confirm
I also noticed when creating the Logical Switches, there is a Mac Learning capability, but from my testing, I found it did not benefited Nested ESXi and the Mac Learn dvFilter was still required.

Community stories of VMware & Apple OS X in Production: Part 7

09.18.2014 by William Lam // Leave a Comment

Company: Fortune 150 (Retail)
Software: vSphere + vSphere Replication
Hardware: Apple Mac Pro

[William] - Hi Vitaliy, thank you for reaching out and wanting to share your experiences with the community on managing a VMware and Apple OS X infrastructure. Can you tell us a little bit about yourself and what you currently do?

[Vitaliy] - I am a Senior Systems Analyst for a Fortune 150 company that wishes to remain anonymous (aka I do not have legal clearance to use the company name). I am part of a team that is responsible for providing IT infrastructure for many creative and marketing applications -- think pre-press and advertising.

[William] - Can you provide us some details about the VMware and OS X infrastructure that you’re supporting? Software/Hardware specs that you decided to go with and the workload characteristics?

[Vitaliy] - Prior to virtualization we were running two dozen Xserves with OS X 10.6 running a wide range of applications from Open Directory to custom in-house scripts. We have virtualized the whole environment with just 4 Mac Pro machines, each machine has 12 cores and 64GB of memory giving us a total of about 128GHz and 256GB of memory.

We have exhausted all the PCI-X slots on the Mac Pro's by adding two dual port network cards and a dual port HBA. As a result we have two redundant management, data, and vMotion ports on each machine. Oh, one thing worth mentioning is that VMware officially only supports 32GB of memory per Mac Pro but we have been running 64GB with no issues. For the past year we have been running vSphere 5.1 and just upgraded to 5.5 last week.

We have been using HP 3PAR SAN for our storage back-end and over the last couple of weeks we have migrated to an Oracle SAN. The whole process was completely seamless and transparent to the users thanks to VMware.

Here is a picture of the Mac Pro setup courtesy of Vitaliy:

[William] - Wow, that’s great to hear you’ve been able to really push the Mac Pro’s. You must have been happy to be able to consolidate all those Xserves! What was your approach for virtualizing OS X from the physical Xserve to Mac Pro? Did you rebuild or leverage some type of V2V?

[Vitaliy] - We decided to rebuild from scratch. We were running an outdated version of OS X 10.6 and all the applications running on top of that were just as old.

[William] - Can you talk to how you provision your OS X Virtual Machines and Applications and how it gets to the end users? Do users get their own systems or is this a shared infrastructure?

[Vitaliy] - It's a shared infrastructure, generally a VM is dedicated to a particular application. We created a "base VM" that has basic settings like power/energy saver settings, local accounts, monitoring software, etc. preconfigured and whenever we need a new virtual machine we simply clone it and change the hostname and IP address on the new VM. Perhaps a template would've been a cleaner solution but this is what we do. We are currently looking into automating configuration with either Puppet or Casper.

When we initially rolled out a couple of OS X virtual machines we noticed that CPU usage on the VMware cluster spiked up to almost a 100% while the virtual machines were idle. It turned out that the default OS X screensaver uses GPU power to generate that flare effect and because not enough GPU memory was available it resorted to using up all the CPU. Disabling the screensaver or switching to a text based one quickly fixed that issue ...

[William] - Thanks for the excellent tip on OS X screensaver, this is a handy one to know about! How do you go about monitoring the Mac Pro infrastructure? What’s the process for replacing failed hardware components and have you had any challenges with this?

[Vitaliy] - We treat it the same way as the rest of our environment -- each vSphere node and virtual machine is monitored via Nagios. We have this cluster running for little over a year now and luckily we have not had to deal with any hardware failure.

[William] - For your OS X Virtual Machines, do you have a need for backups or a DR strategy? If so, could you share some details on what you are currently using?

[Vitaliy] - We have a replica of our production environment at a remote disaster recovery site and we use vSphere Replication to copy all the VMs nightly. We also heavily rely on the snapshot feature prior to making any operating system or application changes, it has been a lifesaver so far.

[William] - Vitaliy, I want to say thank you very much for taking some time out of your super busy schedule to have a chat. Before I let you go, do you have any words of wisdom for others looking to manage a similar infrastructure? Anything you would do differently and any resources you have found useful in aiding you to support a VMware / OS X infrastructure?

[Vitaliy] - Speak to your manager, legal department, or whoever is in charge about interpreting Apple EULA. I have heard of at least three different interpretations and all have legal implications. I am very happy with our environment and would not change a thing if I had to build it again. Your blog, virtuallyGhetto, has been a great resource as you are the only one talking about VMware products running on Apple hardware.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.