ESXi – Page 130

Community stories of VMware & Apple OS X in Production: Part 8

09.25.2014 by William Lam // 3 Comments

Company: Mid-Pacific Institute (Private School in Hawaii)
Software: VMware vSphere and Fusion
Hardware: Apple Mac Pro

[William] - Hi Derick, I appreciate you taking some time out of your busy schedule to talk with us all the way from Hawaii 🙂 Before we get started, can you quickly introduce yourself?

[Derick] - Sure William. My name is Derick Okihara, and I work for Mid-Pacific Institute. We are a private K-12 institution with about 1600 students. My role here is general IT and server administration. I've been working with computers since I was in high school. I have been a long time Apple user (since //gs), but really started working with them professionally about 10 years ago. We currently have a 1:1 iPad program for the students, and 2:1 iPad+laptops for faculty, so there's a lot of technology to support.

[William] - That’s awesome Derick. Look forward to hearing more about your environment. Speaking of which, I hear you are currently managing some Apple hardware running on VMware? Could you tell us a little bit about the hardware configuration and the VMware software you are currently using?

[Derick] - We are currently using vCenter with ESXi 5.5, we have 2 Mac Pro (5,1s) in a cluster with a Synology 1813+ shared storage. The network storage is connected via the iSCSI software initiator using round-robin. We also use VMware Fusion for the Mac. The Mac Pros have 24GB of RAM and 4 port Intel Gigabit Ethernet cards for a total of 6 ports each.

[William] - What made you decide on using a Synology for shared storage and what configuration/capacity did you go with? Were there any other options you were looking at?

[Derick] - A lot of the decisions for this setup was made on price. How this all started, was that I was asked to create an ESXi server to host a VM Appliance to run our campus wide Informacast speaker system. I had already been planning an ESXI deployment on the Mac, testing on Mac Mini. Instead of building a PC server just for this appliance, I was given the OK to build on an existing Mac Pro, so it could serve multiple purposes.

Being forward thinking, I knew we needed redundancy, so I opted to go for network storage. With a tight budget, and being able to use CPUs we already had, I decided on the Synology 1813+ for it's 4 gigabit ports. This allowed me to later add the 2nd host in our vcenter cluster when we expanded.

[William] - Can you talk a little bit about the type of workloads and applications you are currently virtualizing on the Mac Pro’s and are these all OSX VMs?

[Derick] - Since this is still version 1.0, we aren't heavily taxing our cluster. Right now it hosts 5 VMs (2 Virtual Appliance, 2 OS X Server, 1 Windows Server). I'd want to add more RAM as well, OS X VMs are very RAM hungry.

The OS X servers are a student file server (AFP/SMB) and an Apple Caching server / Munki repository. The Windows server is mostly a test bed, the Virtual Appliances are the aforementioned Informacast manager and VCenter Appliance.

Here is a picture Derick's two Apple Mac Pros:

[William] - The Mac Pro’s have a maximum amount of memory that it supports, do you plan on expanding the infrastructure to accommodate additional workloads or would you be looking at upgrading to the latest generation of Mac Pro (black)?

[Derick] - Honestly, with our current needs and budget, I think I would be looking at the next generation Mac Mini combined with some sort of PCI-E enclosure. Like the Sonnet XMac server. I know the Mini will likely not be fully supported, but I like what i've seen on virtuallyGhetto with the current generation 🙂 That is, if the next gen Mac Mini supports 32GB of RAM!

[William] - Very cool! So from your point of view, you would rather have more Mac Mini’s than a couple of Mac Pro’s? It sounds like cost plays a huge factor, but what other constraints or requirements that is making you lean more towards Mac Mini’s instead of going to a new Mac Pro which can get up to 64GB of memory and 6-Core CPU?

[Derick] - Footprint - the Mac Pros we currently have take up a large amount of rack space. Even the new mac pros would not be rack mountable without an additional enclosure. For us, having 3 x Mac Mini with 32GB of RAM would be ideal price/performance ratio. (We have a 3 CPU license). Eventually our Mac Pro 5,1's will die, so I'm already thinking about what's next. Having 3 x Mac Mini servers in a cluster, that takes up only 3U would be pretty sweet!

[William] - Speaking of support, did you purchase any type of extended contracts with Apple on the hardware or you going to deal with them on a case by case basis? Have you had any issues with failing hardware on the Mac Pro’s?

[Derick] - We only had the initial Apple Care (now since expired). We have 1 spare Mac Pro currently running other loads but that could be migrated in the event that we have a hardware failure. We have not had many issues on the Mac Pro 5,1s other than internal hard drive failure. They've been rock solid.

[William] - Has there been any interesting issues or challenges you had faced while setting up this infrastructure? Either the hardware, software or managing the VMs?

[Derick] - This whole process was a learning experience for me. At a previous job, I had inherited an ESXI server running multiple CentOS and Ubuntu VMs, but I had never set it up myself, let alone on a Mac. Thanks to the multiple resources on the web (Rich Trouton's blog, VirtuallyGhetto, and a P2V script from Alan Gordon at MacSysAdmin) the process and gotchas were mostly done before me

The biggest challenge for me was configuring the Synology for iSCSI-round robin. In my research I found that one could utilize multiple gigabit connections with Multi-path IO for higher bandwidth. After lots of configuring and back and forth with Synology / VMware support, I finally found the proper settings that allowed me to utilize more than 1 gigabit link.
However, after I updated to ESXi 5.5, it broke.

I was stuck, because I needed to upgrade to 5.5 in order to run an OS X caching server (12-character serial number fix in 5.5). But Synology said the 1813+ was not compatible with 5.5 and would not help me. Long story short, one of my hosts is running 5.5 (with OS X Caching server) and the other host is running 5.1 (file services) because it needs the greater throughput.

[William] - Derick, I want thank you for taking the time and sharing with us your experiences with managing VMware and Apple hardware. Before I let you go, do you have any tips for our readers that may be in a similar environment (academic) and needs to build out an infrastructure to support their end users? Any gotchas or things you would recommend if you had to do this all over again?

[Derick] - Anyone looking to reduce their machine footprint should definitely look into virtualization. VMware has very attractive pricing for the EDU market if you're looking to build a cluster with high availability, or you can run a single host for free. The best piece of advice I can give is just to test thoroughly. Virtualization is very complicated, and combines a multitude of areas of expertise (Storage, Networks, Workloads, and ESXI platform itself). It can be daunting but it's very rewarding. If you get stuck, just ask William on Twitter @lamw, jk

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Does the ESXi Mac Learn dvFilter work with Nested ESXi on NSX VXLAN's?

09.19.2014 by William Lam // 3 Comments

After publishing my article on the new ESXi Mac Learn dvFilter which helps improve CPU/Network performance when using promiscuous mode with Nested ESXi, I received a couple of questions asking whether the dvFilter would work with NSX VXLAN's? At the time, I had only tested the Mac Learn dvFilter using standard VSS/VDS and not with any VXLAN based networks. I had reached out to a couple of folks asking whether this would work and to my surprise, I actually got back a mix set of answers to it will not work to it could work. One of the reasons that was given to me on why this may not work is that NSX-v (NSX for vSphere) leverages a different "virtual switch" than VSS/VDS and hence the Mac Learn dvFilter would not properly function. This actually would make sense, but because I received other responses negating that fact, I figured I probably should just test it for myself and see.

NSX 6.1 was recently released and I figured this would be a great opportunity for me to learn a bit more about NSX, as I have never played with it before and also test whether Mac Learn dvFilter would in fact work with NSX VXLAN's. In my lab environment I have deployed NSX and I have 3 physical ESXi hosts running VSAN (go SDS!). I deployed both an NSX ESR (Edge Service Router) hosting 2 Logical Networks (aka VXLAN segments) and an NSX DLR (Distributed Logical Router) hosting another 2 Logical Networks.

Here is a screenshot of the 4 Logical Networks, the first two on NSX ESR and the last two on NSX DLR:

Here is a screenshot of both the NSX ESR and DLR:

Note: If you would like to learn more about NSX ESR and DLR, check out this great article by Brad Hedlund who goes into more detail.

For my test, I first enabled Promiscuous Mode and Forged Transmit on the respective Logical Switches which is just a dvPortgroup on the VDS for my NSX ESR setup. I then had 2 Nested ESXi VMs running (without the Mac Learn dvFilter), a Windows "Jump Box" VM and vMA all connected to the same VLXAN network.

I then transfer an ISO from the Windows VM to vMA while running ESXTOP on the physical ESXi host which is hosting these four VMs. As I expected, both the Nested ESXi VMs and vMA were receiving network packets. Next, I installed the Mac Learn dvFilter VIB on the physical ESXi host and added the required VM Advanced Settings to both the Nested ESXi VMs and then re-ran the test. To my surprise, both the Nested ESXi VMs were no longer receiving the erroneous packets! So it seems that using VLXAN with NSX ESR, the Mac Learn dvFilter is working as expected.

To be thorough, I also ran through same test but now for the VXLAN segments backed by NSX DLR. This time, I was really surprised by the results. The test was prior to installing the Mac Learn dvFilter and my expectation was that the two Nested ESXi VMs would be seeing the duplicated network packets from the VDS, but to my surprise, they did not! Both the Nested ESXi VMs were pretty much idling at 0 packets as nothing was being sent to them. I am not exactly sure why I was seeing this behavior, perhaps there is some type of optimization in the DLR? This is something I hope to get an answer from someone in Engineering on why I might be seeing this positive behavior.

To summarize, this myth has been busted and the Mac Learn dvFilter does in fact work with VXLAN networks. If you are using NSX ESR for your VXLAN setup, then you will need to install the dvFilter and if you are using NSX DLR, it seems like you do not need to make any additional changes. After briefly speaking with Christian Dickmann, the creator of the dvFilter as I wanted to share the results with him, I also learned about some interesting tidbits. Christian was not surprised by the results actually, the reason for this is that the VMkernel networking stack was architected and designed to be modular. This meant that, one could switch out the "virtual switch" with other implementations and the underlying dvFilter framework would still continue to work regardless of the "virtual switch" being used.

Additional Note:

I did not get a chance to test with vCNS and VXLAN, but I believe it should work given NSX-v is functional. If you are able to test this, feel free to leave a comment on whether the expected behavior is seen with the Mac Learn dvFilter.
I did not get a chance to test this with vCloud Director with VXLAN based networks, but as I mentioned, this should work. Please leave a comment if you can confirm
I also noticed when creating the Logical Switches, there is a Mac Learning capability, but from my testing, I found it did not benefited Nested ESXi and the Mac Learn dvFilter was still required.

Community stories of VMware & Apple OS X in Production: Part 7

09.18.2014 by William Lam // Leave a Comment

Company: Fortune 150 (Retail)
Software: vSphere + vSphere Replication
Hardware: Apple Mac Pro

[William] - Hi Vitaliy, thank you for reaching out and wanting to share your experiences with the community on managing a VMware and Apple OS X infrastructure. Can you tell us a little bit about yourself and what you currently do?

[Vitaliy] - I am a Senior Systems Analyst for a Fortune 150 company that wishes to remain anonymous (aka I do not have legal clearance to use the company name). I am part of a team that is responsible for providing IT infrastructure for many creative and marketing applications -- think pre-press and advertising.

[William] - Can you provide us some details about the VMware and OS X infrastructure that you’re supporting? Software/Hardware specs that you decided to go with and the workload characteristics?

[Vitaliy] - Prior to virtualization we were running two dozen Xserves with OS X 10.6 running a wide range of applications from Open Directory to custom in-house scripts. We have virtualized the whole environment with just 4 Mac Pro machines, each machine has 12 cores and 64GB of memory giving us a total of about 128GHz and 256GB of memory.

We have exhausted all the PCI-X slots on the Mac Pro's by adding two dual port network cards and a dual port HBA. As a result we have two redundant management, data, and vMotion ports on each machine. Oh, one thing worth mentioning is that VMware officially only supports 32GB of memory per Mac Pro but we have been running 64GB with no issues. For the past year we have been running vSphere 5.1 and just upgraded to 5.5 last week.

We have been using HP 3PAR SAN for our storage back-end and over the last couple of weeks we have migrated to an Oracle SAN. The whole process was completely seamless and transparent to the users thanks to VMware.

Here is a picture of the Mac Pro setup courtesy of Vitaliy:

[William] - Wow, that’s great to hear you’ve been able to really push the Mac Pro’s. You must have been happy to be able to consolidate all those Xserves! What was your approach for virtualizing OS X from the physical Xserve to Mac Pro? Did you rebuild or leverage some type of V2V?

[Vitaliy] - We decided to rebuild from scratch. We were running an outdated version of OS X 10.6 and all the applications running on top of that were just as old.

[William] - Can you talk to how you provision your OS X Virtual Machines and Applications and how it gets to the end users? Do users get their own systems or is this a shared infrastructure?

[Vitaliy] - It's a shared infrastructure, generally a VM is dedicated to a particular application. We created a "base VM" that has basic settings like power/energy saver settings, local accounts, monitoring software, etc. preconfigured and whenever we need a new virtual machine we simply clone it and change the hostname and IP address on the new VM. Perhaps a template would've been a cleaner solution but this is what we do. We are currently looking into automating configuration with either Puppet or Casper.

When we initially rolled out a couple of OS X virtual machines we noticed that CPU usage on the VMware cluster spiked up to almost a 100% while the virtual machines were idle. It turned out that the default OS X screensaver uses GPU power to generate that flare effect and because not enough GPU memory was available it resorted to using up all the CPU. Disabling the screensaver or switching to a text based one quickly fixed that issue ...

[William] - Thanks for the excellent tip on OS X screensaver, this is a handy one to know about! How do you go about monitoring the Mac Pro infrastructure? What’s the process for replacing failed hardware components and have you had any challenges with this?

[Vitaliy] - We treat it the same way as the rest of our environment -- each vSphere node and virtual machine is monitored via Nagios. We have this cluster running for little over a year now and luckily we have not had to deal with any hardware failure.

[William] - For your OS X Virtual Machines, do you have a need for backups or a DR strategy? If so, could you share some details on what you are currently using?

[Vitaliy] - We have a replica of our production environment at a remote disaster recovery site and we use vSphere Replication to copy all the VMs nightly. We also heavily rely on the snapshot feature prior to making any operating system or application changes, it has been a lifesaver so far.

[William] - Vitaliy, I want to say thank you very much for taking some time out of your super busy schedule to have a chat. Before I let you go, do you have any words of wisdom for others looking to manage a similar infrastructure? Anything you would do differently and any resources you have found useful in aiding you to support a VMware / OS X infrastructure?

[Vitaliy] - Speak to your manager, legal department, or whoever is in charge about interpreting Apple EULA. I have heard of at least three different interpretations and all have legal implications. I am very happy with our environment and would not change a thing if I had to build it again. Your blog, virtuallyGhetto, has been a great resource as you are the only one talking about VMware products running on Apple hardware.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.