Community stories of VMware & Apple OS X in Production: Part 3

In this third post of the community stories, I had the pleasure to interview with Blake Garner who shares with us how he leverages VMware and Apple Technologies in his production environment.

Company: Adobe
Product: VMware Fusion
Hardware: Apple Mac Mini & Mac Pro

[William] - Hi Blake, thanks for your comment on Twitter today, it sounds like this blog series might be useful for your organization. I understand that you have some experiences working with VMware and Apple OS X that you can share with the community? Before we get started, can you quickly introduce yourself?

[Blake] - Sure, I have been working at Adobe for 11 years in the Seattle location. Started off as a Dev/Test lab administrator focused on Macs, Printers & color management. Been working with VMware seriously starting with VMware Fusion 1.0 and Lab Manager 2.x.

[William] - Wow, a VMware old timer 🙂 So, I take it from your Tweet that you must be doing some really cool stuff with VMware and Apple OS X? Can you describe the environment and how you leverage these technologies together?

[Blake] - We provide "bare metal" Mac Mini's and Mac Pro's to our users. They are mostly developers and testers of Adobe desktop software. The process involves approvals and users requesting a Mac from our web portal.

Once users have the Mac they load VMware Fusion which is the majority of the time. We do not have a centralized command and control of VMware Fusion. A lot of the teams then hook up VMware Fusion to our testing harnesses. Things like Jenkins and a lot of custom code to manage the VM's. We looked into using ESXi/vSphere/vCloud on the Mac but due to the Apple EULA restrictions it just wasn't a good fit.

[William] - Very cool! This sounds like you’re offering Mac as a Service for your internal customers? How do you go about managing the requests of which Mac Mini or Mac Pro are leased out? Is all this custom software Adobe has built?

[Blake] - Yes we do both Mac and x86 systems (Windows & Linux) as part of our bare metal offering on our internal IT Cloud. The bare metal part is custom code. We have rolled our own deployment systems to go along with it as well. Users access one portal that currently spans vCloud, AWS & bare metal along with a few other services.

[William] - You mention that the majority of users install VMware Fusion on the “bare metal” once they get their assigned Mac Mini or Mac Pro. What is VMware Fusion being used for?

[Blake] - Lot's of automation. vmrun gets a good workout here. One team has a "Test as a Service" that can control VMware Fusion and rollback snapshots to provide a clean testing state. The consumers of the Mac's enlist them into their own existing automation systems. Often if you look at the VM's you will see Creative Cloud applications running through test cases super fast. Builds of products for Mac OS and iOS also happen.

[William] - Ah, so they are leveraging VMware Fusion as a platform to be able to run sort of a “Continuous Integration” build environment for your internal Mac OS X and iOS builds? This sounds like it could be quite challenging to manage, have the end users had any issues or have they automated everything all already?

[Blake] - In a larger software company you end up with a number of approaches. I did not get involved at the build level enough to speak on that. Adobe IT tends to focus on the common services that all the teams can use. In general there is a lot of automation herding going on..

[William] - Gotcha. So, going back to physical Mac Mini and Mac Pro, roughly how many are you managing and what type of configurations did you spec out? Any particular reasoning for choosing these configurations?

[Blake] - Our initial launch of the bare metal mac service has 50 Mac Pro's and 50 Mac Minis split equally between two sites. This service is just a couple months old and we expect it to grow fast as the engineers figure out they can get Mac's if their manager approves a monthly fee to their cost center.

We have one config for Mac Pro and Mac Mini. The Mac Mini 6,2 has 250GB SSD and i7 2.6GHz with 16GB RAM. The Mac Pro 6,1 6-Core E5 3.5 GHz with 500GB SSD, 16GB RAM and lower end GPU's as those are not used that much. We really just looked for a sweet spot that matches what Creative Cloud needs

Here is a picture of one of the Mac Pro racks courtesy of Blake:

[William] - How do you monitor for hardware issues and what is the most common issues have you seen for both the Mac Mini and Mac Pro?

[Blake] - That is a real challenge with Apple hardware. With no out of band management we rely on our staff to troubleshoot via KVM or in person if needed. Pre-release software can really crash a system and that often needs a finger on the power button. I'm looking into using our remote controlled PDU to power cycle systems via the portal. If a system is truly FUBAR we just give people a new one.

In a lot of cases we can simply re-image the system via netboot and add it back to the available pool. VMware Fusion comes really in handy for developing our custom netboot environments as well. Netbooting an OS X VM's is one of my favorite features of VMware Fusion.

[William] - I can definitely see it being easier to re-image than to troubleshoot unless you are seeing a consistent issue. Curious, what version of Mac OS X and VMware Fusion are you using today? In addition to snapshot & Netbooting, any other useful features VMware Fusion provides?

[Blake] - VMware Fusion 6.0.3 is what we support today. On the hardware we encourage people to use the latest release version of OS X. In the VM's it depends on what kind of testing. Validation of pre-release OS's is a big chunk of the testing work. For OS X that happens in VMware Fusion and other x86 it's done in vCloud and some VMware Workstation. The automation capabilities of VMware Fusion are a key component. That's what gives us the upper hand in managing these virtual systems.

[William] - Hey Blake, I really appreciate you taking the time and sharing with us on how Adobe leverages VMware and Mac OS X. Before I let you go, do you have any words of advice for other customers looking to provide a similar type of environment? Any gotchas or things you would change if you could start fresh?

[Blake] - Go talk to those people who have offices crammed full of systems and find out what they are doing. You can often find the common requirements and start building against that. Don't try and dictate the whole solution to engineers. Once they are happy customers it's much easier to get them onboard with centralized services. Provide a set of functional services and engineers will pick it up quickly. Keep adding services and they will grow along with you.

Things that I would change all end up on our features roadmap. I have my eyes on providing an API to access to re-image or reserve bare metal systems and providing vagrant along with VMware Fusion for automation.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

• Community stories of VMware & Apple OS X in Production: Part 1
• Community stories of VMware & Apple OS X in Production: Part 2
• Community stories of VMware & Apple OS X in Production: Part 3
• Community stories of VMware & Apple OS X in Production: Part 4
• Community stories of VMware & Apple OS X in Production: Part 5
• Community stories of VMware & Apple OS X in Production: Part 6
• Community stories of VMware & Apple OS X in Production: Part 7
• Community stories of VMware & Apple OS X in Production: Part 8
• Community stories of VMware & Apple OS X in Production: Part 9
• Community stories of VMware & Apple OS X in Production: Part 10