WilliamLam.com

Community stories of VMware & Apple OS X in Production: Part 3

by // 2 Comments

In this third post of the community stories, I had the pleasure to interview with Blake Garner who shares with us how he leverages VMware and Apple Technologies in his production environment.

Company: Adobe
Product: VMware Fusion
Hardware: Apple Mac Mini & Mac Pro

[William] - Hi Blake, thanks for your comment on Twitter today, it sounds like this blog series might be useful for your organization. I understand that you have some experiences working with VMware and Apple OS X that you can share with the community? Before we get started, can you quickly introduce yourself?

[Blake] - Sure, I have been working at Adobe for 11 years in the Seattle location. Started off as a Dev/Test lab administrator focused on Macs, Printers & color management. Been working with VMware seriously starting with VMware Fusion 1.0 and Lab Manager 2.x.

[William] - Wow, a VMware old timer 🙂 So, I take it from your Tweet that you must be doing some really cool stuff with VMware and Apple OS X? Can you describe the environment and how you leverage these technologies together?

[Blake] - We provide "bare metal" Mac Mini's and Mac Pro's to our users. They are mostly developers and testers of Adobe desktop software. The process involves approvals and users requesting a Mac from our web portal.

Once users have the Mac they load VMware Fusion which is the majority of the time. We do not have a centralized command and control of VMware Fusion. A lot of the teams then hook up VMware Fusion to our testing harnesses. Things like Jenkins and a lot of custom code to manage the VM's. We looked into using ESXi/vSphere/vCloud on the Mac but due to the Apple EULA restrictions it just wasn't a good fit.

[William] - Very cool! This sounds like you’re offering Mac as a Service for your internal customers? How do you go about managing the requests of which Mac Mini or Mac Pro are leased out? Is all this custom software Adobe has built?

[Blake] - Yes we do both Mac and x86 systems (Windows & Linux) as part of our bare metal offering on our internal IT Cloud. The bare metal part is custom code. We have rolled our own deployment systems to go along with it as well. Users access one portal that currently spans vCloud, AWS & bare metal along with a few other services.

[William] - You mention that the majority of users install VMware Fusion on the “bare metal” once they get their assigned Mac Mini or Mac Pro. What is VMware Fusion being used for?

[Blake] - Lot's of automation. vmrun gets a good workout here. One team has a "Test as a Service" that can control VMware Fusion and rollback snapshots to provide a clean testing state. The consumers of the Mac's enlist them into their own existing automation systems. Often if you look at the VM's you will see Creative Cloud applications running through test cases super fast. Builds of products for Mac OS and iOS also happen.

[William] - Ah, so they are leveraging VMware Fusion as a platform to be able to run sort of a “Continuous Integration” build environment for your internal Mac OS X and iOS builds? This sounds like it could be quite challenging to manage, have the end users had any issues or have they automated everything all already?

[Blake] - In a larger software company you end up with a number of approaches. I did not get involved at the build level enough to speak on that. Adobe IT tends to focus on the common services that all the teams can use. In general there is a lot of automation herding going on..

[William] - Gotcha. So, going back to physical Mac Mini and Mac Pro, roughly how many are you managing and what type of configurations did you spec out? Any particular reasoning for choosing these configurations?

[Blake] - Our initial launch of the bare metal mac service has 50 Mac Pro's and 50 Mac Minis split equally between two sites. This service is just a couple months old and we expect it to grow fast as the engineers figure out they can get Mac's if their manager approves a monthly fee to their cost center.

We have one config for Mac Pro and Mac Mini. The Mac Mini 6,2 has 250GB SSD and i7 2.6GHz with 16GB RAM. The Mac Pro 6,1 6-Core E5 3.5 GHz with 500GB SSD, 16GB RAM and lower end GPU's as those are not used that much. We really just looked for a sweet spot that matches what Creative Cloud needs

Here is a picture of one of the Mac Pro racks courtesy of Blake:

adobemacpro
[William] - How do you monitor for hardware issues and what is the most common issues have you seen for both the Mac Mini and Mac Pro?

[Blake] - That is a real challenge with Apple hardware. With no out of band management we rely on our staff to troubleshoot via KVM or in person if needed. Pre-release software can really crash a system and that often needs a finger on the power button. I'm looking into using our remote controlled PDU to power cycle systems via the portal. If a system is truly FUBAR we just give people a new one.

In a lot of cases we can simply re-image the system via netboot and add it back to the available pool. VMware Fusion comes really in handy for developing our custom netboot environments as well. Netbooting an OS X VM's is one of my favorite features of VMware Fusion.

[William] - I can definitely see it being easier to re-image than to troubleshoot unless you are seeing a consistent issue. Curious, what version of Mac OS X and VMware Fusion are you using today? In addition to snapshot & Netbooting, any other useful features VMware Fusion provides?

[Blake] - VMware Fusion 6.0.3 is what we support today. On the hardware we encourage people to use the latest release version of OS X. In the VM's it depends on what kind of testing. Validation of pre-release OS's is a big chunk of the testing work. For OS X that happens in VMware Fusion and other x86 it's done in vCloud and some VMware Workstation. The automation capabilities of VMware Fusion are a key component. That's what gives us the upper hand in managing these virtual systems.

[William] - Hey Blake, I really appreciate you taking the time and sharing with us on how Adobe leverages VMware and Mac OS X. Before I let you go, do you have any words of advice for other customers looking to provide a similar type of environment? Any gotchas or things you would change if you could start fresh?

[Blake] - Go talk to those people who have offices crammed full of systems and find out what they are doing. You can often find the common requirements and start building against that. Don't try and dictate the whole solution to engineers. Once they are happy customers it's much easier to get them onboard with centralized services. Provide a set of functional services and engineers will pick it up quickly. Keep adding services and they will grow along with you.

Things that I would change all end up on our features roadmap. I have my eyes on providing an API to access to re-image or reserve bare metal systems and providing vagrant along with VMware Fusion for automation.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

 

Community stories of VMware & Apple OS X in Production: Part 2

by // 1 Comment

After sharing VMware's story on how they leverage Apple Mac Mini's for their OS X build infrastructure, I thought it was only fair to reach out to Yoann Gini to see if he would also like to share some of his experiences working with VMware and Apple OS X. I was able to catch up with Yoann and you can find our chat transcript below.

Company: Fortune 500
Product: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Yoann, I appreciate you taking some time out of your evening to share with us some your experiences working with VMware ESXi and Apple OS X. Your recent tweet was really the motivation behind this series, so thank you. Before we dive in, can you quickly introduce yourself?

[Yoann] - I’m a french computer scientist, working as a freelance consultant and trainer on Apple products for Enterprise and Education. I also work on network architecture and security, doing reverse engineering for fun in my spare time. All Apple OS X focused. You can find more details on my website.

[William] - Awesome. So, based on your tweet, I assume you have some experience working with Mac Mini's and VMware vSphere? Can you share with us some of the customer environments you have been in and how you have solved the challenges leveraging vSphere?

[Yoann] - Yes, I have two main setup with vSphere at this time (and my lab). One with 10 Mac Minis hosting up to 20 OS X VM which is basically building agent for an iOS forge for a Fortune 500 company (I can’t tell the number of iOS project build on it). The other one with three Mac Mini hosting two VM, one for Open Directory, DNS, File Sharing and the other for e-mail serving around 500 users.

[William] - Wow, Mac Mini's really being used in a Production environment! How cool! What was the reason for selecting the Mac Mini versus an Xserve or Mac Pro? How did the customer react to using a non-supported platform? Were there any challenges?

[Yoann] - When these two projects started, the Xserve was already stopped, so it wasn’t an option. For Mac Mini vs MacPro, it was only a matter of reasonable risk versus unreasonable cost. Mac Mini is unsupported by VMware and Apple as a virtualization node, but it’s really cheap and, it works. Mac Pro is supported, but it so expensive with the following challenges:

- don’t fit in a server rack
- can’t be exploited at 100% (especially the new Mac Pro with super duper graphical card totally useless for most server jobs)
- really can’t be exploited at 100% if you read the Apple EULA who seems to don’t allow us to run more than 2 (or maybe 3) Apple OS X per Mac hardware…

The last point is that the most important decision for one of my customers: buying expensive hardware officially supported can be OK if at least we can run a lot of Apple OS X VM on it. But the Apple limitation is a real PITA when you try to develop Apple OS X Server and Virtualization in the Enterprise. It so stupid that in at the end, customers prefer to place the same amount of money in multiple Mac Mini instead of one good Mac Pro. It allows hardware redundancy for the same price + an iSCSI storage and it leverage the risk due to unsupported hardware.

For me, the real challenge is here, the legal imbroglio with Apple legal things (and contacting Apple SE about this subject does not help, the only answer is, ask your legal department).

They also have other challenges: IT against everything with an Apple on it. It always fun to start a meeting telling the team in charge of Virtualization that they will have to support a non-supported small form factor system without a redundant power supply. But we always find a solution, Apple Consultants are used to this situation. It's a common denominator to all OS X and iOS deployment in enterprise.

[William] - Interesting, so it looks like the Apple EULA played a pretty large role in the organization's decision. At this point, you have selected the hardware platform and you knew you were going to Virtualize on vSphere. Can you talk a little bit about the applications, was this a new environment you were building out or was this a migration from an existing infrastructure?

[Yoann] - For the iOS forge, it was a new environment. The system was a Java based application and a pilot has been done in the past. So blank page here. A project leaded by company needs increasing with iOS software demand. For the more traditional server setup with all internal services like directory service, DNS, mail, etc. It was an existing setup on dying Xserve. We’ve done the migration on vSphere to take away all hardware problem (we’ve got more and more disk failure and random problem on the Xserve in the end).

[William] - For the environment which you had to migrate your existing Apple OS X systems running on the Xserve, what type of tools did you leverage? Were there any tips and tricks you used or things people should look out for if they are attempting a similar migration?

[Yoann] - We’ve taken the opportunity of hardware to Virtualize the systems and  migrate to a newer system version. So we’ve just followed the recommended migration path in this situation. We’ve installed a new system on the vSphere setup and then we’ve imported our data inside with a combination of directory export/import feature and rsync for files.

It was really simple with Apple OS X Server, you just have to ensure that your directory service is there and then put all the data in the good place before starting every services.Another option is use common Apple OS X imaging system like DeployStudio or Carbon Copy Cloner to create a image from your existing system and deploy it on your virtual system.

Is not as simple as vCenter Converter but when we’ve done our “state of the art” migration, we’ve got only a 5 min shutdown on a Sunday morning. All linked service like TSE, Citrix, Cisco Call Manager and custom app haven’t seen any thing. Only a reboot needed for Windows based system.

[William] - Very nice, it sounds like you got the process pretty much nailed down. How about after everything has been migrated over to vSphere. How does the customer manage the environment, are they running vCenter Server or are these stand alone systems?

[Yoann] - In this setup, we have a vCenter Server and we use the vSphere Web Client to handle it. By the way, it work like a charm from Safari on OSX, no more needs of Windows VM on our Mac to manage the setup and create new VMs.

[William] - I am with you on that, I too used to run a Windows VM just to use the vSphere C# Client. I’m glad I can use the vSphere Web Client on my Apple OS X system to manage my vSphere environment. In terms of Apple OS X guest management, how do you go about handling that and how do you go about provisioning new Virtual Machines?

[Yoann] - Just like any other Mac hardware, since ESXi supports NetBoot, I can use my existing provisioning system for free. I know that vSphere include some provisioning system to create VM on the flight when needed but I didn’t have the time to play well with it. At the end, Apple OS X VM are just like real Mac with HA in addition, I use all pre existing system without a change. It can even simplify my deployment (no need of Xsan and Load Balancer for HA for example).

[William] - Yoann, these are some great tips! I wanted to thank very much for taking the time and sharing with us your experiences with running Production Apple OS X workloads using VMware vSphere and Apple Mac Mini’s. Before I let you go, I wanted to ask if you had any recommendations for others looking to either Virtualize their existing Apple OS X deployments or looking to building out a new environment using VMware?

[Yoann] - Yeah, talking about HA, it remind me existing setup I have. I have some customer setup I’ve created and I still maintain who use Xsan (the Apple’s cluster file system) with Barracuda Load Balancer in front of two or more OS X Server to handle HA for all services (web, file sharing, databases, etc.).

It works but it’s hard to maintain and definitively not accessible for un-experienced system administrators. If I had to do it again, this kind of setup will end directly on a vSphere system with Fault Tolerance and things like that. It will be cheaper in so many ways (iSCSI instead of Fibre Channel, less time consuming, no need to have advanced knowledge on all network protocols, no need to play with clustered system like MySQL Cluster who’s a really PITA to make it work, etc.).

I also considered deploying free ESXi for all new setup, whether it is a Mac Mini or Mac Pro. The only challenge is that there is no vCenter Server with Free ESXi and you would need a Windows VM to be able to use the legacy vSphere C# Client. If you want or need to use the vSphere Web Client, you would need a vCenter Server license. However, the vSphere Essential Kit is not that expensive and it make sense for SMBs.

With this kind of a setup, it is really easy to manage: simple to deploy a new VM, simple hardware redundancy and can easily be expanded in the future. Keeping everything simple. Need to add a Windows server for accounting? Add a VM. Need HA? Add a Mac Mini and iSCSI storage. No service interruption.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

 

Community stories of VMware & Apple OS X in Production: Part 1

by // 4 Comments

I caught this tweet from Yoann Gini a couple of weeks back which I thought was quite interesting:

After sharing this tweet internally on our Socialcast group related to all things Apple, I came to learn that Yoann was not the only one running Production workloads on Apple Mac Minis'. It turns out, we at VMware also use Mac Mini's for a very special Production environment. This actually got me thinking about how other customers are leveraging VMware and Apple OS X in their environment? Would it not be cool to hear about how others leverage VMware and Apple Technologies together in their production environments?

This was the primary motivation behind this blog series, the idea is to interview folks from the Virtualization community willing to share their experiences and educate the community on how they leverage VMware and Apple OS X in their Production environments. To help kick off this series, I would like to start off by sharing how VMware leverages vSphere and Apple OS X in our own Production environment. I got in touch with the person responsible for managing this environment and below is our chat transcript.

Disclaimer: The Apple Mac Mini platform is not officially supported by VMware

Company: VMware
Product: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Michael, thanks for taking some time out of your day to share some information about how VMware uses Apple Technologies. For those that do not know you, can you introduce yourself and your role within VMware?

[Michael] - Certainly. I'm Mike Lemoine, I've been at VMware for just over three years, and my title is Senior Tools and Infrastructure Engineer. I'm part of the Build/SCM team here, responsible for the care and feeding of the infrastructure that is used for VMware product builds.

[William] - Build/SCM, that sounds pretty cool! So this is the Infrastructure that Engineering uses to compile and build out the various VMware (Apple) installers, executables and binaries that customers eventually consume?

[Michael] - Yes, it is. While engineers can and do perform local builds on their desktops, or shared interactive environments, the only 'real builds' are the ones that go through this infrastructure. If it didn't go through our machines, it doesn't get released to the world.

[William] - Sounds like a very critical piece of Infrastructure at VMware. So Michael, I heard from someone that you manage a very special build infrastructure at VMware and it involves some vSphere and Apple hardware? Do you mind telling us a little bit about this environment and what it is used for?

[Michael] - I do, indeed! We have a fleet of Apple Mac Minis serving as the basis of our OS X build farm. While they're not on the HCL, they're really our best option for providing an environment for products intended to run on OS X or IOS.  While the Mac Pro is supported, it has a lot of unnecessary equipment which makes it prohibitively expensive (well, wasteful) to use at scale. The Mini, on the other hand, has most of what we need. It's not perfect, but it's the best match we can accomplish without violating the Apple EULA.

[William] - Wow, that is an awesome use case for the Apple Mac Minis! Even though the Mac Mini’s are not on the HCL, they are being utilized for Production workloads and building out products like VMware Fusion, Horizon View Client and iOS applications. Can you tell us a little bit more about the environment, number of hosts, version of ESXi and the amount of capacity it can support?

Here is a picture of the Mac Mini Cluster in the VMware Datacenter. The rack that is used to hold the Mac Mini's is MMR-2G-5URS along with MMR-2G-2URS for brace stabilizer.

vmware-apple-mac-mini-build-cluster
[Michael] - We've got roughly 50 Mac Minis in production (a mix of 5,3 and 6,3 depending on when they were ordered), running ESXi 5.5. Each is stuffed full with the maximum supported config. In the case of the 6,2 minis that's i7 at 2.6ghz and 16G of memory. That's one of the lamentable issues with the mini, that it maxes out at 16G. We run two VMs on top of them, each taking their own spindle and 8G of memory. Our 6,2 minis are presently running 10.8.4 VMs, while the 5,2 minis are running VMs with older versions of OS X for build reproducibility.

[William] - That is a lot of Mac Mini power! Are these ESXi hosts currently being managed by vCenter Server or are they managed individually?

[Michael] Both, actually. Those ESXi hosts are all managed by vCenter, and our build system uses an in-house inventory and lease system to choose among available hosts.  One of the reasons that we can confidently run these Minis in production is due to our automation being written for failure. We assume systems will be wedged, we assume every machine is a landmine. In reality, the Mac Minis very rarely give us any trouble, but knowing that we can lose nearly all of them and still produce builds gives us a sense of safety.

[William] - That is a very cool solution. It sounds like the Mac Minis have been rock solid for our Production usage, but as a backup we still have intelligence built into our software so we can safely rely on the use of the “consumer” hardware. Sounds like a mini SDDC to me! Speaking of hardware failures, what components of the Mac Mini have you seen fail the most and how do you go about getting the parts replaced?

[Michael] - Certainly. We're all about belts and suspenders; nobody wants that 2am call about a production outage. The issues we've had with the minis has been the death of the drives in them.  These machines are very rarely idle, which is a usage pattern that the drives in them simply aren't prepared for. Amusingly, our support for the Minis is the same as anyone else's.  One of the guys in our datacenter will pull the machine and take it to the genius bar, where they will tell us what we already know and replace the drive. The system will then be re-racked, the new drive set up as a datastore, and we deploy a new VM to that drive. Our Minis are all running ESXi off of USB sticks, so losing the drive isn't much of a hurdle.

[William] - Ah, that’s cool that we’ve built that into the design and can easily tolerate host failures and rebuild is not really a big deal. So what about the VMs that were running, is there any data that needs to backed up and restored or is it stateless configuration?

[Michael] - Nothing that needs to be backed up or restored. The systems are all configured via puppet, so all that's necessary is the base OS installation, which we have created a template for. The only manual step involved is running puppet the first time. Then we perform a test build to be sure everything is in order and put the system back into production. The total time a human spends on this once the system is re-racked is probably under ten minutes.

[William] - Are there any plans in the future to upgrade the Mac Mini Cluster to using the new Mac Pro’s or do you see Mac Minis being more than sufficient?

[Michael] - We've looked at the Mac Pro, but the extra cost of the dual GPU makes them expensive. If there was a Mac Pro option with cheap onboard video, we'd probably tolerate the inferior form factor and see what we could do for racking them. We still wouldn't have BMC, but we'd gain a gigabit network port.  

[William] On the topic of support, does the Mac Mini not being officially supported by VMware have any impact on you?

[Michael] - The mac mini not being supported certainly has an impact on us. Our internal experience of support is already inferior to the customer experience. Having to fight the battle of "No, we don't support it, but we unofficially do" adds a cost to every interaction.

[William] - My understanding is that the primary reason for not supporting the Mac Mini’s today is their lack of support for ECC memory. I know that our customers expect an Enterprise ready solution when we certify a platform, but in your opinion is this a requirement we potentially could reduce and do you feel customers would agree?

[Michael] - I think that if we communicate the concerns to our customers and allow them to make their own decision on whether to take the risk of running consumer-grade hardware, it would be better for everyone. Customers would feel more secure with the off-label use, we already do the work in-house to make the Mini a usable platform, and the cost seems very low. I think the only challenge would be clearly enumerating the dangers.

[William] - Michael, I really appreciate you taking the time and sharing with our customers on how VMware leverages the Apple Mac Mini’s for Production. Do you have any tips or tricks you would give to our customers looking at running vSphere on the Mac Mini for more than home labs? Are there any go to resources you would recommend customers to if they are looking to get started with running ESXi on a Mac Mini?

[Michael] - My advice would be to follow our lead. Realize you're using consumer-grade hardware, and plan for failure. The low cost allows for easy redundancy, take advantage of that. In what other situation can you have an entire spare server on hand for $1200? While the information available on the internet is great, and I spent more than a little time on virtuallyghetto.com reading about mac mini attempts, I also had the ability to annoy and harass some of our talent inside VMware to get answers. To be honest, the amount of information you need in order to get ESXi running on a Mac Mini would probably fit on an index card. The challenge is hunting down which gotchas there are for your combination of ESXi version and mac mini revision.  A KB article covering the few pitfalls in the process would be wonderful.

Hopefully you enjoyed this first post in the series and stay tune for a couple other interviews that I am working on. In the meantime, if you are interested in sharing your story on how you use VMware and Mac OS X in Production, you can reach out to me here.