WilliamLam.com

Community stories of VMware & Apple OS X in Production: Part 9

by // Leave a Comment

Company: Connell Insurance
Software: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Bryan, thanks for your time this afternoon. I know you have been pretty busy these last couple of weeks and I am glad we got some time with you to chat about your environment. Before we get started, can you please introduce yourself and what you are currently responsible for?

[Bryan] - Hi William. Thanks for the interest! My name is Bryan Linton. I'm currently the IT Director for Connell Insurance, an independent Insurance Agency in Southwest Missouri. We’ve been around for a long time but have been growing more rapidly in recent years. We currently have around 40 employees.

We have most of our systems in a secure off-site datacenter, but I still need some support systems onsite, and that's where your project bringing together inexpensive Mac Minis with ESXi caught my attention.

[William] - Can you tell us a bit more about your Mac Mini infrastructure? What is the hardware and software configuration and the type of workload you are currently running on them?

[Bryan] – First of all, I didn't buy the "server" versions of the Mac Mini - I just ordered the standard Mac Minis with stock RAM and storage. My only exception to going stock was on the CPU. For that, I went all-out and ordered the i7 quad-core processors since I knew I’d be using them as servers. But I bought a large SSD, a data doubler kit that allowed me to mount the second hdd, and 16GB of RAM, and installed all those myself, since it was cheaper than spec’ing it that way from Apple.

I also bought a 16GB low-profile USB flash drive to use as my install point and boot device. It's working fine booting and running ESXi 5.5. I’ve always used ESXi embedded on my servers going all the way back to version 3.5, so I was already comfortable and familiar with booting and running ESXi from a flash drive. So to summarize the hardware, that's a quad-core i7, 16GB of RAM, a 1TB spinning HDD and a 500GB SSD as datastores, booting ESXi 5.5 from a tiny flash drive that barely protrudes from its USB port.

On the software side, I downloaded your turnkey ISO for ESXi 5.5 for the Mac Mini. Your advice to enable support for the Thunderbolt Ethernet adapter was easy to follow, and that gave me two 1GigE NICs for whatever I need. The installation was about as simple as you can get.

My workload currently is very light. We just opened a branch office, so that’s where I’m using the Mac Mini. I installed a DC for the site, and have set up just one other support machine so far. It runs the management software for our video surveillance, alarm system remote administration, and the controller software for my WiFi access points. I’ll probably add (or move) other support roles to this machine before long. It's not working that hard and to be honest, is probably faster than the 6-year-old Xeon-based server currently filling the support-system role inside our main office. The two offices are very well-connected, so I can move around support systems almost without concern for which site uses them most. So I do foresee loading up the Mac Mini with more work.

Here is a picture of Bryan's setup:

bryan-mac-mini
[William] - What was the deciding factor on choosing a Mac Mini versus Mac Pro or any other platform? Given the Mac Mini is consumer grade hardware and is currently not a supported platform, were there any architecture decisions you had to make on the infrastructure or application side to accommodate for this fact?

[Bryan] – The Mac Pro is fantastic hardware, so if you need heavy-duty power behind your VMs it's certainly worth looking at. But if you really beef up a Mac Pro you're back in the cost realm of what server hardware typically costs. I honestly didn’t look at the compatibility or support of the Mac Pro with ESXi because we didn't need that kind of power for our new branch office.

As for other platforms considered, we’re largely a Dell shop. We use their small-form-factor desktops for our user workstations. I considered using one of our retired workstations as a “server”, but was afraid it would be too slow, even with an SSD, and it didn't have any more RAM capacity than the Mac Mini. Plus it's obviously bigger and less power-efficient, and it’s probably less hardware-compatible with ESXi. That’s why we ultimately chose the Mac Mini.

The main limitation of the consumer-grade hardware in the Mac Mini, for us, was RAM. The Xeon processors in my "proper" servers running in our CoLo aren't ever taxed nearly as much as the other compute resources, so to me the i7 quad-core CPU seemed more than adequate. Having the RAM maxed out at only 16GB, though, made me put extra thought into how I can make the best use of ESXi’s transparent page sharing between VMs. We’re mostly a Windows shop, so it made sense to me to standardize on a single Windows Server version for a given Mac Mini, and try to keep all those VMs at the same patch level. That way, the odds are better that many of the core OS pages in RAM will be identical, meaning the use of TPS will increase, and more RAM will be available to run applications (or even more VMs). We chose Server 2012 Standard as the go-to Windows server OS, and I look forward to loading up the Mac Mini to see just how far ESXi’s advanced memory management techniques will let me push it.

The other constraint was network connectivity. For example, I have just 2 NICs, and I'm currently in the process of working out a backup strategy. I think I'm going to use a backup appliance running on the Mac Mini with a relatively inexpensive NAS as the backup destination, and if needed I can dedicate a physical NIC to that, but with only two NICS, I have to think a lot about network traffic management. I have a couple of other ideas that involve using a thunderbolt dock and/or USB 3.0 NICs to increase the number of USB ports and NICs available to me. The big question there is driver support in ESXi, but I haven’t yet researched or tested those ideas at all.

[William] - You mentioned the current workload is pretty light and you plan to deploy additional Window Servers. What type of workloads will these Virtual Machines be running? Do you see a need to scale up your Mac Mini infrastructure from what you have today?

[Bryan] - I'm not yet ready to put critical production data on them. Not until I have a better feel for what backup looks like, and until I really push the limits of the hardware resources. But support systems like I mentioned above - Wi-Fi controller, NVR for surveillance (with the data stored off-box), things like AV management, Spiceworks, syslog servers, additional DCs to provide a global catalog server, a DFS namespace server, and maybe a replica file server synced with DFS replication are all good possible candidates for running on a Mac Mini.

We have an email archiving system that has to run somewhere. Its data is stored on a remote share and DB, but the processing can happen anywhere, and Mac Mini will handle it fine. That lets me keep resources free for the mission-critical apps that run on my "real" servers that DO contain our critical production data. If I can improve user experience by offloading non-critical support systems to the Mac Mini, there's less resource contention on my vCenter-managed hosts, and user responsiveness should benefit from that.

I also have software firewalls that I use to create a "double-router, Dual-NAT" environment so I can run or build machines in a test, isolated environment, with internet connectivity, with the IP addresses they use (or will use) in production without conflict. They run in perfect isolation. The Mac Mini can host those software firewalls along with any machines that are either being built, or perhaps being restored from backup for exploratory purposes, or even for testing of upgrades or new software in a mirror environment before it goes into production.

As for scaling up, so far I haven't considered adding a Mac Mini ESXi host to my vCenter environment. But I might consider that if two things happened:

  1. Apple starts supporting more RAM in the Mac Mini.
  2. VMware decides to support the Mac Mini hardware officially.

Actually, as I get more time and experience using this setup, number 2 may diminish in importance. Time will tell.

[William] - Your last reply was quite interesting. You mentioned you have not considered adding your Mac Mini ESXi hosts to vCenter Server? I’m curious to hear why the support of Mac Mini would dictate the ease of centralize management? Is it from a support standpoint that you did not want to do this or additional licenses?

[Bryan] - My Dell ESXi servers are managed by vCenter, but I have three of them, which is my license limit currently. If I had a free slot for the Mac Mini I'd certainly use it. But I'd have a hard time justifying the purchase of *additional* licenses for vCenter to bring machines under management that aren't even officially supported. But yeah - if I had the licenses I'd have no hesitation in managing them via vCenter. I get around it currently thru the use of shared data stores.

[William] - Bryan, it was really great chatting with you and thanks again for sharing your experiences on how you have leveraged VMware and Apple Mac Mini’s in your production environment. Here is the last question that I have asked all my past interviewees, do you have any final advice or words of wisdom for someone looking to embark on a similar journey? Any particular resources you would recommend for someone to start with?

[Bryan] - Well, virtuallyGhetto is THE place to go for those resources. Without your pages I would not have made the leap. My advice would be: Don't expect it to do what a $5,000 investment will do. If you plan to run mission-critical apps or host production data, KNOW what your backup and recovery process looks like and TEST it.

If you understand you're striking out somewhat on your own, and you don't mind being a pioneer for the fun of it, do your due diligence, and if it seems like a fit, enjoy it! It's honestly fun to show people my rack. "That's our server. It's running ESXi 5.5." "...Really?!?"

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

 

How to build custom ESXi ISO for Apple Mac Mini?

by // 43 Comments

For those of you who own an Apple Mac Mini 6,2 may recall some of the, lets call them "challenges" on getting ESXi to run on the Mini. These challenges range from known SMC issues from Apple to missing or updated Broadcom tg3 network drivers. Though there workarounds for these issue, the process was quite complex. I took it upon myself to help simplify it by building custom ESXi ISO's for most of the major ESXi releases so that users could simply install ESXi as they normally would and by-pass all the complexity.

This has worked for the majority of folks but I have received several requests for those that may not be comfortable with just downloading a random ISO on the internet, which I can fully understand. The other reason is that some folks would like to build their own custom ISO and include other drivers/packages and others are just interested in the process. This has been on my to-do list for awhile but it was finding the time to document the process but also I normally like to take it a step further and see how I can make it even more simple 🙂

Disclaimer: Running ESXi on an Apple Mac Mini is not officially supported by VMware, please use at your own risk

With the recent release of vSphere 5.5 Update 2, I thought this would be the perfect opportunity to show how you can build your own custom ESXi ISO to run on the Apple Mac Mini 6,2.

Note: Earlier versions of Mac Mini should work fine for the most part without additional modifications.

Before I get started, I would also like to mention that several of the "challenges" such as having an updated Broadcom tg3 drivers have been fixed in the latest ESXi 5.5 Update 2 release, so out of the box you will be able to see the on-board network device working as expected and Ethernet Thunderbolt will also be functional if you are using that device with no additional drivers being required. I have been able to successfully install the default out of the box ESXi 5.5 Update 2 ISO from VMware on my Apple Mac Mini 5,3 without any additional changes.

Here is the process for building your own custom ESXi ISO for your Mac Mini:

Step 1 - Download the ESXi ISO you wish to work with

Step 2 - You will need access to a Linux system (recommend CentOS) that has mkisofs utlity, which is used to author an ISO

Step 3 - Download my custom.tgz which will automatically handle the SMC issue for Apple Mac Mini 6,2

Step 4 - Download my ghettotize-esxi-iso.sh which is a shell script that will automatically take an ESXi ISO and author a new ISO containing the fixes. The script is pretty straight forward and you can take a look at the script for all the details.

Here is an example of running the script against the latest ESXi 5.5 Update 2 ISO:

build-custom-esxi-iso-for-mac-min-0
As you can see at the end of the script, you should get a new authored ISO with a -NEW in the filename:

build-custom-esxi-iso-for-mac-min-1
Once you have the new ISO, you can then take that and load that onto a USB device. I like using unetbootin which is a handy utility that is supported on all platforms and creates a bootable USB device with the ISO provided. As you can see the process is pretty straight forward and though it took a bit of "experimentation" on my end to make it completely seamless, you can see there is too much to the process in general.

Community stories of VMware & Apple OS X in Production: Part 4

by // Leave a Comment

Company: AutoTrader.com
Product: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Chris, good afternoon. I know we have chatted a few times on Twitter before but for the folks that do not know you, can you quickly introduce yourself and what you do?

[Chris] - My name is Chris Nakagaki and I work for Autotrader.com as Sr. Systems Engineer. My current role involves day-to-day operations of VMware vSphere products in addition to defining best practices around the virtual infrastructure. Not to mention, help drive automation in my organization. Occasionally, I'll post something useful on my tech.zsoldier.com blog.

[William] - You had sent me an email after I published the first community story around how VMware leverages Mac Mini’s. I hear you are doing something pretty cool with the Mac Mini’s as well for your organization? Could you share some details on how your organization is using VMware and Mac Minis?

[Chris] - A couple of our subsidiary companies, in this case (vAuto and AutoTrader.com) essentially needed OS X VM's for QA testing of iOS applications, and general Mac browser testing. Rather than delivering individual Mac Mini to every developer and/or VMware fusion, etc. vAuto first approached us and we came up with this idea of clustering some Mac Mini's. They wanted to run ESXi on them to host relatively small VM's that could be centrally managed and accessed from any number of developers and with the Apple EULA, this was the only option due to the restriction. Besides that, it was just a really cool idea since we're all Mac/Apple fans anyways

[William] - That’s awesome, never seen a customer come up with both the request and a solution at the same time 😉 Have you had any experiences running vSphere on the Mac Mini’s before? Any challenges you faced while exploring this solution?

[Chris] - Thankfully you (William Lam) had run into a lot of the problems for us already.  So it was really easy to setup using the custom ISO and VIB you created to put our little 'MacCloud' together. The other non-software aspect we ran into though was the fact that the Mac Mini's do not have an out of band management interface. So we are currently looking for some smart power supplies and/or iKVM so that we can actually place these in our 'real' datacenters.

[William] - Hey no worries, I rather be the guinea pig and get all the kinks out so customers like yourself can just enjoy the benefits of running vSphere and ESXi on Apple hardware! How large is the MacCloud right now and what is the current hardware and software configuration?

[Chris] - Our MacCloud is only 3 Mac Mini's right now as we're still kind of 'feeling' it out. But soon after vAuto started using it, the word got out and we setup some test systems for our AutoTrader.com developers. In addition, our client engineering group uses a tool called Casper to manage our Macs. He needed a Distribution Point, preferably a system that had AFP, so we set him up with one and he was able to use it to deliver updates/applications. And the I/O for VSAN is so good with the SSD's, it screams.

Each Mac Mini is the 'Server' version, i7@ 2.6GHz, 1 SSD (128GB), 1HDD (1TB) and 16GB memory using USB to boot into ESXI. For the software, we are currently using vSphere Enterprise Plus and the vCenter Server happens to be the VCSA. The MacCloud is also being monitored by vCOPs

Here is a picture of the front/back of the Mac Mini rack:

autotrader-mac-mini-1

autotrader-mac-mini-2
[William] - That’s amazing, it sounds like the environment is really satisfying your developers and I can see why word has spread. So, did I read that right? You are currently using VSAN on the Mac Mini’s!? How has the performance been and what made you decide to leverage VSAN?

[Chris] - The minute VSAN went into beta, that's all I could personally think about for my own home lab with the Mac Mini's. That just naturally translated when the business had a need and I could satisfy my curiosity in one fell swoop. So being that these workloads aren't heavy I/O, I haven't been all that concerned with it. The VM Storage Policies have been all left at default because I don't see a need to change right now.

My team and I are actively keeping a pulse on all the users of the VM's hosted on here though. vCOPs shows that everything is working efficiently, but we want to make sure that is translating to a good user experience. The Casper DP is one in particular that I'm curious about since the disk I/O profile on that one should be a bit more consistent.

[William] - That is really cool to see customers already leveraging VSAN for their production usage and great to hear the experiences has been solid so far. You mention the use of vCOPs for monitoring the VMs, are you also using vCOPS to monitor the underlying Mac Mini’s and how do you handle hardware issues?

[Chris] - Honestly, right now, we're just relying on the vCenter CIM service to tell us if it finds a hardware problem. The first obvious problem I have with that is I'm not so sure it would notice a hard drive failure. Like VMware, we'd probably just bring it into a local Apple store and have any components still under warranty replaced. 'Normally' we have SNMP traps sent from vCenter to HP BSM. Being that this is such a small environment with lots of questions, we simply use vCOPs to alert us of any unusual behavior or problems. Many of of our vCenter alarms are 'self-correcting' alarms.

[William] - It sounds like your MacCloud is quite mature with so many different capabilities. Any plans in the near term to expand, I can already see more developers asking for similar setup? Will you be increasing your Mac Mini VSAN Cluster or potentially create a new one?

[Chris] - Most likely yes. My hope is that Apple and VMware will see the value in these community initiatives to hopefully make a 'support' Mac Mini with some native 10Gb capabilities. In the meantime, I can only see this growing to some really awesome potential.

[William] - Awesome to hear! Well, I do not want to take up any more of your time but before we conclude. Is there any tips or recommendations you would offer other fellow vSphere Administrators looking to run vSphere on Mac Mini? Any words of wisdom that you can offer?

[Chris] - Download William's ISO, upload and install the VIB from local system. Remote VIB install doesn't work because of static line (might be specific to windows) and last but not least. TRY IT!

[William] - haha. Thanks for the plug!

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.