A killer custom Apple Mac Mini setup running VSAN

*** This is a guest blog post from Peter Bjork ***

The first time I was briefed on VMware VSAN, I fell in love. I finally knew how I would build my Home Lab.

Let me first introduce myself, my name is Peter Björk and I work at VMware as Lead Specialist within the EMEA EUC Practice. I fortunately have the opportunity to limit my focus on a very few products and truly specialize in these. I cover two products; VMware ThinApp and VMware Workspace Portal and one feature; the Application Publishing feature of VMware Horizon 6. I’m an End-User application kind of guy. That said, you should understand that I’m far from your ESXi and vSphere expert. If you want to keep up with the latest news in the VMware End-User Computing space make sure to follow me on Twitter, my handle is @thepeb. When I’m not a guest blogger, I frequently blog on the official ThinApp and Horizon Tech blogs.

In my role I produce a lot of blog posts and internal enablement material. I perform many tests using early code drops and on a daily basis I run my home lab to deliver live demos. I need a Home Lab that I can trust and that supports all my work tasks. I started building my lab many years ago. It all started with a single mid tower white box, but pretty soon I ran into resource constraints. I started to investigate what my next upgrade would look like.

I had a few requirements:

• Keep the noise down
• Shouldn’t occupy that much space
• Should be affordable
• Modular, I do not have money to buy everything upfront so it should be something I could build on top of.
• Should be able to run VMware ESXi/vSphere
• Should be cool

Being an Apple junky for many years, my eyes soon fell on the Apple Mac Minis and I stumbled over this great blog by William Lam that you are reading right now. At the same time I started to hear about VSAN and my design was pretty much decided. I was going to build a Mac Mini cluster using VSAN as storage. While I have Synology NAS I only use it for my private files. It is not used in my home lab and for reasons I can not really explain I wanted to keep it separate and use a separate storage solution for my home lab.

Now that I have decided to build my home lab, I went and bought my first Mac Mini. To keep cost down I found two used late 2012 models with i7 CPUs. Since VSAN requires one SSD and one HDD I had to upgrade them using the OWC Data Doubler Mounting Kit. I also upgraded the memory to 16GB RAM in each Mac Mini. This setup gave me some extra resources and together with my old Tower Server I could start building my VSAN Cluster. I started with the VSAN beta. I quickly realized that VSAN didn’t support my setup. I waited for the GA release of VSAN and on the release date I decided to go for a pure Mac Mini VSAN setup so I stole my families HTPC which was a late 2012 Mac Mini model with a i5 CPU. (I managed to get away with it because I replaced it with an Apple-TV.) I took one HDD and the SSD from my old Tower Server and put it into the i5 Mac Mini. While I managed to get VSAN up and running it was only running for an hour or so before I lost one disk in my VSAN setup. I recovered the disk back up through a simple reboot but then the next disk went down. The reason for the instability is that the GA release of VSAN did not support the AHCI controller. Hugely disappointed I had to run my home lab on local attached storage and my dreams of VSAN was just that, dreams. In all my eagerness I’ve already migrated the majority of my VMs onto the VSAN Datastore so I pretty much lost my entire home lab.

After complaining to my colleagues, I found out that AHCI controller support for VSAN was coming in vSphere 5.5 U2. I heard it was likely to solve my problems. So the 9^th October came and vSphere 5.5u2 was finally here. To my joy, my three Mac Minis were finally able to run VSAN and it was completely stable.

Lets take a closer look at my setup. Below is an overview of the setup and how things are tied together.

My VSAN Datastore houses most of my VMs. My old Tower Server is connected to the VSAN Datastore but does not currently contribute any storage. On the Tower Server I host my management VMs. Since I got burned loosing all my VMs, I made sure I keep my management VMs on a local disk in the Tower Server. Since my environment has been running quite stable for nearly two weeks now I’m considering migrating all of my VMs onto the VSAN Datastore.

I have noticed one issue so far which is with my i5 based Mac Mini. One day it was reporting not connected in the vCenter Server. The machine was running but I got a lot of timeouts when I pinged it. While I was thinking about rebooting the host it showed up as connected again and since then I’ve not noticed any other issues. I suspect the i5 CPU isn’t powerful enough to host a couple VMs and being a part of the VSAN. When I saw it disappear it might have been running some heavy workloads. So with this in mind I would recommend running i7 Mac Minis and leave the i5 models for HTPC work loads :).

Another thing I’ve noticed is that the Mac Minis are running quite hot. There is no power saving functionality that is active and my small server room doesn’t have cooling. The room is constantly around 30-35 degrees Celsius (86-95F) but the gear just keeps on running. The only time I got a little bit worried was when the room’s temperature peaked at 45 degrees Celsius (113F), for Sweden, that is an exceptionally warm summer day. Leaving the door to the room opened for a while helps cool things down. I’m quite impressed by the Mac Minis and how durable they are. My first two Mac Minis have been running like this for well over a year now.

Here’s a picture of my server room. While I do have UPS there is no cooling or windows so the room tends to be quite warm. Stacking the Mac Minis on top of each other doesn’t really help cooling either. When I started stacking my Mac Minis on top of each other I realized how stupid it would be to have three separate power cords. So I ended up creating a custom Y-Y-Y-cable (last Y is for future expansion).

The power cord is a simple lamp cable (0.75mm2) that has the three original Apple power cables butchered together. The Y-connector was found in a local Swedish hardware store. Since the Mac Mini’s maximum continuous power consumption is 85W, a 0.75mm2 cable would work perfectly. A 2 meter (6.56 feet) long 0.75mm2 cable is able to support at least 3A. My three Mac Minis only consume 1.1A (3x85W / 230V = 1.1A). In 120V countries you would have 2.5-3A running through the cable but this wouldn’t be a problem.

Since the Mac Minis only have a single onboard NIC and I wanted to have two physically separated networks I had to get an Ethernet Thunderbolt NIC. As shown in the overview picture I’m am running both VSAN traffic and VM traffic over the same NIC. This is probably not ideal from a performance point of view but for my EUC related workloads I’ve not noticed any performance bottlenecks. On the other hand, I’m very pleased with the performance and with the benefits of having shared storage, so things like DRS and vMotion can deal with the balance between my hosts and I’m super happy with this setup.

I found that the easiest way was to use VMware Fusion to install ESXi onto a USB key. Then I simply plug in the USB key in my Mac Mini and I’m up and running. I need to use external monitor and keyboard to configure the ESXi initially.

As for the next steps I’m planning on getting an SSD and an extra HDD for my Tower Server. This would allow my Tower Server to participate in the VSAN Cluster and contribute additional capacity. If the opportunity arises and I can find another Mac Mini with an i7 CPU for a decent price I would also like to replace the i5. Other than that, I don’t think I need much else. Well, I could always use a little bit more RAM of course (who doesn’t) but disk and CPU runs very low all the time.

Technical details:

• All Mac Minis are late 2012 models
  • All SSD disks are different models and vendors. Their capacity ranges from 120 to 250GB. Since I’ve had a couple SSD crashes I made sure to purchase the more heavy-duty models offered from the vendors. But none of them are designed for constant use in servers.
  • All Mac Minis have 16GB RAM (2x8GB)
  • I have 1TB HDD in my two i7 Mac Minis and 500GB HDD in the i5 one.

• ESXi installed on USB key

• My Tower Server specs are:
  • Supermicro Xeon E3 motherboard, uATX (X9SCM-F-B)
  • Intel Xeon E3-1230 3.2GHz QuadCore HT, 8MB
  • 4x8GB 1333MHz DDR3 ECC
  • Barracuda 500GB, 7200rpm, 16MB, SATA 6Gb/s

To wrap up, I’m very pleased with the setup I‘ve built. It works perfectly for my needs. Lastly, I do recommend having a separate management host, as I found it extremely useful when I had to move VMs back and forth to test earlier releases of VSAN. I also recommend going for the i7 CPU models of Mac Mini for better performance.

Download the VMware ESXi 5.5u2 Mac Mini ISO from virtuallyGhetto:

• https://mega.nz/#!EJNSFJyb!hm-AWAiqEisDnMV9XpZphSLn_puJLu9RTep9R83N6rY

Apple Thunderbolt Ethernet Adapter VIB:

• https://s3.amazonaws.com/virtuallyghetto-download/vghetto-apple-thunderbolder-ethernet.vib

UPDATE (01/15/15) - Peter just shared with me a new custom Mac Mini Rack that he had built and welded together, check out the pictures below to see what it looks like.