xserve

ESXi 6.0 on Apple Xserve 3,1

11.17.2015 by William Lam // 76 Comments

A couple of months ago, I shared a guest blog post from one of my readers John Clendenen who was able to get ESXi 6.0 running on an Apple Xserve 2,1. At the end of that article, it was hinted that John was also looking into getting ESXi 6.0 running on an Apple XServe 3,1 and you can the details below after several months of investigation.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

First an update on my Xserve 2,1’s. I had them running for over 100 days without any issue! However, now that I have the 3,1 working reliably, it is time that I part ways with my Xserve 2,1’s. I currently have them up on eBay. Here is the link: http://www.ebay.com/itm/231752771080?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649

Anyway, onto the Xserve 3,1.

I came across an Xserve 3,1 on eBay about a year ago. It was badly photographed, and the seller didn’t really know what he/she had. It wasn’t getting much attention, so I thought I might get it cheap. I ended up paying $500 for it which I felt ok about, but not great.

When it arrived, it had no processors, heatsinks or airflow duct. I immediately messaged the seller, and was able to get $350 refunded to me. I found the missing parts for under $100 over the next few weeks, and developed an intimate understanding of the Xserve 3,1 hardware.

At this point, I had no familiarity with vSphere at all. I was running OS X server and virtualizing a few services in Fusion. It was only through researching the Xserve 3,1 to find the missing hardware that I discovered that VMware had supported once as an ESXi 5 host. This made me wonder if it might still be possible to run ESXi on it, despite it no longer being supported.

I have found, after a considerable time investment, that the Xserve 3,1 can run ESXi 6, just as I found the Xserve 2,1 can run ESXi 6. However, unlike the Xserve 2,1, the Xserve 3,1 took months of troubleshooting before I had it running as a reliable ESXi host.

As it turns out, despite how much time it took me to get it working, there are only 2 serious issues with the Xserve 3,1 running ESXi 6. The first is somewhat specific to my configuration, but the second will be relevant to all configurations.

The first issue concerns booting into ESXi on a headless Xserve 3,1. The issue is limited to configurations where ESXi is booting from a drive installed in the optical bay (my original configuration). I have since changed my configuration and swapped the ESXi boot drive from the optical bay to the first hard drive bay. I have had no issue since I made this change.

For my configuration, I used an OWC bracket to replace the optical drive with an SSD. I installed ESXi onto it without issue. During installation, it was connected to monitor, keyboard, etc. I ran some VM’s on it to make sure it worked, and there were zero issues. I was relieved! So, I put it in the rack, wired it up and turned it on. Nothing. The Xserve lit up, and it was clear that it got through POST, but ESXi was clearly not booting.

Long story short, when no monitor is plugged into the Xserve 3,1, it will not automatically boot into ESXi if the boot drive is installed in the optical bay. The Xserve boot options can even be programmed through the front panel, but no configuration will make it reliably boot from the optical bay when a hard drive is installed. It is truly baffling, and if anyone has some insight here, or if it is a problem specific to my particular Xserve, I would love to know.

The solution, in my case, was to plug a keyboard into the Xserve, and hold down option for a few minutes while it boots (bringing up the boot options). Once all LED activity has normalized and the fan has settled down, I released the option key and pushed the arrow buttons. I think you only need to push the up button, but I always just pressed all of them to be sure. Then I pressed enter, and ESXi will boot. I have since simply swapped the boot drive to the first drive bay. Ideally, I’d have the other drives in the hot-swap bays, but I felt it was too much trouble to keep it in the optical bay.

The second issue concerns the onboard NIC. Once I had ESXi up and running, everything worked fine for anywhere between a few hours and 2 days, after which the Xserve 3,1 host would disappear from the VCSA and become completely unresponsive (no ping/ssh/etc). The length of time before failure made this issue especially difficult and time consuming to diagnose.

After nearly a month of frustration and disappointment, I determined that ESXi actually continued to run, but all network connectivity was ceasing. The only solution I have found is to install a 3rd party NIC and completely avoid using the onboard NIC. Even in standby, the onboard NIC can cause problems, but when it is completely unused, both for management and VM traffic, it no longer causes any problems.

This has been superficially improved with the last update, but use of the onboard NIC should still be completely avoided. The ESXi host will remain accessible via the VCSA, but the network management will become grayed out after a day or so. I suspect this is a driver issue in ESXi, but I really do not know.

Beyond these 2 issues, I have had no problems. Since the last update, even the performance and hardware status tabs are functional. RDM is not available, but not recommended in the first place. The Apple RAID backplane will not be recognized, but this was even the case in ESXI 5 when it was officially supported by VMware.

I hope that my efforts here will save others a lot of time and frustration. I think that for a lot of IT infrastructures, ESXi on an Xserve might make sense. It can run non-critical OS X services (which are hopefully the only kind of services you’re trying to run in OS X).

Summary

Completely avoid using the on-board NIC. Silicom NIC’s are recommended.
Find a standard backplane. The RAID backplane is useless in ESXi.
A 2.5” drive can be installed in the optical bay, but booting from it is problematic

The Xserve 3,1 with the Silicom NIC installed

The 6 ports are a tight squeeze, but they just fit. My other 2 EXSi hosts are Supermicro Nodes, also with Silicom NIC’s and I had to use a Dremel to grind off part of the chassis to make all the ports accessible. But the Xserve works out of the box.

The OWC SSD “Data Doubler” bracket in the optical bay. Booting from here is a pain, but putting an additional SSD here works great for host caching.

The standard backplane is difficult to find, but is a great asset for vSphere. It is easy to distinguish it from the RAID backplane which would have a heat sink here.

There are no complications during installation/initial configuration.

Apologies for not having a longer uptime. I updated to ESXi6.0U1a 12 days ago, but I’ve had the Xserve 3,1 up for months. If something changes, I will post an update here, but I am confident that the system is stable.

This is the final stage of my home lab. The Xserve 3,1 is 1 of 3 ESXi hosts. These are accompanied by a primary domain controller (Samba4), a media server (Emby) and a home-grown NAS (Centos7). Networking in the back is Ubiquiti. I use this lab to prototype production environments for clients, and of course to run my home media services 🙂

ESXi 6.0 on Apple Xserve 2,1

08.20.2015 by William Lam // 9 Comments

I really enjoy hearing from my readers, especially when they share some of the unique challenges they have come across or boundaries they have pushed with our VMware software. Several weeks back I received an interesting email from a reader named John Clendenen who was able to get ESXi 6.0 running on both his Apple Xserve 3,1 as well as Xserve 2,1! To be honest, I was pretty surprised to hear that this worked and not only that, there was not a whole lot that John had to do to get it working. I thought this was a pretty cool setup and asked if John was interested in sharing more details with the VMware/Mac OS X community in case others were interested.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

For the past 5 years, I have lived in New York where I work at various print and post-production studios. The IT situations in many of these locations is often home-grown and sub-adequate, if not barely functional, so I took it upon myself to learn OS X Server administration. I have a background in computer science and networking, so it wasn’t a huge leap to step into administration. I was able to gradually build one studio’s IT infrastructure form a single AFP share with incessant permissions problems, to an Open Directory driven, single sign-on infrastructure with mobile homes, messaging etc.. However, with each OS X Server update, things broke, and I was spending a lot of time putting out fires.

This led me to pursue virtualization in hopes of isolating services on separate machines, without telling the studio to go buy 8 Apple Mac Minis. However, I needed to learn how to do this, before pitching it to them, so I looked on Craigslist for some cheap hardware. I found an Xserve 2,1, which I was able to talk down to $100. I also found a brief post in a thread that said that the Xserve 2,1 ran ESXi 5.5 without issue. I figured I’d take the plunge and just resell it on eBay if it didn’t work out.

Since then, my home lab has grown considerably, and I have learned that the best way to provide Mac services, is simply to use Linux (I’ve had a great experience with Netatalk). That said, Open Directory, Software Update, Caching and a few other services still need to be run on a Mac, so it’s still necessary to have the hardware. For a while, I had 3 Xserves, all purchased very cheaply, running VMs. I just sold one, and will sell another in the next month or two in favor of some Supermicro hardware (I really am mostly running Linux VM’s at this point). I’ll keep the one to run a few Mac OS X VMs. I’m still working on getting Samba 4 to work as the PDC, but once that is running smoothly, I’ll have a functional environment that I can take to the studios where I work (with approved hardware of course). Then I’ll have an improved work experience, while also pulling some extra income on the installation/maintenance.

Anyway, you’ve come here to read about running ESXi 6.0 on an Xserve 2,1. Well, it works. It’s 7 years old, and not officially supported, but you already know that. So, if you were considering it, I doubt anything here will dissuade you. On top of that, there’s not much for me to say other than, it works. No tricks to it, just install and you’re off.

That said, I do have some recommendations and tips to help you if you want to get the most out of this hardware. 2 months ago, since I swapped out the RAID card for the standard backplane/interconnect and upgraded the RAM. It hasn’t skipped a beat since.

My System

This is my home custom 13u rack with sound insulation and active air flow. Networking is in the back (Ubiquiti). It sits in a lofted storage area with an A/C and vents into the bathroom.

Here you see an Xserve 3,1 on top of an Xserve 2,1. There’s a blank unit because I just sold an Xserve 2,1 on eBay, and the other 2,1 will be for sale soon as well to make room for a 4 node 2U Supermicro Server. The NAS is comprised of a 4u head unit and a 4U JBOD running Centos. Last of course, is a 2U CyberPower UPS which is really just to condition the power and keep brownouts from taking down the system.

I have about a dozen VM’s running between the 2 Xserves. I have each Mac OS X service separated on it’s own installation. This way I can run updates per service. It’s especially nice to have Open Directory separate from less important services. I also have Debian, Centos and OpenBSD VMs running various services. Getting the Xserve 3,1 running ESXi 6 is possible, but more problematic. Now that I have it working, I’m dropping the 2,1’s simply because the processors aren’t multithreaded. I am currently working on a companion article to this one, detailing my experience with the Xserve 3,1, so that information will be available soon.

Storage

1. Don’t use the RAID backplane/interconnect or count on using it. ESXi 6 does not recognize it, and RDM appears to work at first and then will crash your VM and never show up again. You can have it installed in the Xserve without any issue, but you’ll get a lot more mileage out of the hardware if you have the standard backplane/interconnect.

The backplane you want appears periodically on eBay. The part number is: 661-4648

2. If/once you have the non-RAID backplane/interconnect, keep in mind that it is SATAII and will only support 3Gb/s. I am using 3 old 500Gb WD RE3’s, but I’d recommend using some older SSDs that will max out the SATAII interface without paying for speed you can’t use. Be sure you consult the Apple Drive Module compatibility page to make sure you have the right drive caddies. They all fit, but they don’t all work.

Apple Drive Module Compatibility Chart: https://support.apple.com/en-us/HT1219

3. PCIE Flash is a good idea, whether you use it for cache, VM’s or as the ESXi boot disk, it is by far the fastest storage option. I have not invested in it, but the good people at Mushkin have told me their Scorpion PCIE flash will work in ESXi 6. Please contact them yourself though before investing in one of their cards. I have not tested them. While this will give you the best performance out of your Xserve 2,1, it seems like overkill to me, but hey, if you want to push this thing, you might find some meaningful performance gains here.

Mushkin Scorpion PCIE Flash: http://www.poweredbymushkin.com/index.php/products/solid-state-drives.html

4. It might occur to you that replacing the optical drive with an ssd might be a good idea. While MCE Technologies makes “Optibay” for Xserve 2,1, the connection is IDE, so this is not recommended. I also don’t know if ESXi would recognize it. My gut says probably, but again, it’s too slow to be useful. It isn’t that cheap either.

MCE Technologies Optibay: http://store.mcetech.com/mm/merchant.mvc?Store_code=MTOS&Screen=PROD&Product_Code=OBSXGB-XS

Network

There are a lot of options here. You can plug any network card that you want really as long as you can at least find an anecdotal account of it working on ESXi 6.

I have one such anecdote for you, and it is also a strong suggestion. The company Silicom makes/made several models of gigabit NICs which are all incredibly inexpensive, are all over eBay and work in ESXi 6. Buy the 6 port model. It’s cheap and you’ll get great scaling with ESXi load balancing across 8 gigabit ports.

The Xserve 2,1 has the added advantage here of an optional PCIX riser. If your model has one, or you find one for cheap, you can save even more on the Silicom NIC. The PCIE models go for $60-$80 on eBay, while the PCIX models go for $40.

Memory

ECC DDR2 is pretty cheap and easy to find used. I recommend memory4less though. I had to return some RAM that was mislabeled from a random eBay distributor. Memory4less will get it right. http://www.memory4less.com/

Processors

One great perk of the Xserve 2,1, is that you can upgrade the single processor hardware to dual processor. You can pick up an additional processor on eBay for $40 or so, but you’ll need to get a heat sink as well. The single processor units come with a fake aluminum heat sink, but do not use it. You want a copper one. I believe the heat sinks in the Xserve 1,1 are the same. Don’t forget the thermal paste.

Minor Issues

1. The Performance tab throws an error.

2. Not sure about the hardware sensors, but it looks like not everything is working even if it’s showing up. I did not do any testing here.

Stay tune for Part II of John's guest blog post on running ESXi 6.0 on Xserve 3,1.

Community stories of VMware & Apple OS X in Production: Part 6

09.10.2014 by William Lam // Leave a Comment

Company: Public Education K-12
Software: VMware vSphere
Hardware: Xserve

[William] - Hi Pete, thanks for reaching out on Twitter and offering to share your experiences in managing VMware and Apple OS X in an academic environment. Can you start off by quickly introducing yourself and what your role is currently?

[Pete] - My name is Pete Wann, I've been a sysadmin for over 15 years, mostly in education. I switched to Mac at the OSX transition because I was really interested in the Unix (BSD) foundation. My interest in Unix was piqued by my exposure to Solaris in the military, and since then I've tried to focus my career around all the various flavors out there, it just so happens that I like Macs, and it's been a good niche to be in. The community is awesome and ridiculously supportive.

My current role is as a Principal Systems Technologist at Oracle. I work for our Global IT group, but I primarily support a subset of our Marketing department. I'm responsible for the infrastructure around our video, print, and web production efforts. Although, the specific implementation we're going to discuss was done at my last position, with a large school district in Alaska.

[William] - Thanks for the background Pete. So I hear you were involved in an implementation that involved VMware and Apple OS X Technologies, can you share with us some more details about the environment?

[Pete] - Well, as you know, Apple discontinued the Xserve in 2010. (boo! hiss!) This was disastrous for that environment since the schools were very far apart, and our WAN links were slow and sometimes tenuous, in addition to some decisions made before I arrived about how home directories were handled, we needed to have some kind of server presence in every school. Since we couldn't count on having someone in each school who was comfortable going into a server closet to reset a system, we really needed Lights-Out-Management on whatever hardware we put out there.

Additionally, this was by far the largest Open Directory deployment that we (or Apple) had ever heard of. We had both computers and users in OD, and with our sometimes rickety WAN, we needed to have OD replicas as close to the clients as we could get, so again, a server presence in every school.

Eventually we migrated all of our user authentication over to AD, but still used OD for some computer management functions (mostly we used JAMF Casper for imaging and package deployment), so we still needed separate OD replicas for each school. (Each school was its own OU within OD so that we could distribute computer management tasks.)

[William] - I too remember the EOL announcement of the Xserve, it definitely had an impact to everyone who relied on that hardware. It sounds like you had a decent Apple Infrastructure, where was all this running? Physical or Virtual?

[Pete] - At the time, ESX did not support the Apple RAID card, so I could not use the internal storage with any of the systems I had available, which was fine with me, since I didn't want any moving parts on the hosts if I could avoid it, to hopefully increase longevity.

So, after much bugging of the powers-that-be, I got three licenses for vSphere for the three Xserves I scrounged from our secondary schools, removed all internal storage, then installed ESXi on a small USB drive on each host. I used the built-in iSCSI support in ESXi to connect to our NetApp storage, and integrated the Xserves with the rest of our vSphere environment, with full support for vMotion and everything. It was really easy, and worked insanely well.

We wound up virtualizing about 20 hosts across the three Xserves, mostly OS X, but also a couple of Linux hosts to act as web front-ends for our Casper environment. I fought hard to make the Xserves full-fledged members of the vSphere deployment, but my counterparts on the Windows side resisted harder. I still think that was a waste of available CPU power, but such is life.

[William] - Wow, this is pretty cool! I think this is the first implementation that I have heard of that leverages external storage w/Apple hardware. Could you share some details about the hardware specs for the Xserve and how you came to this particular configuration?

[Pete] - Well, in the case of the Xserves, we lucked out by having already ordered 77 of the last generation before Apple announced the end-of-production. We were in the process of transitioning from Xserve G5s to Intel in all the schools.

I was at the MacTech conference in LA when word came out that the Xserve was killed (Can you imagine the mood in that room?) and immediately got in touch with my boss to ask for as many more of the last generation we could afford to buy. Initially my intention was to go with Parallels Server, and we did buy it and deploy it at a couple of sites, but let's just say that didn't go well, and I jumped off that path as soon as ESXi 5 was released.

Initially I wanted dual-processor systems with the internal SSD and maxed RAM (I believe 48GB on that model), and since I was still thinking in terms of what Parallels Server supported, I got 3 internal 1Tb drives to use for local storage. Unfortunately, the option of adding the internal SSD as a fourth drive disappeared almost as quickly as it appeared, and we missed the window. I got the rest of what i asked for, though.

Once I discovered that ESXi 5 didn't support the Apple internal RAID controller, I had to find another solution for storage, since I didn't want to run everything, Hypervisor and VM Storage on USB drives. Fortunately for me, our vSphere environment was already configured to connect to our NetApp NAS, so it was trivial to add that storage for the VMs once the Xserves were added as hosts to the vSphere DC.

I also managed to scrounge additional NICs for the Xserves to give the nodes more network capacity for the guest VMs. So I think ultimately we wound up with 6 total 1Gb connections — 1 management, 1 vMotion etc., and 4 on a vSwitch for guest VMs. The three Xserves were segregated into their own vDC to avoid confusion for our management and SysAdmins.

[William] - How did you go about monitoring this infrastructure? Any challenges or gotchas you found while building and managing this environment?

[Pete] - Honestly, no. We used all of the same management tools that we used for our wider vSphere environment, and it all just worked.

At the time, I believe they were implementing some monitoring tools from Symantec, but I left while that was still being implemented. Before that was in place, it was largely a manual process. I stayed as hands-off as possible once I had my environment up and running because I take a "less is more" approach to being a SysAdmin. 🙂

The ONLY gotcha, and it was very easily overcome, was the lack of support in ESXi 5.0 for the Apple internal RAID controller. That turned out to be good for us, as it forced us to use the existing vSphere infrastructure.

As for management, we just had to embrace a new way of deploying VMs, but there again, once I built a template for vSphere, it was trivial to deploy new Mac VMs, which I then configured as needed. If we'd had a larger environment, I would have leveraged tools like Puppet or Casper to auto-configure hosts to our needs.

[William] - In building out this environment, it sounds like you learned quite a bit. Was this something you already had some experienced with or were you learning on the job? If the latter, were there any key resources you leveraged that helped you build and manage such an infrastructure?

[Pete] - I had experience with VMware from my previous job, where I got involved in deploying new VMware nodes to help transition to a virtual datacenter. In truth, it worked so well and was so easy to set up, I didn't really need support except for gathering the specifics of our environment.

There was literally no difference between the setup for generic x86 hardware and Xserve as far as I could see. The only difference was that in addition to all the other guest OSes, we could also run OS X on these hosts.

[William] - Pete, I would like to thank you very much for your time this afternoon and sharing with us your experiences. I think this has been very informative/educational and should help others thinking about building or managing a similar type of environment. Before we finish up, do you have any words of wisdom or advice to others looking to start a similar project and perhaps also working in the academic/education field?

[Pete] - I would say that if you're thinking about it and if you think that virtualizing OS X will help, then go for it. It's actually easier than you probably think. Also, I'd say to remember that as a SysAdmin, managing up is just as important as managing your systems. Keep your eyes open to what's happening in your industry, and try to be prepared for new things and opportunities to save money and improve efficiency. Especially in public K12, budgets are shrinking, but demands (particularly on IT) are increasing. Don't be afraid to speak up if you think you can find a way to save money and provide the same or a better level of service for your students.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.