A couple of months ago, I shared a guest blog post from one of my readers John Clendenen who was able to get ESXi 6.0 running on an Apple Xserve 2,1. At the end of that article, it was hinted that John was also looking into getting ESXi 6.0 running on an Apple XServe 3,1 and you can the details below after several months of investigation.
Disclaimer: This is not officially supported by VMware, please use at your own risk.
*** This is a guest blog post from John Clendenen ***
First an update on my Xserve 2,1’s. I had them running for over 100 days without any issue! However, now that I have the 3,1 working reliably, it is time that I part ways with my Xserve 2,1’s. I currently have them up on eBay. Here is the link: http://www.ebay.com/itm/231752771080?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649
Anyway, onto the Xserve 3,1.
--
I came across an Xserve 3,1 on eBay about a year ago. It was badly photographed, and the seller didn’t really know what he/she had. It wasn’t getting much attention, so I thought I might get it cheap. I ended up paying $500 for it which I felt ok about, but not great.
When it arrived, it had no processors, heatsinks or airflow duct. I immediately messaged the seller, and was able to get $350 refunded to me. I found the missing parts for under $100 over the next few weeks, and developed an intimate understanding of the Xserve 3,1 hardware.
At this point, I had no familiarity with vSphere at all. I was running OS X server and virtualizing a few services in Fusion. It was only through researching the Xserve 3,1 to find the missing hardware that I discovered that VMware had supported once as an ESXi 5 host. This made me wonder if it might still be possible to run ESXi on it, despite it no longer being supported.
I have found, after a considerable time investment, that the Xserve 3,1 can run ESXi 6, just as I found the Xserve 2,1 can run ESXi 6. However, unlike the Xserve 2,1, the Xserve 3,1 took months of troubleshooting before I had it running as a reliable ESXi host.
--
As it turns out, despite how much time it took me to get it working, there are only 2 serious issues with the Xserve 3,1 running ESXi 6. The first is somewhat specific to my configuration, but the second will be relevant to all configurations.
The first issue concerns booting into ESXi on a headless Xserve 3,1. The issue is limited to configurations where ESXi is booting from a drive installed in the optical bay (my original configuration). I have since changed my configuration and swapped the ESXi boot drive from the optical bay to the first hard drive bay. I have had no issue since I made this change.
For my configuration, I used an OWC bracket to replace the optical drive with an SSD. I installed ESXi onto it without issue. During installation, it was connected to monitor, keyboard, etc. I ran some VM’s on it to make sure it worked, and there were zero issues. I was relieved! So, I put it in the rack, wired it up and turned it on. Nothing. The Xserve lit up, and it was clear that it got through POST, but ESXi was clearly not booting.
Long story short, when no monitor is plugged into the Xserve 3,1, it will not automatically boot into ESXi if the boot drive is installed in the optical bay. The Xserve boot options can even be programmed through the front panel, but no configuration will make it reliably boot from the optical bay when a hard drive is installed. It is truly baffling, and if anyone has some insight here, or if it is a problem specific to my particular Xserve, I would love to know.
The solution, in my case, was to plug a keyboard into the Xserve, and hold down option for a few minutes while it boots (bringing up the boot options). Once all LED activity has normalized and the fan has settled down, I released the option key and pushed the arrow buttons. I think you only need to push the up button, but I always just pressed all of them to be sure. Then I pressed enter, and ESXi will boot. I have since simply swapped the boot drive to the first drive bay. Ideally, I’d have the other drives in the hot-swap bays, but I felt it was too much trouble to keep it in the optical bay.
The second issue concerns the onboard NIC. Once I had ESXi up and running, everything worked fine for anywhere between a few hours and 2 days, after which the Xserve 3,1 host would disappear from the VCSA and become completely unresponsive (no ping/ssh/etc). The length of time before failure made this issue especially difficult and time consuming to diagnose.
After nearly a month of frustration and disappointment, I determined that ESXi actually continued to run, but all network connectivity was ceasing. The only solution I have found is to install a 3rd party NIC and completely avoid using the onboard NIC. Even in standby, the onboard NIC can cause problems, but when it is completely unused, both for management and VM traffic, it no longer causes any problems.
This has been superficially improved with the last update, but use of the onboard NIC should still be completely avoided. The ESXi host will remain accessible via the VCSA, but the network management will become grayed out after a day or so. I suspect this is a driver issue in ESXi, but I really do not know.
--
Beyond these 2 issues, I have had no problems. Since the last update, even the performance and hardware status tabs are functional. RDM is not available, but not recommended in the first place. The Apple RAID backplane will not be recognized, but this was even the case in ESXI 5 when it was officially supported by VMware.
I hope that my efforts here will save others a lot of time and frustration. I think that for a lot of IT infrastructures, ESXi on an Xserve might make sense. It can run non-critical OS X services (which are hopefully the only kind of services you’re trying to run in OS X).
--
Summary
- Completely avoid using the on-board NIC. Silicom NIC’s are recommended.
- Find a standard backplane. The RAID backplane is useless in ESXi.
- A 2.5” drive can be installed in the optical bay, but booting from it is problematic
The Xserve 3,1 with the Silicom NIC installed
The 6 ports are a tight squeeze, but they just fit. My other 2 EXSi hosts are Supermicro Nodes, also with Silicom NIC’s and I had to use a Dremel to grind off part of the chassis to make all the ports accessible. But the Xserve works out of the box.
The OWC SSD “Data Doubler” bracket in the optical bay. Booting from here is a pain, but putting an additional SSD here works great for host caching.
The standard backplane is difficult to find, but is a great asset for vSphere. It is easy to distinguish it from the RAID backplane which would have a heat sink here.
There are no complications during installation/initial configuration.
Apologies for not having a longer uptime. I updated to ESXi6.0U1a 12 days ago, but I’ve had the Xserve 3,1 up for months. If something changes, I will post an update here, but I am confident that the system is stable.
This is the final stage of my home lab. The Xserve 3,1 is 1 of 3 ESXi hosts. These are accompanied by a primary domain controller (Samba4), a media server (Emby) and a home-grown NAS (Centos7). Networking in the back is Ubiquiti. I use this lab to prototype production environments for clients, and of course to run my home media services 🙂
I wonder if adding a dummy dongle on the DVI port would allow for a workaround for booting without a monitor attached
http://www.amazon.com/CablesOnline-Graphics-Display-Detection-VE-D11/dp/B00N85QYGS
Using an HDMI one on my headless MacMini at home to work around a the graphics performance issue OS X has when there is no monitor connected.
Good afternoon, thank you for another useful news is always nice to hear from you something new about VMvare.
Curious. I have ESXi 6 (GA build 2494585) running on similar hardware, an Xserve3,1 with a single CPU, and I have no issues with the onboard NICs. It's been running for 161 days and been connected the whole time.
This is very interesting. Could you list the model and driver for the onboard NIC? I would be surprised if it's different, but the motherboard is not the same between the single and dual cpu Xserve 3,1's. This was a change from the 2,1's which used the same motherboard for both models (making the single core model upgradeable).
The dual processor Xserve 3,1 has an Intel 82574L NIC using the e1000e driver.
It's possible that my Xserve is somehow faulty, but I did have it running in OS X with no network connectivity issues using the onboard NIC for several months.
John, I'm running the same NIC, Intel 82574L with the same driver, e1000e. I only have vmnic1 connected, as I haven't set up a more complicated multi-network configuration yet.
Same as you, this was running OSX for months without issue. I convinced a colleague to let me put ESXi on it as it was just wasting its usefulness running a single instance of OSX. Strangely, the only issues I've experienced with it on both ESXi and OSX is the 1TB drive in the right-most slot (I think, C0:T2:L0) isn't recognised.
Have you tried wiping all the partitions on that drive? I had the same issue and it appeared after I wiped it. Do you have the ssd installed? That was the only thing I wasn't able to get my hands on.
That is the same nic and driver. Very strange that it works for you. Maybe mine is damaged. I wouldn't be surprised given the state I received it in.
We're running into the same problem as you, John - I'm guessing it comes into play when more than one NIC is being utilized. We're using all 4 (Our Xserve has the two-port add-on card in it) - two for iSCSI, one for VM traffic, one for vMotion. Goes dead to the world every 1-3 days. I'd be very curious what specific model of Silicom NIC you're using, so I can try to lay my hands on the same thing.
What model of NIC did you put in the Xserve to bypass the on-board?
PEG6i
There are a few on eBay right now. At a glance they look a little high. If you wait you might be able to find one for around $60.
Is the standard backplane part you reference in your discussion of the Xserve 2,1 the same part number to look for with the Xserve 3,1? I've been using a Mac mini as one of the hosts in my VMware environment but want more memory and network cards and am thinking of swapping the mini for an Xserve 3,1.
No, it is a different part, and unfortunately, it is hard to find.
One thing you might want to consider is using a 16 or 32 GB thumb drive in the front USB port (I use ones from sandisk which are tiny and barely stick out) for the ESXi boot drive. That frees up all 3 drive bays for VM storage, and since you can only go up to 2TB in each, every one available helps.
I got around the NIC problem by using a 4 port Intel card. I actually gave up on 6 and am running 5.5, but will probably have to go to 6 sometime soon to support newer OS X versions cleanly.
I have been running ESXi 5.1, then upgraded to 5.5, then upgraded to 6.0U3 and it's been online in production for years with no NIC issues with the onboard NIC. It's possible that the people experiencing issues with it have faulty boards since these are really old motherboards at this point. I've had many motherboards have failing components but everything else continued to function with an add in card in the past. (bad nic in one, bad sound card in one, bad onboard video in one).
On another note, has anyone had their VMs do funny things like hang up for no reason?
Also, I have a 2nd NIC for iSCSI + VM + Falut Tolerance traffic installed (Intel 6 port Pro/1000 or something). So I'm using the 2 teamed onboard NICs for management/vMotion and haven't had any issues with the hosts disappearing.
It's entirely possible that the issue was fixed by 6.0U3. I experienced the NIC issue in 6.0, and even in 6.0U1, it was partially fixed. Unfortunately, my license is now expired, so I cannot test beyond 6.0U1.
Hi,
So what do you use for the Datastore. I have ESXi running but it cannot see the internal Hard Drives as they are RAIDed do i need to reformat the drives not to use RAID or do i need to use an external NAS? Thank for any advice..
ESXi will not recognize anything plugged into the RAID backplane. You have to replace it with the standard backplane or use a NAS. In my lab, I have SSD's in the Xserve that are used for cache and my datastores are NFS.
Thank you that clears things up for me perfectly.
William, are there any tricks you've come across for getting ESXi to recognize a PCIe SSD card? It came from OWC and I was hoping to use it to put the VMs on for performance. Suggestions?
Jonathan Fletcher
Xserve 3,1 8CX 2.93 with EXSi 6 u2.
Jonathan, you ever get the PCIe based SSD from OWC to work? I am about to attempt the same. Thanks!
Nope. Sorry. Works great with Fusion on Mac OS, but I'm not nearly on the same level as Mr. Lam, so I don't really have any way to figure out ESXI. I gave up a while back. I don't even remember all that I tried. ::-(
No problem instig8r! Was just curious - I have gone ahead and made the leap to ESXi 6.5 with a clean install on my 2009 Xserves (3,1 model). No issues deploying at all. It even recognized my OWC 1TB SSDs (the Mercury Electra "3G" product) and am using local VM storage on them. 6.5 also perfectly recognized my 3rd party PCI cards (USB3 cards from NewerTech as well as ESATA cards from the same). This will allow me to use Pass-through if/where necessary. Pretty happy I moved off of ESXi 5.1U2, this is a WHOLE new world for me now. Just wanted others to know it works.
SomeDude, could you write that up and give tips about some good configuration options? Or can you point me to another one that somebody else has done? One of my issues was that the machine was locking up at one point in the install process, but I couldn't tell if it was a hardware or a software issue. If I have some instructions to follow, then I can tell better what I might have done wrong. Suggestions?
No instructions needed, I did absolutely nothing special to get 6.5 running...no hacks at all, just used the latest ESXi version from early Feb 2017 (the one currently posted on the VMWare site), created a bootable USB from it, plugged it into one of the rear USB ports on the xserve, booted with Option held down to choose the "EFI Boot" volume (the USB stick with ESXi installer on it), and then once booted to the installer, chose to install it onto a front-mounted tiny little 32GB USB stick that is nearly flush with the front of the Xserve. Installed beautifully, rebooted and came up with awareness of all hardware. No issues to report thus far, running about 6 VM's on one of them right now.
Yeah, I didn't have installation problems even back in the initial 6.0 release.
Did they fix the NIC issue where the onboard ethernet would crap out after a day or so of being on? It'd be nice to return to using those onboard interfaces for management.
They seem to have - I am using the onboard NIC's on my XServe's and no issues with connectivity from them. Been running all day long for 3 days anyway, no drops - time will tell, but so far so good.
Hey SomeDude you stated that "...It even recognized my OWC 1TB SSDs (the Mercury Electra “3G” product)" I tried 3 different OWC 1TB SSDs (Same 3g ones you have) in 3 different XServe3,1 (Non-Raid BackPlane) and no matter what I do the XSERVE3,1 does not see the drives. Anything you did differently?
LJS
I think only certain drive modules/caddies will work. I don't remember which ones though. I think the SAS models work if you have any of those.
Yes, I think John is correct, it must be a SAS model, such as this one for instance http://www.ebay.com/itm/661-5679-Apple-Intel-Xserve-Early-2009-2TB-SAS-ADM-Drive-Module-655-1563B-A1297-/172502233836?hash=item2829ef32ec:g:m38AAOSwA3dYioYw
Also, just to make sure, I'm not sure your proficiency level with ESXi, but just to be sure, you do know that you have to then find the hardware in the 6.5 UI, and format the drive as VMFS 5 or 6, right? I mean, the drive/volume is not just going to be 'seen' by ESXi, you have to actually format it. Assuming you know all that, I think I recall that 1 of the drives I had an initial problem with it being 'seen' by ESXi 6.5. I had to boot into a Mac OS partition (usb stick IIRM), and then format the drive as EXT4 from the OS X Disk Utility program. Once I did that, then moved back to booting into ESXi, I was then able to 'see' the drive in ESXi and proceed to format it as VMFS 6 as desired. I hope one of these things helps you. These 1TB SSD's literally saved me in so many ways, it really is night and day compared to the old Apple 7200RPM drives that these shipped with.
If you look at an enlargement of that drive, you will notice that it is a SATA drive. Like I said below, they are ALL SAS to the backplane, but the drives themselves can be SAS or SATA, depending on the module. So 99% of the SAS ADMs on eBay actually have connectors for SATA drives (well except for the other 99% that have IDE connectors, but those are for the older boxes). You have to look closely to be sure.
So, SomeDude, how do you get Disk Utility to format EXT4? That is not an option in my El Capitan. Do I need an older Disk Utility, or is there some trick? Which OS X did you use for that?
Guys, anybody ever get Pass-thru working with Mac hardware? Specifically on our XServes, or anything else for that matter?
Xserve2,1 had it in 6.0. No idea if it's stable though. Xserve3,1 is a no go.
Thanks John - now I'm wondering if I could get it to work on a 2012 Mac Mini to pass through the Thunderbolt port. ESXi itself runs beautifully of course on the Mini, but not sure about pass thru. Hmmmmm.
John/SomeDude
Thanks for the reply, yes I know the drives don't just present themselves. They are not even visible when I boot into OSX on the XServe. All the Apple Drive Modules I have are Part# 620-4091 SATA HDD.
Can I replace them with the SAS versions without replacing the backplane?
LJS
The backplanes are all SAS. It's just the drives that are different. The SAS to SATA drive modules are far more common and easier to find. The ones that take actual SAS drives have been eluding my searches, or already come with drives in them and are very expensive.
Ok, so I was able to dig this up: https://support.apple.com/en-us/HT1219
I remember the SAS modules working on Xserve2,1 but for 3,1 I think the 1TB MB838G/A is what you want.
Remarkably, it's still in the Apple Store https://www.apple.com/us-iphotoprint/shop/product/MB838G/A/apple-drive-module-1tb-sata-for-xserve
Also some on eBay...
Thanks for the link! I've never seen that, but I don't understand your point, John. That is a SATA drive, as you can tell from the link and from the Apple chart you posted to. Not a SAS. The ADMs for the Intel Xserves all have SAS connections to the backplane, but the drives themselves are either SATA or SAS, and the modules have the correct interface to mate the drive with the SAS backplane. THEY ALL look the same on the outside. If they don't have a drive in them you look at the cables for the drive. If they do have a drive in them you look for SATA logo or SAS designation on the drive's sticker.
Are you installing SAS drives? SAS is backwards compatible. The backplane supports SAS/SATA. Modules either support SAS/SATA or just SATA. SAS ADMs are also identifiable by an orange ribbon.
John, no I thought the OP of this portion of the thread was and you told him to use the SAS ADM. I'm thinking now that you meant for the SAS backplane and not for a SAS drive. If they are for the Intel boxes they are all SAS, so that's why I thought you weren't talking about that. But you simply meant not to use the ones that are for non-intel boxes. Where is the orange ribbon? I found a relatively inexpensive new SAS 1TB drive on OWC, so I looked around for a SAS module. They are definitely not the same as the cable to the drive is different. Not sure how one module could interface with both SAS and SATA. The cables appear to be very specific.
See above. I think 1TB MB838G/A is your best bet. Maybe someone else who has physical access to their Xserve can confirm. Mine is an hour away in a datacenter.
Do you have an Apple Part Number from an actual Apple SAS ADM? Like I wrote aboe the Part Number for the SATA version is: 620-4091 SATA HDD
LJS
There seems to be a bunch of part numbers for the carriers, but the ONE I have says 620-4575.
In another comment I made above I mentioned that you really have to look closely at the items' photos to make sure they have SAS connections to the drives to be sure.
Or actual SAS drives in them.
Yeah, that's the real issue hunting these down as most sellers don't even know what they have and are unwilling remove the drive to look for the part number ending with SAS HDD CARR
Just look at the sticker on the drive. It'll tell you if it's a SAS or a SATA drive.
Solved!!
Here's the magic part number TV823ZM/A
Just added: 2 of the SAS ADM TV823ZM/A -- It's Actually a Promise Part Number
1 OWC 1TB Mercury Electra 3G SSD Drive and it's working !!
1 OWC 1TB Mercury Extreme Pro 6G SSD Drive and it's working !!
So the magic is in the SAS Module which I will start testing for durability today.
Now if I could just rig the Xserve case to accept a 3rd Party Raid Controller, but the lack of additional power on the power supplies would make that messy.
LJS
Do you know what the original capacity of that SAS module was? A lot of times they're on ebay with the original capacity listed and not the part number.
It's an original promise part that comes with a 450GB SAS drive in the SAS ADM Part# TV823ZM/A.
I picked up a dozen of them to use in my 4 Xserve3,1 that will ultimately be running esxi 6.0U3
LJS
Very nice, great info!
Where did you find a dozen of them?
memory4less.com including the SAS drives which I don't need.
I paid $270@
LJS
After reading about the problems with onboard NICs and 6.0, I decided to go with 5.5. However, I just had one host go offline. So it looks like there could also be a problem with the onboard NICs on Xserve 3,1 under ESXi 5.5.
I am using both NICs, one for storage (physically distinct network), one for everything else (Guest VMs and management). The plan now is to get NIC card on the VMware compatibility list, and upgrade to 6/6.5.
I've tried installing 6.5, and had an issue installing to the Apple SSD, so might just go down the path of USB boot for ESXi as well, and all storage will be NAS/SAN.
Ciao
I have some problem to istal the software i put inside a bootable key but the xserve don't read it at the start, can you explane the procedure to istall it ?
Thank's for all.
Ciao
Sergio
I had the internal SSD working fine on my 3,1 Xserve with 6.0 but 6.5 is not able to partition/format it. I installed to a sandisk nub usb drive like others have done, but it still sees the SSD but can't use it. It does not see the other ADM drives. The ADMs are 620-4575 SAS HDD CARR. They were the 1TB modules so part number 607-4363. Do I need different ADMs or has someone gotten the 128GB internal SSD working in 6.5?
Thanks!
Tim
My ADMs are still working (with 3rd party SSD's in them), but I'm not sure what the part number is. I believe they are one of the 1TB SATA models. There used to be a compatibility matrix on Apple's site somewhere, but I can't find it anymore. I remember it was the one that had the most cross compatibility.
John, are you able to get me the part number on the standard backplane? All I'm finding on eBay are ones with two heatsinks and ones with one heatsink.
Never mind. It turns out the one I ordered from eBay is the standard non-raid one.
Glad you figured it out!
Sorry I couldn't get you that part number. My Xserve is in a datacenter an hour away. Is the part number printed on the board somewhere? It would be great if you could post it here for anyone who needs it in the future.
I was able to take a photo of the two part numbers. They are J5050001342GA and 630-9690. You can see the photo here https://imgur.com/gallery/6MaSf
anyone have a quick and painless way of cloning an esxi boot usb drive? (without using dd command as it takes forever) I want a backup incase my existing usb drive fails (I'm using it as the boot drive on my XServe3,1 running ESXi 6)
Thanks
LJS
Hi
Was wondering to get around the hard drive issue and the hard to find non-raid backplane, would PCI-E SSD would work with Xserver 3,1 and vmware Esxi 6.x (6.5 or even 6.7)
I have been looking for the non-raid interconnect backpanel for Xserver 3,1 part # 630-9690 and its truly hard to find, I cant afford to be honest the ADM Part# TV823ZM/A option
Though to ask if someone had any success with PCI-E SSD such as this one in Xserver 3,1 (Early 2009) and Vmware Esxi 6.x
https://www.ebay.com/itm/SEDNA-PCI-Express-PCIe-SATA-III-6G-SSD-Adapter-with-1-SATA-III-port-and-/173533849313?oid=262293141315
Thanks
It’ll probably work as long as it isn’t nvme.
I got one to work, so I guess my drive isn't NVME. I think it was an mSATA card on a small PCI card. It's been a couple years since I set it up and haven't had to touch it since.
I still have an xserve 3,1 in my 6.0u3 cluster running great. 8x 2.26ghz cores, 96gb ram, 8gb usb flash boot, built in dual 1g NIC, ISP2432 4gb 2 port FC 4x PCI card, and the performance kicker... an Intel Optane P900 NVMe 4x PCI card for virtual flash cache.
Hi
Were you guys able to get GPU passthrough on vmware esxi 6.5 and xserv 3.1? the one i have comes with builtin Nvida 960S, dont want to end up buying PCI GPU and it wont be supported on the xserv 3,1 (late 2009) or wont wrok
Also any suggestion on any tested GPU that is compatible with this module and supported for GPU PCI passthrough on vmware esxi 6.x
Thanks
Quick question for anyway using vmware esxi on xserver, how do you remote reboot or KVM to it if you have an issue? I know LOM works only if the xserver has OS installed on it
If anyone can advise would be greatly appreciated as i am facing this issue
Afaik, there's not a official oobm solution besides LOM. xserve does have a serial port, so you can plug that into a host and control it from there. You would probably need to do some configuration in ESXi to enable it though.
I believe I read somewhere you can use the LOM web interface as long as you turn it on BEFORE you install ESXi.
That is correct! I have LOM established for my ESXi hosts by doing so under OSX 10.6.x Server *prior* to nuking them and putting down ESXi install. It continues to work nicely. BTW, though I did this in Snow Leopard Server, you should also be able to do it via Lion or Mt Lion server also. Good luck!
I appreciate all the help, well I didnt have this setup prior installing Esxi and its now in production so i have no way to get this fixed right?
If anyone can advise how you get it from within Esxi that would be awesome to know, my next step if this wont work is probably buy a kvm over ip but i wont be able to remote reboot if i need
I have OS X 10.6 Server on a USB stick that I can boot from to make changes to LOM.
I do something similar. I am running 10.6 on a VM on the Xserve itself. I only boot it up when I need to make changes. I also have a copy of that VM on my MacBook in VMware Workstation.
You can pass LOM through to a VM? How does that work? All the hardware passthrough is grayed out for me.
John
Do you mind sharing how you did 10.6 on USB as i might be able to replicate that to get LOM to work
Ryan
Would be great to know how to get LOM to work through VM, i have Osx installed on a vm on xserve but its grayout, just to confirm it need 10.6 right or any other version?
It was a while ago, but I think I just installed to the USB through the normal installer. I had the DVD installer at the time.