Inquiries from customers on the support for ESXi on the latest 2019 Apple Mac Pro 7,1 has slowly been trickling in since the release of the system in late December. Officially, VMware currently does not support this platform and until we have a unit in-house to investigate further, this is the official stance.
With that said, several folks from the community have reached to me and shared some of their findings as it relates to ESXi with the new Mac Pro. A huge thanks goes out to Mike Rimmer who was able to go through the installation process and identified that the on-board NICs were not automatically detected by ESXI and the installation was unable to proceed. With the extensibility of the Mac Pro, Mike was able to add a supported Intel-based NIC to the system so that we could further understand the issue.
Upon closer investigation, it looks like the new Mac Pro uses two Aquantia based 10GbE NIC which is simliar to the 2018 Mac Mini which requires the Aquantia ESXi driver which was developed earlier last year.
AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]
Vendor ID: 0x1d6a
Device ID: 0x07b1
Although Mike did not have a chance to confirm this assumption, I did get validation from another customer who made the same observation when he attempted to install ESXi and once the Aquantia ESXi driver was incorporated into the latest ESXi 6.7 Update 3 image, both on-board NICs were automatically picked up by ESXi and installation was successful.
UPDATE (09/02/21) - Per this official blog post, VMware will no longer pursue hardware certification for the Apple 2019 Mac Pro 7,1 for ESXi.
UPDATE (04/28/20) - ESXi 6.7 Patch 02 resolves a number of the issues mentioned below, please take a look at this blog post here for more details.
UPDATE 1 (01/16/19) - Thanks to our Graphics team who was kind enough to loan me their 2019 Mac Pro which literally came in yesterday! I had an idea which I wanted to run an experiment on which was to add a PCIe card w/M.2 NVMe SSD and see whether or not the Apple T2 Security Chip would have any affect on whether or not ESXi would be able to see the device. I was not super optimistic but I had a need for an additional M.2 device, so I went ahead and purchased a $15 PCIe adaptor. I was pleasantly surprise to see that ESXi not only detected the device but I was able to format a local VMFS volume and power up a functional VM! I guess this makes sense as only the Apple SSD's are cryptographically tied to the T2 chip and other PCIe devices would not be and this would allow customers to take advantage of this system right now for running non-MacOS guests (yes, T2 still affects the SMC).
🔥 BOOM! 🤜🎤🔥
PCIe adaptor w/M.2 NVMe is NOT affected by the Apple T2 Chip! ESXi is able to see the device but more importantly, I was able to format local VMFS volume and power up a VM! Guess it makes sense, Apple SSD are cryptographically tied to T2#ESXiOnMacPro2019 pic.twitter.com/hod8Irckj9
— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) January 17, 2020
I also ran another experiment by connecting a Thunderbolt 3 chassis which also had a supported M.2 NVMe to see if I was going to be lucky again. Although it looks like ESXi 6.7 Update 3 has resolved the PSOD'ing issue, ESXi was not able to see anything on the other end.
Note: Secure Boot must be disabled on the Mac Pro before you can install ESXi, you can find the instructions in this Apple KB.
This was certainly some good news but like the 2018 Mac Mini, the new 2019 Mac Pro also ships with the Apple T2 Security Chip which has proved challenging for ESXi as mentioned here along with some known caveats. For now, I would hold off making any purchases of the new Mac Pro if you intend to run ESXi. VMware does officially support ESXi on the last current generation of Mac Pro 6,1 along with Mac Mini 6,2 and Mac Mini 7,1 which are all on the official VMware HCL.
I will continue to update this article as new information and findings are shared with me.
I think the biggest caveat is the inability to boot Mac OS vms due to T2 blocking SMC. I really hope apple starts helping out since the new rackmount form factor has some serious potential for vm xcode build and test farms. I'd really like to be able to replace our developers old minis with an actual rackmount system that could integrate with vsphere & our FC storage.
I'd put one I think significant caveat on the ' officially support ESXi on the last current generation of Mac Pro 6,1' comment at the end. We've been running three of them for several years, and they've always had a problem of seemingly randomly dropping their network connections, causing APDs, etc. But the problem I encountered when trying to contact VMWare support is that because the PCI IDs of those NICs aren't in the HCL, VMWare Support refuses to work on the case, and refers me to the 'hardware vendor' since they claim that entry into the HCL is wholly dependent on the vendor. Well, to my knowledge Apple isn't going to support ESXi on their stuff, so we're stuck.
Hi Kazuto,
We've had plenty of customers call in for Mac Pro 6,1 and in fact, MacStadium is a very large customer who's also got a huge fleet of these systems as they're quite popular as well. I know there were some support changes to educate GSS on the Mac Pro system as its not a common system that we get called about compared to the rest of the x86 systems. If you're having trouble, feel free to reach out to me directly via DM and we can make sure you're fully supported, as long as you've got valid SnS with VMware Support.
Cool. Can you install a vCenter Server Appliance VM on that ESXi host and then use that vServer appliance vSphere Client to build guest VMs on that ESXi host?
Thanks for research and investigation. Overall great news except:
>running non-MacOS guests (yes, T2 still affects the SMC)
Maybe it's possible to emulate SMC same way as QEMU and Proxmox does? Maybe some extension for ESXi to do so? Anyway we're ok with Apple license since it runs on Apple hardware.
Inability to run macOS guests makes whole ESXi on MacPro kind of useless
Hey, found it was very interesting and useful.
Thanks
MEDIASHAREiQ Team
https://www.mediashareiq.com/
I was able to get about as far as you have with the new MacPro and ESXi. Having the SMC not recognised by VMs is a bit of a deal breaker though. Do we know if there is any progress between Apple and VMWare or are both companies waiting for the other? There has been some progress getting Linux to see the internal ssd but I haven't seen any mentions of the SMC in these projects. As it stands with the Mac mini I can't see ESXi on the new MacPro anytime soon which is a shame as its really perfect for it.
Has anyone tried to install ESXi 7 on the new macpro ?
Thanks 🙂
Yes, I had tested this awhile back (pre vSphere 7 GA) and the behaviors are the same as described above.
Ok, bad news....
Thanks for your reply and all your great articles !
Hello William,
I installed a PCIe card with a 1Tb M.2 NVME as what you have done and the OS install won't continue and hangs at the error below:
More than one module named /sb,v00
More than one module named /s,v00
More than one module named /esxupdt.v01
Boot module signatures are not valid
I also created an USB 6.7 U3 with the Marvel NIC drivers. Is there anything I'm missing?
Not having as much success as others seem to be. I can't get patch 6.7patch 3 to install, I get a PSOD on boot. 6.7 patch 2 installs and runs fine, though.
I was able to get 7.0U1a installed, and I have a functional Catalina VM. Keep in mind - DO NOT install the 7.0U1b patch update - it screws with the SMC exposed to the guest OS, and the macOS guest will beachball.
I did follow the instructions above for the Marvell NIC drivers, but I just couldn't get the machine to pull an address from DHCP. I went out and bought the standard Intel 2 port SFP+ 10Gb fiber NIC (X540 I think?), installed that, works like a charm out of the box.
For the internal NVMe storage, I used a 4TB WD Black PCI card. Cost about $999 from Amazon, but I anticipated this and had the Mac Pro configured with the absolute minimum storage.
One thing that I'm unfortunately banging my head against here is the GPU pass through. Each time I enable a GPU on the guest, I start to get a reboot loop. @William Lam if you happen to read these comments, I could really use some help, and I've got a support contract... 🙂
Are you having problem on the aquantia aqc107 10g onboard nics on vsphere 7??
Halfway through May, 2021, I'm wondering if anything has improved WRT installing and running ESXi on Mac Pro 7,1.
Hi guys, I just installed 7.02 on my Mac Pro 7.1 with external nvme and used an external NIC unitec type c connector,
Next I’ll order an internal pci for the storage and network , any new updates from any one?
@Gili I've had great luck with the Western Digital AN1500 NVMe SSD Add-in Card. Performance is screaming fast. Last I checked, the 4TB model can be had on Amazon for about $1,000. VMware recognizes it on install.
Thanks @n3dwilson !
I will try it and also I need a good internal NIC for the system ( the internal was not working for me)
Gili
@Gili I always had issues getting the internal NICs to work, although to be honest, I haven't tried since the Atlantic drivers were included with ESXi out of the box. Startech makes a PCIe 10 Gigabit SFP+ NIC, based on the Intel 82599 chipset, that I have been using.
Out of curiosity, have you actually been able to get a macOS guest to install without a reboot loop? I ran into all kinds of problems with this, and sadly had to use the ESXi unlocker in order to get it to work, which is far from ideal.
Sadly , same here . I'm tried to install the unlocker because of the reboot loops but that didn't worked also.
the Atlantic drivers seem seems to be identified and connected but I didn't received automatics Ip (DHCP) so I'm working with a different usb NIC model.
@William any movement on this from the VMware side of things? I'd love to start testing again, but doesn't seem like a lot has changed in the last year or so. 🙁
I received a brand new Mac Pro Rack mount and see there is an internal USB port. In my other servers we install to SD cards on internal risers. Can one use that internal USB port to install ESXi onto a USB storage device so the entire internal storage is available?
I was able to get the 7.0U2a installer to run and during installation it saw the USB drive I had in the internal port and I was able to complete the install to it. However, I did notice that no other storage was found, like the internal SSD. And I have not found a way to get it to boot to that internal USB drive so any tips and pointers for that would be appreciated too.
Thank you in advance.
@simplijim please be advised, before you go down this road, that it appears that VMware has pretty much not been responsive to anything regarding ESXi on the 2019 Mac Pro for almost a year and a half. @lamw always shares when he gets new information, but sadly, there hasn't been much for a very long time. It is starting to feel like either VMware has decided not to pursue development or support of the 2019 Mac Pro, or Apple has sent them a cease and desist. It's pretty discouraging.
With that said - I would highly recommend installing an additional PCIe internal storage card. I've used a Western Digital Black 4TB PCIe card on three Mac Pro installs that I have here. You can set ESXi to boot off this internal device, and also use it for your local datastore.
The internal SSD is protected by Apple's T2 security architecture. I know that some folks have gotten ESXi to recognize it, but in my opinion, the juice isn't worth the squeeze.
If you want to boot to the internal SSD, which you will need to do from time to time to install macOS updates - plug an Apple keyboard and a display into the Mac Pro, turn the power on, and hold down the option key on the keyboard until you see the dialog that allows you to select an OS to boot. Pick Macintosh HD, and you will boot from the internal.
@n3dwilson, thank you. I found @lamw's virtualghetto site a long time ago when I was tasked with trying to get Mac Mini's to be VMHosts (successfully with that resource) and then the Coffee Can Mac Pro's with the Pegasus external storage arrays (again with this resource) so when the opportunity to try with the latest Pro was hinted at, I jumped.
I can get it to boot from the internal USB drive using the hold-the-option-key method but was looking for a way that would not require user intervention....In the mean time, and whilst still "in development" I'll have to be physically present to reboot.
I'm willing to try getting it to use the internal as I'm not on a timeline other than "whenever you can get it to work, we'd really appreciate it". 🙂
I just un-boxed it today so I'm just starting to explore. Found the NIC issue to still be there with 7.0U2a: https://williamlam.com/2021/03/aquantia-marvell-aqtion-atlantic-driver-now-inbox-in-esxi-7-0-update-2.html
I'll keep poking at it, I know @lamw and others are as well.
@simplijm ahh, yeah I see what you are trying to do. It sure would be nice if you could add IPMI functionality to Mac Pros as well!
If your Mac Pro is already booted into macOS, you can ssh into it and try the following:
# get a list of bootable volumes
sudo systemsetup -liststartupdisks
# get current startup disk
sudo systemsetup -getstartupdisk
# set the startup disk
sudo systemsetup -setstartupdisk /path/to/USB/Volume
If you are booted into ESXi, you could maybe try something like:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vcenterhost.doc/GUID-5CF0FA44-E322-461C-9814-CE0A9A2F1BF9.html
Hope this helps.
You can set the boot disk at the boot selection menu (pressing Option during POST) by holding down CTRL before hitting Enter to boot from the selected Disk. The icon under the disk should change from an Arrow to a Circlular Arrow. Subsequent restarts will boot from this disk.
After some trial and error, I was able to get 6.7 U3 Build 15160138 installed and up and running. I added in the Marvell driver to a custom ISO build, Installed and then loaded the community NVMe driver from esxcli.
Interestingly enough, I WAS able to see the SSD and reformat as a VMFS volume (although only 1.8TB instead of the full 2TB).
Now it boots to the internal USB stick where I have ESXi installed. This is what I was after.
Now to get a VM going.....
I’ve asked for an external update which should hanswer questions around 2019 Mac Pro. Will share its published
Long time reader, first time commenter. Thanks for everything you've shared over the years. I've been saved by technical details you've posted more than a few times.
I've been trying to just get our cheese grater to boot an ESXI usb stick or mounted ISO for a couple days with no luck. I've tried the syslinux + copy + edit and every combination of rufus settings for creating a usb stick. I've disabled secure boot protections and even disabled system integrity protection. Still no luck. USB sticks that boot on dell hosts are also not booting on the mac pro 2019. I see the device come up as UEFI Boot (no USB symbol on drive) but I always end up at system recovery to select boot disk and restart.
After years of managing esx clusters on Mac Pros it feels bad to need to ask but, how are you all making your bootable media, and is there something other than disabling the protections that's needed?
Hey Tyler, what kind of ISO are you using? I am able to boot off ISO over KVM to our new Mac Pros with a custom ISO generated via PowerCLI. All it has special is the VIBs for the ATTO adapters in it. ESXi-6.7.0-20200604001-standard seems to work the best for us. Upgrading or installing any newer version has broken ESXi in our experience.
FYI - In case folks have not seen/heard the news, VMware will no longer VMware will no longer pursue hardware certification for the Apple 2019 Mac Pro 7,1 for ESXi.
Reference: https://blogs.vmware.com/vsphere/2021/09/update-apple-2019-mac-pro-71-for-esxi.html
Hey @lamw - man, that makes me so very sad. Thanks for the information, as always. But it sure would have been nice to make use of these machines in a workflow supported by VMware. Do you think that there is a snowball's chance in hell of VMware supporting ESXi on Apple Silicon, or is this likely the end of the road?
Has anyone found a single PCIe3.0 NIC that's both Mac happy and VMware 7.0 compatible?
you throw a super generic Intel X520 into a 2019 Mac pro and it just stops booting.
I've used these, and have never had any issues: https://www.startech.com/en-us/networking-io/pex20000sfpi
Keep in mind - it's not easy finding generic Mac drivers for these cards - I think you'd have to purchase a specific adapter from SmallTree to get the driver support. These will work out of the box with VMware, and are 100% stable. For macOS guests, just use VMware's built in networking, or you can also pass through one of the onboard Atlantic 10 GbE ports.
That's interesting. This is just the same Intel 82599 on a PCIe 2.0 platform as the Intel X520 that doesn't let the Mac boot. Smalltree is an interesting idea though. this card in particular I'll have to look into more:
https://small-tree.net/products/10gbe-switches/p2e10g-2-xr/
thanks for the suggestion n3d.
Quick update for anyone googling around. The MacPro 7,1 has some hidden limit on the number of PCIe 2.0 cards, or lanes it will talk to simultaneously. the Intel X520 actually will boot on this hardware, assuming you don't have PCIe 2.0 cards in most of the other slots. I don't know what the exact limit is, but if you were trying to pack wall to wall PCIe devices into this unit, I would recommend making many, or all, of them PCIe v3.
Good to know about the PCIe card limit!
Sadly, I'm still stuck on getting VMs to boot without the unlocker installed. Looks like this is a bit of a lost cause at this point. 🙁
oh wow, you got MacOS VMs to boot with the unlocker? Good job! It gets so much worse than that. With a MacPro 7,1 and Vmware 7.0.2, 7.0.2d, or 7.0.3, Mac VMs don't boot with OR without the unlocker installed. And of course my Dswitch is running at version 7.0.2, so I can't try 7.0.1. I've gotten a High Sierra VM to boot maybe 1 in 10 times, but Mojave+ all do the Dontstealmacos boot loop and gray screen of death. If you have some magic version of the unlocker tool that works on 7.0.2, please link me. It's not violating anyone's EULA to leverage the $20,000 sunken cost that VMware and Apple have abandoned us with. @n3dwilson
@Jesus - damn. Yeah, without the unlocker, I just get the reboot loop as well. So frustrating.
So I have a MacPro7,1 that has been up for 117 days running 4 VMs. The firmware version is 1715.40.15.0.0, which I believe was distributed with the macOS Big Sur 11.6.1 installer.
On the VMware side, the installed profile is ESXi-7.0U3-18644231-standard.
For the unlocker, I grabbed it from GitHub here:
https://github.com/shanyungyang/esxi-unlocker
Be sure to build it first, instructions are in the README, and then copy (scp) the resulting .tgz file to the ESXi host. I use a Mac as a desktop computer, I've built this several times on said Mac, but have never tried with Windows or Linux. Linux would probably work.
I have only added three configuration options in the .vmx file for the guest. The first is required, and the second/third are only if you plan on mucking around with PCI passthru.
smc.present = "TRUE"
pciPassthru.use64bitMMIO = "TRUE"
pciPassthru.64bitMMIOSizeGB = "64"
Good luck!
your reply is already more than charitable and I appreciate you. If you have a chance, do you think you could check and see if the smc-test command returns SMC = present regardless of whether or not the unlocker tool is installed? I have observed it to list present both ways, which makes it difficult to know if I'm encountering an issue with the installer, or the environment more broadly.
I hate to even ask since I don't want you to uninstall it as a favor to a stranger and accidentally bring down your environment. That Mac firmware version is good info. I'll have to compare that to what I have.... I think on the MacOS side I had upgraded to Catalina or Big sur, not that I'll have any way to adjust it or roll it back to match yours.
I've been using the pre-compiled unlocker.tgz files this entire time, since I can't get it to compile on MacOS.
@Jesus -
The esxi-smctest.sh script should always report "smcPresent = true" if you are running on genuine Apple hardware, regardless of if the unlocker is installed or not. ESXi will load a kernel module called "applesmc" on boot if it detects actual Apple hardware.
To build the unlocker from source is pretty easy - it's just Python.
Here are the steps I took - I tested this just now on a lab machine.
**** On Mac Desktop ****
cd /tmp
git clone https://github.com/shanyungyang/esxi-unlocker.git
cd esxi-unlocker/
./esxi-build.py
scp esxi-unlocker-302.tgz *protected email*:/vmfs/volumes/datastore1/
**** On VM Host ****
cd /vmfs/volumes/datastore1/
tar xzvf esxi-unlocker-302.tgz
./esxi-install.sh
reboot
**** To verify, on VM Host ****
cd /vmfs/volumes/datastore1/
./esxi-smctest.sh
/bin/vmx
smcPresent = true
custom.vgz false 36014512 B
Just verified that I am able to successfully boot a macOS guest running Monterey 12.2.1. Impossible to do without the unlocker though - it does the dreaded reboot loop after it causes a kernel panic at Dont_Steal_MacOS.cpp. It appears to be looking for a key in the NVRAM - "OSK0", that it can't find.
It seems like one way to tell whether you're using the native SMC or the unlocker SMC is whether running ./esxi-smctest.sh shows
smcPresent = true
or the fuller
/bin/vmx
smcPresent = true
custom.vgz false 34482448 B
It might actually be working now! the difference would somehow be using the self-compiled version of 3.0.2 that I made with Big Sur vs. a pre-compiled version of 300 or 303 from the releases area of the github. I'll have to test with more OSX versions and more reboots, and check passthrough as well. First time all year this hasn't looked completely hopeless. Thanks for all the notes @n3dwilson You can't imagine how helpful they were.
For anyone googling around, I've learned that these from Sonnet that utilize the Intel 82599 chipset are very much supported in ESXi both 6.x and 7.x as well as MacOS and Windows. https://www.sonnettech.com/product/presto10gbesfp.html
They make a copper 10GBase-T card just like it for 250 USD for those who don't wanna deal with optics. I have 2 on order and will report back when they are working as promised 😉
Yes, this 2x10G Base-T card from Sonnet and the related fiber/optical versions of the same are great 2x10GB cards - work beautifully with Mac Pro 2019 and ESXi 7.0.3(g) - https://www.sonnettech.com/product/presto10gbaset.html - even supports 2x10GB NIC teaming/Link Agg if your switch supports it of course.
Great to see folks up and running on 7.x with the mac pro.
I was stuck running on 6.7U3 with a 2020 patch. That was working perfectly; however no upgrade after that seems to allow the aquantias to boot. Since it's still 6.x, I am going to try the community drivers next to see if I can get up to the very latest version of 6.7 with working Aquantias.
@n3dwilson couple questions!
What nics are you using for your 7.0 build? You mention the startech card. I looked all over their site and they don't mention esx 7.0. They do mention the vmlinux compatible versions of ESX. So I'm just wondering if you're using that startech card on ESX 7.x.
Also a bit hard to figure out but wondering if that card will support an ethernet (copper) SFP? It keeps talking about fiber but it's kind of weird because the SFP is not installed. The SFP is what does the heavy lifting IMHO but I don't think this will hold an ethernet sftp (like https://www.amazon.com/10GBASE-T-Transceiver-Copper-Compatible-SFP-10G-T-S/dp/B06XQBFHNL?th=1 )
A couple updates on my experiments with mac pro 2019 and aquantia using ESX 6.7 build 19195723 (latest) last night and this morning:
-I noticed there is a vib called "qcnic" bundled in later versions of ESX 6.7 that contains the marvell aquantia driver. It may be identical to the Atlantic driver we have been adding to the installer.
-Since those newer versions of ESX 6.7 are not working correctly, I tried building the latest ESX 6.7 images without the qcnic vib while including the community vmklinux based driver (AQtion-esxi-2.2.2.3.0-esxi1.6) OR the Atlantic (1.0.2.0 build 8169922) driver. I verified in turn that these drivers were exclusively loaded using "esxcli software vib list". Each build exhibited the same bad behavior I described above with the latest version of ESX 6.7.
So no love for the aquantia cards past 6.7 U3 so far.
I have some good news for the brave amongst us. Who don't mind living in unsupported territory.
I am now using vsphere 7.0.3 (70U3d) build 19482537 on my Mac Pro 2019 with all the latest firmware updates. It is running steady with heavy workloads with the aquantia drivers that were released quite some time ago.
I have been able to use the 2 10gbe nic cards on a gigabit switch (actually an ASUS AX11000) as well as a new Netgear XS508M 10gbe switch. All seem to be working perfectly for me. As opposed to my previous experiments with ESX 7.0 on the box, the Nics are able to autonegotiate to the correct speed in both situations.
I am not running the nvme fling; instead I reserve the onboard NVME for running Mac OS or Windows (boot camp) hosts. I am booting off USB which is actually the slot located on the motherboard so all external slots are open. I have a promise J2i caddy up near the 28 core CPU ; I have 2 Micron 9TB drives mounted. ESXi is able to see these both easily and both are allocated right now as my VMFS storage.
I have tested USB gigabit ethernet adaptors from TPLink including the UE300C (USB C style) and UE306 (USB type A style). They all work as well. IN theory just leveraging those gigabit USB adaptors on available external ports, I could easily have 6 Nics on the back and a couple more on the top (desktop model). More than enough for any kind of network topology you have.
Although I'm still searching for a good NVME raid that is reliable for the system, I can currently utilize the promise pegasus R4i RAID.
I am accessing the raid for Time machine backup server and Plex Server utilizing passthrough from a VM running Mac OS Big Sur.
It's a caveat that I do still need to use the unlocker (https://github.com/shanyungyang/esxi-unlocker.git) python script for the Mac OS VM to launch, even though tests indicate the SMC is visible. So that's something I have to manage. I am accessing the Mac VM right now seamlessly by connecting to it using Apple Screen sharing in full screen. I attempted to passthrough the RX6900 AMD card; however I was unsuccessful in getting the VM to boot. The Mac VM has 32GB of RAM and 8 CPU virtual cores allocated so plenty able to run big apps, albeit without the GPU.
It's been a fairly good experience using the hosted Mac VM with plenty of compute and no visible lag. So I'm still using the Mac "as a mac" while hosting a pretty heavy ESX workloads (including a VM with 600Gb of RAM and 32 dedicated virtual cores). With all that, the machine is still at less than half capacity and purring like a kitten with the fans audible but not nearly as high as they are able to go.
======
Caveat emptor as I said for the brave. If this configuration breaks in an upcoming ESXi release, it's not VMware's fault. This is not a configuration they need to regression test! However, I am thinking that as long as the Mac Mini 2018 is supported, the aquantia cards and many other things will continue to work "just fine".
Really really glad this thing is not the paperweight I thought it was going to be when VMware announed they would not pursue HCL.
None of this stuff would be working without Martin's help- we are all indebted.
@Greg - Thanks a ton for the detail - i just received a 28-core rackmount 2019 Mac Pro and will be doing VERY similar to yourself - your post was from May 2022 so my question is have you been able to run any later version of ESXi 7.x than the 7.0.3 (70U3d) that you stated, and still have working standard Apple 10G internal NICs? Any/all other updates would be helpful to know, if I may ask. Thanks again, the above is already a ton of help.
We recently bought another batch of these, configuration is exactly the same with the caveat that they came with Monterey. We can not vmotion to the new hypervisors due to what we suspect is a SMC mismatch. Has anyone come across this yet? Suggestions on how to maybe downgrade the SMC version? I verified "smcPresent = true"
I am thinking of buying another newer one and hope I don't hit this issue. Some questions:
-Are you running ESX 7.0.3 latest?
-Are you only having trouble with vmotion of Mac OS?
-Have you tried to boot the older boxes to Mac OS and then run all the latest updates in Big Sur and/or Monterey? Those do include firmware updates.
I am hoping to use vmotion with the new box, but for me it won't be mac OS I vmotion. Still important.
so i just tried wiping the internal monterey drive and installing big sur in its place and that didn't do the trick. Also tried using the esxi unlocker project and still no luck. Running vim-cmd hostsvc/hosthardware | grep smcPresent returns smcPresent = true, on both the working systems and new busted. My hope was clean install of big sur would flash the SMC back but i'm not sure thats the case. Haven't tried catalina yet. Not sure if there is any other mechanism to flash the smc
I'll be getting the other one soon I will see if it's a problem for me. But you haven't mentioned if you are vmotioning mac OS vms or linux? It matters I think.
Also, just to be clear: You want to run Mac OS on the OLD system so that you can potentially Advance the firmware so that both the new machine and the old machine have the same firmware. There would be no way to get a new mac to go "back" to an older firmware. Macs have never been able to run versions of mac software (firmware or software) that existed prior to the day that macintosh shipped. So you need to flash the older box and cross fingers. But please let us know what you are vmotioning.
Apologies. We are running MacOS vms, turns out vmotion was just the beginning. We cannot run MacOS at all on these, not even the ISO will boot. We can cold migrate but the vms will not boot. The dream is dead.
sorry ISO will NOT boot
to be clear as mentioned above everything we are doing is not supported. but people are making macOS on VMware work.
https://www.macincloud.com/
maybe that system for example is running on Mac mini but I seriously doubt it given the memory ceiling.
The esxi unlocker python script is a must for me even on Apple hardware are you using it? of course it was created to allow running mac vms on a non-Mac server but… Of course it works on this Mac too and others on this forum are using it as well. There seem to be various versions of the script floating around… and 7.0.3d really only works with the latest one. does that break v motion? I don’t know. I just know that without it I can’t boot Mac VM’s either.
so my advice would be to make sure you try that.
I have a more basic problem right from the start on the 2019 Mac Pro - my bootable ISO of 7.0U3g via USB stick just quits out after it begins the install process. I choose the EFI Boot option and it gets thru about 2 minutes of booting, i see the messages scrolling by and then at a certain point it just craps out and nothing, no display, nada. When preparing the USB stick, I didn't edit boot.cfg or anything, not sure if I'm supposed to. I literally just followed this tutorial here to create the bootable ESXi stick and so far no luck. https://virtuallywired.io/2020/08/01/create-a-bootable-esxi-7-usb-installer-on-macos/
I slowed down a video of the last message before it goes dark...it says "Located crypto module: VMWare's ESXBoot Cryptographic Module, v1.0
It goes to "Shutting Down firmware services..." as the final message before darkness. Right after the crypto module message noted above.
See if adding “norts” helps https://williamlam.com/2022/10/vsphere-8-on-apple-mac-hardware.html
I always uses unetbootin, it’s multiple platform and you load ISO and it does rest. Creating bootable USB is super easy using this graphical tool 🙂
Thanks William! I knew there had to be a better way! Unetbootin it is for all future images, i actually think that's what I used years ago to make my xserves into ESXi hosts - incidentally, if I use that method, do you still have to rename the ISOLINUX.CFG on the flash drive and rename it SYSLINUX.CFG file like the tutorial I posted is showing?
Nope. No additional tweaks required 🙂
Total failure thus far - I am using netbootin , and have tried 3 different ISO's, and tried them with and without the norts=1 line in the boot.cfg file inside of the EFI directory. All six tries result in the same result - it gets thru loading a bunch of libraries, including tools.t2 etc and then spits out a "Shutting down firmware services..." and goes black. The 3 ISO's I tried were the 7.0U3(g) ISO , a 6.7U2 ISO, and even a known good ISO that is working for Greg right now that he was kind enough to share with me. Even his known good ISO fails on my 2019 Mac Pro, same model as his. I have a hunch my EFI is somehow different and maybe Apple nerfed the EFI in the latest Ventura firmware update for 13.5.2 patch.
I was going to say this seems like it’s FW related and in the past, I’ve seen customers run into inconsistencies. The “shutting down” is expected when bootloader finishes booting and passes control over to VMkernel … but something (FW issues) would caused it to fail and hence it stops after the last successful operation
Yeah, unfortunately I'm down the rabbit hole now of trying to figure out how to downgrade Apple firmware/EFI....seems like it's not possible. Anybody have any ideas on how I could get the known good firmware that Ned and others posted? Ned's post mentions he was running fine with 1715.40.15.0.0 on his 2019 Mac Pro.
Keeping in touch with Brian offline; I think what people forget is that that shutting down statement is the last thing you see when the kernel is not able to drive the console further. In fact I see that every time I boot because I've disabled it. In fact I think it's possible ESX fully boots up after this point - just not displaying the progress bar.
Looks like Brian may be using the thunderbolt port. Saw an old thread where HDMI may be the only way to get past this point. That may be the trick.
Hard to set up the box initially without getting the console going there. Mine is disabled now and I back up the USB boot module on occasion to make sure I can back out when I do upgrades etc.
Correct you are sir! THANK YOU immensely! It was the silly business of me using the TB3 port on the MPX graphics card which was working for everything else video wise. As soon as I switched to the lone HDMI port on the same MPX W5700XT graphics card, it indeed was getting thru boot just fine with 7.0.3(g) - so after choosing Installation now, I'm getting the red error message saying 'Logs are stored on non-persistent storage. Configure syslog or a scratch partition' - of course that's true, because I'm trying to install ESXi on the very USB that I booted from (simple 128GB USB stick). I have a Micron 9300 PCIE-based NVME drive with 8 TB of space on it in this machine, but i suspect ESXi won't let me install to that, since it never gave me the option. I'm getting closer here for sure! Thank you to Greg for putting up with my drama and teaching me an awful lot along the way. This is gonna be fun.
Ok, phew, all good thus far - I've installed ESXi 7.0U3g on (for now) a 128GB Sandisk USB thumb drive mounted on the inside USB port - I can boot up fine into ESXi now and we are up at 1GB for now, will try 10G later. First task is to create a datastore, because of course it sees none, and when I try to create a VMFS store, it doesn't see the Micron 9300 NVME-based PCIE disk at all. So won't be able to run my 1st VM until I get storage sorted. I don't want to do NFS or iSCSI at present, as I will be moving locations soon, etc. Local disk is what it's got to be. Any suggestions on why it can't see the Micron disk? Ahh wait, that disk was formatted as APFS under native macOS originally, so perhaps if I reformat it as exFAT, then ESXi will see it? It's worth a shot and all I have to do is reboot into the macOS startup volume which is left on the internal Apple SSD(t2 based). Will try that and report back.
No luck on the nvme based PCI detection - what's weird is ESXi sees the 'vmhba1' non-volatile memory controller and has the driver 'nvme_pcie' assigned to it, but when I try and add a datastore and do the whole VMFS formatting, it does not even show the Micron 9300 as an available device. Says "no available storage device" basically. What type of disk would work? I am considering getting standard 2.5 inch SSD's and populating them in this internal cage using the internal SATA ports - https://www.sonnettech.com/product/fusion-flex-j3i/overview.html - any suggestions for getting the Micron disk usable though also?
So up and running with ESXi 7.0.3 on the 2019 Mac Pro and using the Sonnet Fusion Flex drive caddy to house 3x4TB SSD's connected via 2xSATA connectors and 1xUSB (internal) connector...working great.
-PCIE USB card from Sonnet based on the FL1100 chipset also working great. https://www.sonnettech.com/product/allegrousb3pcie4port.html
-WD Black AN1500 PCIE card w/ NVME 4TB storage arrives today (thanks for the suggestions above Ned!) and I expect that to work nicely based on the feedback above also.
-Additional 2x10G NIC installed (copper CAT6A version) - Sonnet Presto PCI card - it's in and working at 10G now for eventual 20G LAG as supported by the card and my upstream switch.
Time to try some pass thru for Mac OS VMs etc.
Hello Brian, could you please send me a link to which NVMe drives you used please? Thanks
Also, did you use this product to hook up the NVMe M.2 ssd?
https://www.amazon.com/gp/product/B01GCXCR7W/ref=as_li_ss_tl?ie=UTF8&psc=1&linkCode=sl1&tag=photoshopcafe&linkId=4ec41fc92ca5cf3cbfc8823e33649849&language=en_US
Brian (and I) are using the AN1500 Western Digital he mentioned. Makes a great cache tier for vsan BTW. I have vsan working flawlessly on 2 of the mac pro's with a mac mini as the 3rd/observer
I am using 8TB Micron drives for the capacity tier. The mini uses the internal nvme for the capacity tier and an OWC thunderbolt external M2 NVME drive for the cache tier.
I have a ubiquity 10gbe switch carrying vmotion, vsan, provisioning, and VTEP traffic.
Everything is working and stable for the most part.
we have used this method to deploy dozens of macpro 2019 7,1 systems but recently bought a batch from a refurbisher who had upgraded them to sonoma and 3 out of 4 of them freeze when trying to boot from esxi installer usb at Initializing ACPI. I have tried with various versions of 6.7 and 7.0 but they all freeze at the same point. I nuked the disk on one of them and did a clean cloud recovery back to big sur but still see the same behavior. It seems the cloud restore didn't rollback the firmware version still at 2022.100.22.0.0 (iBridge: 21.16.4222.0.0,0). I tried passing pciExperimentalFlags=1 both from SHIFT+o boot environment and from altering the boot.cfg file to include it but no luck. Has anyone run into this issue? Any suggestions on more steps i can try?
Have you tried using these tips https://williamlam.com/2023/10/troubleshooting-esxi-shutting-down-firmware-services-and-uefi-runtime-services-rts-error-message.html in case newer ESXi bootloader has some fixes
Thanks for this! Unfortunately Option A had no effect, Option B led me down a rabbit hole of trying every esxi 8 build available for download and all freeze at the same point. I was able to get a recent arch linux distro (that i just happened to have on a usb) to boot and capture dmidecode and dmesg. https://pastebin.com/a6cYD2xD https://pastebin.com/Nh8048ZU
William is pointing out that the last status message can be confusing. Question: Did you disable the console in anticipation of gpu passthrough by setting:
"esxcli system settings kernel set -s vga -v FALSE"
?
This will cause everything to stop displaying after "shutting down firmware services" or potentially a different message.
I have been updating macOS bare metal (isn't that a weird thing to say!?) on occasion and so far I haven't had any firmware updates disable the box.
How can i set esxcli system settings kernel set -s vga -v FALSE from the preboot installer environment? Once it freezes i can't get to a TTY. I have an older system that works and tried pulling that installed drive and putting it in the non-working systems and it fails to boot the installed esxi and freezes at the same point "initializing acpi"
Hi William Lam,
I'm not an expert but trying to install ESXi 8.0 on Mac Pro 7.1 (2019) and installer struck at "VMKAcpi_Late Init". Hoping to get help completing the any version of ESXi installation.
I tried the ESXi 6.7 U3 as well but that get struck at initializing acpi.
Thank you
Point where it got halt : VMK init (127/190): VMKAcpi_LateInit
VMKAcpi_LateInit with 2022.100.22.0.0 is a pretty serious issue.
I accidentally took this firmware update disguised in the software update bundle. One of my two machines is now stuck here. This is definitely a firmware incompatibility.
I called apple support but downgrading the firmware was not something they could handle on the support line. And the only way I've seen this done successfully was on youtube, and it involved desoldering the eprom for the mac, then resoldering it after the downgrade, which included obtaining and modifying the rom file.
Was anyone able to pursue Apple or VMware support successfully regarding this? I'll resort to replacing the motherboard if necessary.
is there a new option with this VMKAcpi_LateInit error?
Thanks
Stephan
I’m stuck at the same VMKACPI issue tried every version of esxi definitely a firmware update issue.
Apple Configurator Firmware revive: failed
Brand new motherboard install: failed
ROM save from good and restore to bad (desoldered): failed
Motherboard from parted working Mac pro into mine with my components: failed
Buy new, unbooted mac pro 2019: succeeded
Thousands later, I really don't see any option but replacement.