The expandability of the Apple 2019 Mac Pro (7,1) has been the primary reason VMware customers have been so excited for this new platform for virtualizing macOS on ESXi. The most common request that I hear from customers is for GPU passthrough.
Although VMware does not officially support GPU passthrough, even for the existing Apple hardware systems on the VMware HCL, this has been a topic I have been keeping an eye out on, especially from what the VMware community is doing in this space.
My intention for this blog post is to provide a resource for the community on capturing the success and failures when attempting GPU passthrough on a 2019 Mac Pro. For those interested and have capable hardware, you may want to start with the VMware HCL for GPU passthrough devices listed under Virtual Dedicated Graphics Acceleration (vDGA). This may be your best chance to successfully passthrough a GPU that will be recognized by either a macOS or Linux/Windows guest operating system.
If you would like to share your experiences, feel free to leave a comment or reach out by filling the contact form.
Disclaimer: Although ESXi installs and runs on the Apple 2019 Mac Pro 7,1 it is currently not certified on the VMware HCL. There are no timelines on the certification due to challenges with COVID-19.
Radeon Pro W5700X MPX Module
Thanks to fellow reader Ned who recently reached out and shared his experience with the Radeon Pro W5700x MPX Module from Apple. Ned's initial attempt to passthrough the GPU to a macOS guest caused it to kernel panice when loading the Radeon Framebuffer Driver (RadeonX6000Framebuffer). I had suggested using a GPU that was on VMware's HCL to see if that made a difference or experiment by using a Windows guest to see if the behavior was the same.
It turns out that by using a Windows 10 (64-Bit) guest, Ned was successful in passing through the Radeon GPU to the VM and this looks to be an issue with the macOS driver, at least running in a VM. This testing was done using latest ESXi 7.0 Update 1b release.
Great to read that, but passthrough for all platform can be much better than a mac. Like using an Asrock.. to pass the main pci you have to turn off the onboard one. Perhaps a 'sink' function that can patch the esxi console could be a way somehow ?
This post is for folks who have a need to virtualize macOS guests and need GPU passthrough capabilities. Per Apple and VMware's EULA, to legally virtualize macOS, you must use Apple Hardware. This post is NOT geared towards folks who do not have these requirements and can run generic x86 platforms where GPU passthrough is well understood 🙂
Thanks, William! What's the forecast for ESXi on Apple Silicon? It looks like the new tech is going to give everyone a run for their money in the performance for the buck department. Thanks!
No insights for ESXi on Apple Silicon but I know the Fusion folks are working on their initial enablement https://twitter.com/VMwareFusion/status/1326229094648832000
That'll be a good place to start. Thanks!
About HCL: I am having quite some discussions with support about HCL certification of Mac Pro. They tell me that Apple has to do the certification. Is that true? I find it kind of strange that Apple has interest in VMware HCL certification at all!
VMware Support is correct, hardware certification of ANY kind is usually performed by the hardware vendor themselves. They determine which platform and subsequent components they wish to certify and that's how the HCL is updated. Apple is an outlier as they have not join VMware's partner program nor shown any interests in applying for hardware certification which would also include GPUs. That is why these efforts have mostly been community driven and VMware doing its best to support our customers. If you have any feedback/concerns in these areas and you have an Enterprise agreement with Apple, I highly recommend you share it with the account team.
Hmm if that is so who certified the old Mac Pro / new Mac Mini? Support is not helping (not even try to help) since they say that Apple needs to put the system on the VMware HCL - which they will not do i guess.
Markus, as I've stated in my previous reply, in case it wasn't clear. VMware has taken on the responsibility of certifying these platforms to better support our customers. This is not ideal for number of reasons.
The 2019 Mac Pro is currently NOT certified per my note in the blog post. So they can NOT help and even if it was certified, there are no plans for GPU passthrough certification
Hi William! Since you mentioned GPU passthrough, any hopes of VMware realising a Horizon agent for Macs one day? It would be cool to use some Mac-VDI in a few use cases.
I'm not aware of any plans to do so from our End User Computing Business Units
Hey all, sorry I've been dark for a bit - I had to put these machines into production as bare metal and abandon my ESXi testing for a bit.
Currently, I've been trying for the last day or two using vSphere 7.0.2. If I create a new VM from scratch and install macOS on it, or if I export a functioning VM out of VMware Fusion to my ESXi host, I get the dreaded reboot loop, which kernel panics and produces the following error message:
DarwinPanic: panic(cpu 7 caller 0xffffff7faa9a4b63): "DSMOS: SMC read error K0: 133"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/DontStealMacOS/DontStealMacOS-31.40.1/Dont_Steal_MacOS.cpp:192
The only way to get past this, besides possibly starting a fresh VM with Mountain Lion and upgrading through the versions, is to install the ESXi unlocker. This should not be necessary, since I am attempting to virtualize macOS on Apple-branded hardware (MacPro7,1), but it was the only way to move forward.
The next step from here was to try to get the GPU passthru to work. If you follow William's excellent guide on the 2018 Mac mini, it pretty much works as advertised. I did add a specific hardware label to both the GPU and the associated HDMI Audio device. This is a significant improvement from ESXi 7.0 and 7.0u1, which would kernel panic on boot.
When I boot a VM, I am able to see the GPU in the system profiler. Unfortunately, I can't seem to get macOS to actually use the GPU for either acceleration or to drive a display. I did try disabling the kernel VGA driver, which didn't seem to help much. I also tried setting svga.present = "FALSE" in the vmx file, but this was a bad idea - the machine couldn't find an OS after doing so! The log message I got looked like this:
[msg.Backdoor.OsNotFound] No operating system was found. If you have an operating system installation disc, you can insert the disc into the system's CD-ROM drive and restart the virtual machine.
Setting svga.present back to "TRUE" again will allow the VM to boot, but the GPU is not used:
Graphics/Displays:
Display:
Type: GPU
VRAM (Total): 128 MB
Device ID: 0x0405
Revision ID: 0x0000
Displays:
Display:
Resolution: 1024 x 768 (XGA - eXtended Graphics Array)
UI Looks like: 1024 x 768
Framebuffer Depth: 24-Bit Color (ARGB8888)
Main Display: Yes
Mirror: Off
Online: Yes
Automatically Adjust Brightness: Yes
Vendor ID: 0x15ad
Display:
Type: GPU
Bus: PCIe
PCIe Lane Width: x32
Vendor: AMD (0x1002)
Device ID: 0x7310
Revision ID: 0x0000
Sadly, I'm going to put this machine back into production as bare metal yet again. 🙁
@n3dWilson, I cannot get a VM to boot at all. I've tried creating an ISO to boot from, using recovery DMGs, even installed Fusion in a macbook and copied a VM to the MACPRO and it just loops. Verified "smc.present=TRUE" in all vmx files. Can't get into recovery mode in any of my tests..... any pointers???
@n3dwilson JFYI,
GPU passthrough on the bignavi 21 RX6900 is working GREAT on my Macpro 7,1, but it only works properly for windows. With Windows I am able to load AMD Radeon drivers. I am able to utilize it - even for games- on ESXi. With an additional USB card installed, I am able to pass that through to the Windows VM to drive my razer keyboard and mouse. Pretty sick "using ESXi 7 as your desktop".
But the issues you mention with mac OS are identical for me. Somehow, the system profiler sees the card and reports it. But no direct output to the display. Photoshop reports unknown GPU "with zero memory". So no graphics accel as you say. I did also set pciPassthru.64bitMMIOSizeGB (to 32 in my case, which works for windows) and pciPassthru.use64bitMMIO to TRUE. Those were likely necessary for the windows VM to work.
I am passing through the Promise RAID to this mac OS VM and is working brilliantly- both as a Plex server and a "very large" time machine backup. I'm finding myself using Adobe more and I'd really like to use the graphics card for both its ability to drive the monitor well and also the GPU accel. That being said, with this monster I'm still able to use photoshop and illustrator with many gigabytes of RAM through apple screen share or just the Fusion console on my laptop. And that is not a horrible experience.
I realize my config is about as unsupported as they get; this is more or less a note that "I think I'm almost there" on this topic, but not quite.
Mac OS GPU passthrough is now working on Mac pro 2019. And it was really hacky.
In a nutshell, if you use the partially documented oprom vmx setting (see https://kb.vmware.com/s/article/2142307 for an example) in conjunction with downloading the correct oprom for your AMD card- vmware UEFI will load the oprom to contact the card and boot with the Apple Logo. It seems this need for bios being present for booting may be specific to Mac OS.
This is likely why the availability/support of the new Apple Big Navi video cards came with associated "firmware updates". I remember I could not boot with my RX 6900 active on my mac, until after a firmware update supported it. But the truth is I don't know "why" it works.
I found the solution here-
https://twitter.com/qwertyoruiopz/status/1406408923570130951
I got my oprom for my RX6900 XT at techpowerup. I see many AMD oproms for Apple as well, so if you have a mac card you may be in luck, but you'll probably have to disable it for ESX; I didn't as this is an additional card in my machine.
Apple AMD oproms:
https://www.techpowerup.com/vgabios/?architecture=AMD&manufacturer=Apple&model=&interface=&memType=&memSize=&since=
Mine:
https://www.techpowerup.com/vgabios/?architecture=AMD&manufacturer=Sapphire&model=RX+6900+XT&interface=&memType=&memSize=&since=
The notes on twitter leave the oprom file somewhere outside the VM folder, but I placed the rom file directly into the VM's folder in VMFS, and specified the oprom file name with "no path".
I was shocked to see the apple logo appear on bootup. I am having spotty luck with Mac OS booting in general due to MacOS protections even though I'm running on a mac. I still am using unlocker, and it seems to need semi-regular updates.
All ".vmx" changes:
Of course you will need to add the passthrough for your card:
pciPassthru1.id = "00000:016:00.0"
pciPassthru1.deviceId = "0x73bf"
pciPassthru1.vendorId = "0x1002"
pciPassthru1.systemId = "626bab0c-a3c8-d260-c547-cc2db7072d8e"
pciPassthru1.present = "TRUE"
Advanced settings added through VM options tab->Advanced->Edit Configuration button
svga.present = "FALSE"
pciPassthru.use64bitMMIO = "TRUE"
pciPassthru.64bitMMIOSize = "32" (from https://earlruby.org/2022/02/calculating-the-value-for-64bitmmiosizegb/)
pciPassthru1.filename = "Sapphire.RX6900XT.16384.201210.rom" (this will be specific to card obviously)
pciPassthru1.opromEnabled = "TRUE"
I am using vcenter so I always make these changes through the vcenter UI.
This overall setup of ESXi on the mac pro2019 with 7.0.3d is working great. I have heavy workloads that run for weeks at a time (months if I didn't tinker) while I get full utilization of my mac as a desktop - both for windows and mac. I added an FL1100 based USB card that works on both mac and windows- I do need to pass that through to the VM as well; Host USB mappings don't seem to work.
I am running 10 VMs consuming over 700Gb of RAM and half the 28 core CPU capacity. But I can switch to the Mac or Windows VM and turn my monitor to the display port input and it's like I'm not running ESX at all. I can use photoshop or run games fully accelerated with almost no lag.
Basically ESXi is my desktop OS at this point.
This is extremely helpful Greg, nice work indeed sir. I just acquired a 2019 MacPro and will be placing ESXi 7.x on it - I plan to follow your excellent guidance, and will document the results extensively for posting. Would you be open to contacting directly to discuss if I get stuck?
Hi Brian, did you see my previous response? Let me know. There are other tricks. Happy to do DM if need be.
I'd love to chat Greg - how do we DM on this thing?
Go ahead and google my name + vmware + adobe and send me a linked in request; let's go from there.