While official support for running ESXi on Apple Hardware (Mac Pro and Mac Mini) has concluded with ESXi 7.x, it certainly has not discourage users from experimenting with newer ESXi releases. In fact, ESXi 8.x still runs like a champ on most x86 Apple Hardware platforms including the vintage Apple Mac Mini 7,1 (2014) and to the most recent Apple Mac Mini 8,1 (2018)!
For those with an Apple Mac Mini 8,1 (2018), the onboard Apple NVMe storage device can not be used out of the box with ESXi, without the use of the Community NVMe Driver for ESXi Fling. Surprisingly, the Community NVMe Driver for ESXI Fling is still functional even with the latest ESXi 8.x release, even though it was only created and supported during ESXi 7.x timeframe.
From the recent reports from the VMware Community, the Apple NVMe device can be detected up until ESXi 8.0 Update 1c (Build 22088125) but after upgrading to ESXi 8.0 Update 2 (22380479), it was no longer showing up.
Initially, I had simply assumed that this was because the Community NVMe Driver for ESXi needed to be updated to properly support ESXi 8.x, but given the fact that it had worked up to ESXi 8.0 Update 1c, lead me to believe this may not have been the case.
After returning from a much needed break from the US Thanksgiving holiday, I decided to dust off my Mac Mini 2018 and see if anything had stood out. I was able to reproduce the observations folks were making that after upgrading to ESXi 8.0 Update 2, the NVMe storage device was no longer being picked up.
However, the issue was NOT actually with the Community NVMe Driver for ESXi, as the Apple NVMe was being detected as you can see from the vmkernel log snippet below, but something had caused the device registration to fail:
vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu0:524564)HPP: HppCreateDevice:2766: Created logical device 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001'.vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu0:524564)HPP: HppClaimPath:3604: Added path 'vmhba0:C0:T0:L0' to logical device 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001'. Total Path Count 1vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu9:524902)HPP: HppRegisterDevice:3068: Registering logical device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001'.vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu0:524902)ScsiDevice: 4473: Computing hash based VML id on the string 'NVMe APPLE SSD AP0512M C079037005QJRRYAP 00000001' for the device t10vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu0:524902)NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001 (len=76)vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu2:524902)ScsiDeviceIO: 10485: QErr is correctly set to 0x0 for device t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001.vmkernel.log:2023-11-25T19:26:08.117Z Wa(180) vmkwarning: cpu2:524324)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu8:524902)ScsiDeviceIO: 10992: Could not detect setting of sitpua for device t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001. Error Not supported.vmkernel.log:2023-11-25T19:26:08.117Z In(182) vmkernel: cpu6:524902)ScsiDevice: 2880: Failing registration of device 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Unsupported device attributes isLocal:1, isSSD:0, isNVME:1 is512SWE:1
After a bit of digging around internally, I came across an issue that had been reported when an NVMe device was using 4Kn SWE sector format that could cause a device to fail to mount.
After speaking with one of the Engineers, the Apple NVMe issue was not related to the reported issue that I had found but she was able to help identify the problem that I was seeing. Basically, there were some improved checks put in place as of ESXi 8.0 Update 2 to better detect NVMe devices. The Apple NVMe device uses 4Kn format, which is currently not supported and to work around this issue, the developers for the Community NVMe Driver for ESXi did not mark the device as an SSD, since 4Kn HDD (spinning rust) is supported. The Engineer was kind enough to relax the check for this specific scenario and the fix will be available in a future ESXi 8.x patch.
Interestingly, while waiting for the initial response, I had decided to go through the upgrade workflow again but instead of upgrading to ESXi 8.0 Update 2 using the ESXCLI command-line method (which I suspect most folks use for their single host environment), I ended up upgrading from another USB device that contained the ESXi 8.0 Update 2 ISO image. To my surprise, the Apple NVMe was properly detected after the upgrade!? 😮
After testing a few more scenarios, I found that if you have ESXi 8.0 Update 1c with the Community NVMe Driver for ESXi installed on a USB device and you then upgrade from another USB device that contains ESXi 8.0 Update 2, then you will end up with the Apple NVMe device fully accessible after the upgrade!
The following scenarios below will cause the Apple NVMe to be NOT detected:
- If you have ESXi 8.0 Update 1c with the Community NVMe Driver for ESXi installed on a USB device and you upgrade using ESXCLI, the Apple NVMe will NOT be detected after the upgrade
- If you have ESXi 8.0 Update 1c with the Community NVMe Driver for ESXi installed on the Apple NVMe, no matter how you upgrade, the Apple NVMe will NOT be detected during the upgrade and would prevent you from upgrading as you have no viable disk selection
- If you perform a fresh installation of ESXi 8.0 Update 2, the Apple NVMe will also NOT be detected
Additionally for scenario 2 and 3, I noticed the errors were slightly different and only showed up in the vmkwarning.log versus the vmkernel.log file:
2023-11-26T22:35:01.388Z Wa(180) vmkwarning: cpu4:524654)WARNING: VMK_PCI: 593: 0000:02:00.0: failed to allocate 4 MSIX interrupts
2023-11-26T22:35:06.116Z Wa(180) vmkwarning: cpu1:524661)WARNING: ScsiPath: 8584: Adapter Invalid does not exist
2023-11-26T22:35:06.790Z Wa(180) vmkwarning: cpu2:524661)WARNING: ScsiPath: 8584: Adapter Invalid does not exist
2023-11-26T22:35:07.682Z Wa(180) vmkwarning: cpu11:524661)WARNING: ScsiPath: 8584: Adapter Invalid does not exist
2023-11-26T22:35:10.726Z Wa(180) vmkwarning: cpu0:524326)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001
2023-11-26T22:35:10.726Z Wa(180) vmkwarning: cpu6:524815)WARNING: HPP: HppRegisterDevice:3116: Failed to register device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Not supported
2023-11-26T22:35:10.786Z Wa(180) vmkwarning: cpu6:524639)WARNING: VMW_SATP_LOCAL: satp_local_updatePath:830: Marking path vmhba34:C0:T0:L0 as STANDBY
2023-11-26T22:35:10.786Z Wa(180) vmkwarning: cpu6:524639)WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:461: Selected current STANDBY path vmhba34:C0:T0:L0 for device Unregistered Device to activate. This may lead to path thrashing.
2023-11-26T22:35:10.786Z Wa(180) vmkwarning: cpu6:524639)WARNING: VMW_SATP_LOCAL: satp_local_tryToActivatePath:1450: Checking TUR on STANDBY path vmhba34:C0:T0:L0
2023-11-26T22:39:47.636Z Wa(180) vmkwarning: cpu7:524321)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001
2023-11-26T22:39:47.636Z Wa(180) vmkwarning: cpu8:524815)WARNING: HPP: HppRegisterDevice:3116: Failed to register device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Not supported
2023-11-26T22:39:47.637Z Wa(180) vmkwarning: cpu2:524318)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001
2023-11-26T22:39:47.637Z Wa(180) vmkwarning: cpu8:524815)WARNING: HPP: HppRegisterDevice:3116: Failed to register device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Not supported
2023-11-26T22:39:47.637Z Wa(180) vmkwarning: cpu2:524320)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001
2023-11-26T22:39:47.637Z Wa(180) vmkwarning: cpu8:524815)WARNING: HPP: HppRegisterDevice:3116: Failed to register device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Not supported
2023-11-26T22:39:47.638Z Wa(180) vmkwarning: cpu1:524320)WARNING: ScsiDeviceIO: 4523: ModeSense 0x1a request failed - blocking page:0x1c subpage:0x2 t10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001
2023-11-26T22:39:47.638Z Wa(180) vmkwarning: cpu8:524815)WARNING: HPP: HppRegisterDevice:3116: Failed to register device with uid 't10.NVMe____APPLE_SSD_AP0512M_______________________C079037005QJRRYAP___00000001': Not supported
For now, if you have ESXi installed on a USB device and you wish to upgrade to latest ESXi 8.0 Update 2 and still be able to use the Apple NVMe for VMFS datastore, you can upgrade by putting the ESXi 8.0 Update 2 installer on a separate USB device and simply boot from that device to perform the upgrade. If you have ESXi installed on the Apple NVMe or prefer a cleaner upgrade path, then I would suggest waiting for a future ESXi 8.x patch that includes the fix mentioned earlier.
Thank you Will for taking the time to look into this and advise us.
As you mentioned and discovered, the Apple NVMe using the community driver was setup as HDD in 8.01c but even if it is re-configured as SSD via vCenter, the update from 8.01c to 8.0 U2 via ESXCLI fails to load it since it still uses the 4Kn format. The community NVMe driver does load just fine tho as TB3 NVMe drives are using it!
If we have ESXi 8.01 installed on the Apple NVMe currently, should we try creating a custom USB installer using vCenter for 8.0U2 (which includes the community NVMe driver) and try to update to 8.0U2 by booting the USB installer and performing an upgrade? I think you said that worked for you....
Or do you recommend waiting for the next 8.0U3 release and update via ESXCLI?
Thanks.
I've already clearly articulated whats possible in the blog post, please re-read if its not clear. If that scenario doesn't apply to you or you prefer a smoother path, then wait for the future ESXi patch
Sorry, I was just looking for confirmation that both USB to USB and USB to Internal NVMe upgrade did work for you, while ESXCLI method failed.
Again, please re-read post. I believe I’ve already clearly articulated what works and what doesn’t