Several folks in the community had reported issues that after upgrading from ESXi 6.7 to 7.0, their Samsung NVMe PCIe SSDs were no longer showing up. The first report of this was from Ivo Beerens and eventually found that reinstalling ESXi from scratch works but certainly that was not ideal. Just yesterday, I saw that Jeffrey Kusters also shared a similiar issue and used a different workaround which allowed him to upgrade. I also reached out to VMware Engineering as they thought this was a strange behavior but needed to see the logs to understand what was actually going on. Since Jeffrey's setup was an upgrade, I was able to get a copy of his vm-support bundle to provide to Engineering.
Within minutes of looking at the support bundle, they quickly identified the issue and this was caused by using the incorrect ESXCLI command to upgrade a standalone ESXi host from 6.7 to 7.0. Instead of using "esxcli software vib update" command, folks should be using "esxcli software profile update" which has always been the correct command to use when upgrading an ESXi image. In fact, this has been in the vSphere documentation for quite some time and here is the ESXi 7.0 version of that documentation. More importantly, the incorrect command only upgrades the ESXi 6.7 VIBs that exists and does not install any of the ESXi 7.0 VIBs, which means after the upgrade, you are not only missing the nvme-pcie VIB but many other ESXi 7.0 VIBs!
tl;dr - If you are going to use ESXCLI to upgrade your standalone ESXi host, please make sure to use the correct command or you will have issues. Below are the two commands you will need to determine which ESXi Image Profiles are available given an offline bundle and then updating to a specific image profile.
List Image Profiles from ESXi 7.0 Offline Bundle:
[[email protected]:~] esxcli software sources profile list -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -------------------
ESXi-7.0.0-15843807-standard VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
Upgrade to a specific Image Profile from ESXi 7.0 Offline Bundle:
esxcli software profile update -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard
Hello William,
Great blog. However many folks have been using the "incorrect" command for years and it worked fine.
You can also look at the KB below and the "incorrect" command is used in examples there as well
https://kb.vmware.com/s/article/2008939
That KB talks about "patchig" ... not upgrading
You are correct. It gets very confusing as some people used that command to upgrade as well. It worked until now. Vmware should have one command and to confuse with upgrading vs updating. The updating command section is under upgrading esxcli documentation on the vmware docs website.
Hello,
Your update command is wrong, it should be "esxcli software profile update"
Thanks for the catch. Its been fixed
Hello William,
I read your article about some NVMe not being recognized, and since I have 2 of them, I decided to try it out and installed ESXi 7.0 on my SYS-E300-9D-8C8NTP system using bootable USB instead of upgrade my existing 6.7.
Onboard NVMe (Samsung) was recognized without problem. However, my ADATA XPG SX8200 Pro NVMe installed on 90-degree angled bracket inserted into PCIe slot - did not 🙁
I got it installed few months ago on 6.7 and it was recognized immediately. but since this fresh install on USB, it didn't recognize it. I guess I won't upgrade my system for now. Any tips how to identify if there is a problem with this driver on 7.0?
Same thing here. Have you fix this or till now facing this issue with 8200?
Exact same problem over here too with same NVMe board.
Just adding my two cents here as well. 6.7 works great for NVMe. New install of 7.0 doesn't who my 1 TB NVMe drive. Hopefully able to find some work arounds as I was still have to do the nvme.v00 driver copy to make 6.7 work.
Actually - the same work around worked for me just now. However - the file names are different...
mv nvme_pci.v00 nvme_pci.v00.old
mv nvme.v00 nvme_pci.v00
from the VMW_bootbank_nvme_1.2.1.34-1vmw.650.2.50.8294253 patch
Can you explains more of this workaround?
This worked great for me! It's a life saver! You might want to make note of the file rename above.
I should say the author of the article should mention the file rename up above.
Thanks for sharing the updated instructions Dave and the confirmation from Evan. However, this blog pos is really about incorrectly upgrading to ESXi 7.0. The details have been updated in i https://www.williamlam.com/2019/05/quick-tip-crucial-nvme-ssd-not-recognized-by-esxi-6-7.html which is the blog post that details about NVMe devices that used to work and copying 6.5 NVMe VIB
Would these Cisco instructions be incorrect as well? They seem to go back and forth with update and upgrade between 5.5 to 6.x
https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/213689-upgrade-vmware-esxi-from-version-5-5-to.html#anc6
The same seems to happen with ESXi-7.0U2c-18426014-standard (Build 18426014). I lost access to my Samsung 970 after upgrade. Offline bundle for U2c doesnt seem to be out. So for now I am stuck without my samsung...
I was able to rollback with Shift-R during boot and the samsung 970 showed up again.