WilliamLam.com

Important - NVMe SSD not found after upgrading to ESXi 7.0

by // 17 Comments

Several folks in the community had reported issues that after upgrading from ESXi 6.7 to 7.0, their Samsung NVMe PCIe SSDs were no longer showing up. The first report of this was from Ivo Beerens and eventually found that reinstalling ESXi from scratch works but certainly that was not ideal. Just yesterday, I saw that Jeffrey Kusters also shared a similiar issue and used a different workaround which allowed him to upgrade. I also reached out to VMware Engineering as they thought this was a strange behavior but needed to see the logs to understand what was actually going on. Since Jeffrey's setup was an upgrade, I was able to get a copy of his vm-support bundle to provide to Engineering.

Within minutes of looking at the support bundle, they quickly identified the issue and this was caused by using the incorrect ESXCLI command to upgrade a standalone ESXi host from 6.7 to 7.0. Instead of using "esxcli software vib update" command, folks should be using "esxcli software profile update" which has always been the correct command to use when upgrading an ESXi image. In fact, this has been in the vSphere documentation for quite some time and here is the ESXi 7.0 version of that documentation. More importantly, the incorrect command only upgrades the ESXi 6.7 VIBs that exists and does not install any of the ESXi 7.0 VIBs, which means after the upgrade, you are not only missing the nvme-pcie VIB but many other ESXi 7.0 VIBs!

tl;dr - If you are going to use ESXCLI to upgrade your standalone ESXi host, please make sure to use the correct command or you will have issues. Below are the two commands you will need to determine which ESXi Image Profiles are available given an offline bundle and then updating to a specific image profile.

List Image Profiles from ESXi 7.0 Offline Bundle:

[root@e200-8d:~] esxcli software sources profile list -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -------------------
ESXi-7.0.0-15843807-standard VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54

Upgrade to a specific Image Profile from ESXi 7.0 Offline Bundle:

esxcli software profile update -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard

Quick Tip - Allow unsupported CPUs when upgrading to ESXi 7.0

by // 69 Comments

As outlined in the vSphere 7.0 release notes (which everyone should carefully read through before upgrading), the following CPU processors are no longer supported:

  • Intel Family 6, Model = 2C (Westmere-EP)
  • Intel Family 6, Model = 2F (Westmere-EX)

To help put things into perspective, these processors were released about 10 years ago! So this should not come as a surprise that VMware has decided remove support for these processors which probably also implies the underlying hardware platforms are probably quite dated as well. In any case, this certainly has affected some folks and from what I have seen, it has mostly been personal homelab or smaller vSphere environments.

One of my readers had reached out to me the other day to share an interesting tidbit which might help some folks prolong their aging hardware for another vSphere release. I have not personally tested this trick and I do not recommend it as you can have other issues longer term or hit a similiar or worse situation upon the next patch or upgrade.

Disclaimer: This is not officially supported by VMware and you run the risk of having more issues in the future.

Per the reader, it looks like you can append the following ESXi boot option which will allow you to bypass the unsupported CPU during the installation/upgrade. To do so, just use SHIFT+O (see VMware documentation for more details) and append the following:

allowLegacyCPU=true

There have also been other interesting and crazy workarounds that attempt to workaround this problem. Although some of these tricks may work, folks should really think long term on what other issues can face by deferring hardware upgrade. I have always looked at homelab as not only a way to learn but to grow yourself as an individual.

Note: The boot option above is only temporarily and you will need to pass in this option upon each restart. It looks like this setting is also not configurable via ESXCLI which I initially had thought, so if you are installing this on a USB device, the best option is to edit the boot.cfg and simply append the parameter to kernelopt line so it'll automatically be included for you without having to manually typing this. If this is install on disk, then you will need to edit both /bootbank/boot.cfg and /altbootbank/boot.cfg for the settings to passed in automatically.

This is ultimately an investment you are making into yourself, so do not cut yourself short and consider looking at a newer platform, especially something like an Intel NUC which is fairly affordable both in cost as well as power, cooling and form factor.

Really cool updates with OVFTool 4.4 and support for vSphere 7

by // 5 Comments

vSphere 7 has officially GA'ed this morning and with folks starting to download ESXi and the vCenter Server Appliance, do not forget about all the supporting tools such as the latest PowerCLI 12.0 release which includes a number of enhancement as well as the various vSphere Management and Automation SDKs.

One of my most frequently used tools on a daily basis, some times even more than PowerCLI is OVFTool which is now at version 4.4 which officially supports vSphere 7 but it also includes a number of really awesome enhancements and bug fixes. 

While looking over the OVFTool release notes, I noticed a few interesting tidbits that I thought was worth calling out:

OVF Tool now can upload disk files to the host in parallel, and download disk files from the host in parallel. OVA is unsupported. Parallelism is limited by the number of CPUs. See the --parallelThreads=N option in the OVF Tool User's Guide for details.

This is a most welcome feature for customers with extremely large VMs where upload and/or downloads of OVAs can take a considerable amount of time as only a single CPU thread is used. With this feature, you can now enable multiple CPU threads with the --parallelThreads parameter which should really with performance! Even for smaller size VMs, you can still benefit if you have additional CPU resources to allocate and something I will be using going forward!

For multi-disk virtual machines, OVF Tool now includes the --multiDatastore flag to specify datastore per disk. See the OVF Tool User's Guide for details.

This is another welcome feature for customers where you might have an OVA that contains multiple VMDKs and want to explicitly place them on specific datastore.

The ARM64 architecture on Linux is now supported.

Finally, I thought this was very interesting to see that OVFTool has been ported over to ARM64 for Linux which means we can run now run OVFTool on a Raspberry Pi or even an Amazon A1 EC2 Instance! This might come handy in the future and I wonder if OVFTool for ESXi would be the next logical step? 🙂

I highly recommend you check out the rest of the release notes as it contains many more enhancements and fixes, many of which I have reported from the community and/or by our customers. I think this is certainly one of the tools you can upgrade immediately as it has great backwards compatibility with older vSphere releases but you can also take advantage of all the new features mentioned above immediately. If there are other OVFTool improvements or enhancements you really would like to see, feel free to leave a comment along with the use case and I will past that on to Engineering.