WilliamLam.com

Important - NVMe SSD not found after upgrading to ESXi 7.0

by // 17 Comments

Several folks in the community had reported issues that after upgrading from ESXi 6.7 to 7.0, their Samsung NVMe PCIe SSDs were no longer showing up. The first report of this was from Ivo Beerens and eventually found that reinstalling ESXi from scratch works but certainly that was not ideal. Just yesterday, I saw that Jeffrey Kusters also shared a similiar issue and used a different workaround which allowed him to upgrade. I also reached out to VMware Engineering as they thought this was a strange behavior but needed to see the logs to understand what was actually going on. Since Jeffrey's setup was an upgrade, I was able to get a copy of his vm-support bundle to provide to Engineering.

Within minutes of looking at the support bundle, they quickly identified the issue and this was caused by using the incorrect ESXCLI command to upgrade a standalone ESXi host from 6.7 to 7.0. Instead of using "esxcli software vib update" command, folks should be using "esxcli software profile update" which has always been the correct command to use when upgrading an ESXi image. In fact, this has been in the vSphere documentation for quite some time and here is the ESXi 7.0 version of that documentation. More importantly, the incorrect command only upgrades the ESXi 6.7 VIBs that exists and does not install any of the ESXi 7.0 VIBs, which means after the upgrade, you are not only missing the nvme-pcie VIB but many other ESXi 7.0 VIBs!

tl;dr - If you are going to use ESXCLI to upgrade your standalone ESXi host, please make sure to use the correct command or you will have issues. Below are the two commands you will need to determine which ESXi Image Profiles are available given an offline bundle and then updating to a specific image profile.

List Image Profiles from ESXi 7.0 Offline Bundle:

[[email protected]:~] esxcli software sources profile list -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -------------------
ESXi-7.0.0-15843807-standard VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54

Upgrade to a specific Image Profile from ESXi 7.0 Offline Bundle:

esxcli software profile update -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard

Quick Tip - Allow unsupported CPUs when upgrading to ESXi 7.0

by // 58 Comments

As outlined in the vSphere 7.0 release notes (which everyone should carefully read through before upgrading), the following CPU processors are no longer supported:

  • Intel Family 6, Model = 2C (Westmere-EP)
  • Intel Family 6, Model = 2F (Westmere-EX)

To help put things into perspective, these processors were released about 10 years ago! So this should not come as a surprise that VMware has decided remove support for these processors which probably also implies the underlying hardware platforms are probably quite dated as well. In any case, this certainly has affected some folks and from what I have seen, it has mostly been personal homelab or smaller vSphere environments.

One of my readers had reached out to me the other day to share an interesting tidbit which might help some folks prolong their aging hardware for another vSphere release. I have not personally tested this trick and I do not recommend it as you can have other issues longer term or hit a similiar or worse situation upon the next patch or upgrade.

Disclaimer: This is not officially supported by VMware and you run the risk of having more issues in the future.

Per the reader, it looks like you can append the following ESXi boot option which will allow you to bypass the unsupported CPU during the installation/upgrade. To do so, just use SHIFT+O (see VMware documentation for more details) and append the following:

allowLegacyCPU=true

There have also been other interesting and crazy workarounds that attempt to workaround this problem. Although some of these tricks may work, folks should really think long term on what other issues can face by deferring hardware upgrade. I have always looked at homelab as not only a way to learn but to grow yourself as an individual.

Note: The boot option above is only temporarily and you will need to pass in this option upon each restart. It looks like this setting is also not configurable via ESXCLI which I initially had thought, so if you are installing this on a USB device, the best option is to edit the boot.cfg and simply append the parameter to kernelopt line so it'll automatically be included for you without having to manually typing this. If this is install on disk, then you will need to edit both /bootbank/boot.cfg and /altbootbank/boot.cfg for the settings to passed in automatically.

This is ultimately an investment you are making into yourself, so do not cut yourself short and consider looking at a newer platform, especially something like an Intel NUC which is fairly affordable both in cost as well as power, cooling and form factor.

Automated vSphere 7 and vSphere with Kubernetes Lab Deployment Script

by // 112 Comments

I know many of you have been asking me about my vSphere with Kubernetes automation script which I had been sharing snippets of on Twitter. For the past couple of weeks, I have been hard at work making the required changes between the vSphere 7 Beta and GA workflows, some additional testing and of course documentation. Hopefully the wait was worth it (I think it is) and if you enjoy the script or have benefited, please consider adding 🌟to the Github repo to show your support! Thanks and enjoy

The Github repository:

Before getting started, please carefully read through the requirements section along with the complete sample end-to-end execution if you are new to vSphere with Kubernetes. You will need to have a VMware Cloud Foundation (VCF) 4.0 license before you can get started and specifically an NSX-T Advance license which is one of the required parameters within the script. If you do not have access to a VCF 4 license, I strongly recommend taking part in the recent VMUG Advantage Homelab Group Buy effort which I had started to easily get access to the latest VMware releases along with a nice 15% discount!

The script supports deploying both a standard vSphere 7 environment with just VCSA, ESXi and vSAN as well as the complete solution which includes NSX-T to support vSphere with Kubernetes. For more details, please refer to the FAQ.