WilliamLam.com

How to deploy Tanzu Kubernetes Grid (TKG) Cluster with Antrea CNI 

by // 1 Comment

I have been working with Tanzu Kubernetes Grid (TKG) quite a bit lately and using their new slick TKG CLI for deploying standalone Tanzu Kubernetes Clusters (TKC) which can run in both VMware Cloud on AWS as well as your on-premises vSphere 6.7 Update 3 environment. If you have vSphere 7 and the vSphere with Kubernetes capability, it also supports TKG deployments natively as part of that solution but you can also use TKG CLI to deploy TKC's.

Out of the box, TKG includes all the necessary software components to deploy a production grade, upstream and conformant Kubernetes distribution. For most customers, the "batteries included" type of offering is more than sufficient but for some customers who may wish to customize some of these components further when running the standalone distribution. One such example is swapping out the default Container Network Interface (CNI) which uses Calico for a different CNI with more capabilities.


As you may have guess from the title of this post, we will be replacing Calico with Antrea which is another open source CNI. In fact, Antrea was started by VMware last year and uses Open vSwitch (OVS) to provide network and security capabilities to Kubernetes. You can read more about Project Antrea here and more details about its architecture can be found here.

Disclaimer: This is currently not officially supported by VMware. I do know the TKG team is looking at Antrea support in the future.

[Read more...]

Heads Up - Nested ESXi crashes in ESXi 7.0 running on older CPUs

by // 27 Comments

Thanks to Patrik Kernstock, who works in our Technical Support organization at VMware, for making me aware of an issue related to Nested ESXi running on an ESXi host that has been upgraded to ESXi 7.0. Several folks in the community have noticed after upgrading their Intel NUC 7th Gen and deploying a Nested ESXi VM and powering on an inner-guestOS would causes the Nested ESXi VM to crash.

Upon further investigation, it looks like this is not specific to the Intel NUC platform but rather with a specific generation of CPUs which are Intel Sky Lake-based and as a result, some customers are noticing this affect on their 7th Gen NUC.

UPDATE (06/23/20) - ESXi 7.0b has just been released and contains the fix for the Nested ESXi VM crash. If you are using an Intel NUC 10, do not just apply the patch as the updated ne1000 VIB within the patch will override existing Intel NIC driver causing the network adapter to no longer function. It is recommended that you download the patch and replace the default ne1000 VIB using Image Builder with the Intel NIC version before applying the update. To download the patch, please visit VMware Patch Portal site.

The good news is that this issue has already been reported and we should have a fix in a future update of ESXi. In the meantime, you can still run Nested ESXi and Nested Virtualization on these affected CPUs, you just will not be able to power on inner-guest VMs. Big thanks to Patrik for helping out with the testing and triaging this internally.

Important - NVMe SSD not found after upgrading to ESXi 7.0

by // 17 Comments

Several folks in the community had reported issues that after upgrading from ESXi 6.7 to 7.0, their Samsung NVMe PCIe SSDs were no longer showing up. The first report of this was from Ivo Beerens and eventually found that reinstalling ESXi from scratch works but certainly that was not ideal. Just yesterday, I saw that Jeffrey Kusters also shared a similiar issue and used a different workaround which allowed him to upgrade. I also reached out to VMware Engineering as they thought this was a strange behavior but needed to see the logs to understand what was actually going on. Since Jeffrey's setup was an upgrade, I was able to get a copy of his vm-support bundle to provide to Engineering.

Within minutes of looking at the support bundle, they quickly identified the issue and this was caused by using the incorrect ESXCLI command to upgrade a standalone ESXi host from 6.7 to 7.0. Instead of using "esxcli software vib update" command, folks should be using "esxcli software profile update" which has always been the correct command to use when upgrading an ESXi image. In fact, this has been in the vSphere documentation for quite some time and here is the ESXi 7.0 version of that documentation. More importantly, the incorrect command only upgrades the ESXi 6.7 VIBs that exists and does not install any of the ESXi 7.0 VIBs, which means after the upgrade, you are not only missing the nvme-pcie VIB but many other ESXi 7.0 VIBs!

tl;dr - If you are going to use ESXCLI to upgrade your standalone ESXi host, please make sure to use the correct command or you will have issues. Below are the two commands you will need to determine which ESXi Image Profiles are available given an offline bundle and then updating to a specific image profile.

List Image Profiles from ESXi 7.0 Offline Bundle:

[root@e200-8d:~] esxcli software sources profile list -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -------------------
ESXi-7.0.0-15843807-standard VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54

Upgrade to a specific Image Profile from ESXi 7.0 Offline Bundle:

esxcli software profile update -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard