WilliamLam.com

Important - NVMe SSD not found after upgrading to ESXi 7.0

by // 15 Comments

Several folks in the community had reported issues that after upgrading from ESXi 6.7 to 7.0, their Samsung NVMe PCIe SSDs were no longer showing up. The first report of this was from Ivo Beerens and eventually found that reinstalling ESXi from scratch works but certainly that was not ideal. Just yesterday, I saw that Jeffrey Kusters also shared a similiar issue and used a different workaround which allowed him to upgrade. I also reached out to VMware Engineering as they thought this was a strange behavior but needed to see the logs to understand what was actually going on. Since Jeffrey's setup was an upgrade, I was able to get a copy of his vm-support bundle to provide to Engineering.

Within minutes of looking at the support bundle, they quickly identified the issue and this was caused by using the incorrect ESXCLI command to upgrade a standalone ESXi host from 6.7 to 7.0. Instead of using "esxcli software vib update" command, folks should be using "esxcli software profile update" which has always been the correct command to use when upgrading an ESXi image. In fact, this has been in the vSphere documentation for quite some time and here is the ESXi 7.0 version of that documentation. More importantly, the incorrect command only upgrades the ESXi 6.7 VIBs that exists and does not install any of the ESXi 7.0 VIBs, which means after the upgrade, you are not only missing the nvme-pcie VIB but many other ESXi 7.0 VIBs!

tl;dr - If you are going to use ESXCLI to upgrade your standalone ESXi host, please make sure to use the correct command or you will have issues. Below are the two commands you will need to determine which ESXi Image Profiles are available given an offline bundle and then updating to a specific image profile.

List Image Profiles from ESXi 7.0 Offline Bundle:

[[email protected]:~] esxcli software sources profile list -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip
Name Vendor Acceptance Level Creation Time Modification Time
---------------------------- ------------ ---------------- ------------------- -------------------
ESXi-7.0.0-15843807-standard VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools VMware, Inc. PartnerSupported 2020-03-16T10:48:54 2020-03-16T10:48:54

Upgrade to a specific Image Profile from ESXi 7.0 Offline Bundle:

esxcli software profile update -d /vmfs/volumes/e200-8d-local-datastore/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard

Automated NSX-T 2.0 Lab Deployment

by // 21 Comments

Last week, I had spent some time exploring and getting myself more familiar with NSX-T, which is the next generation release of the NSX platform from VMware. One of the first thing I do when learning about a new product is to setup a lab environment that I can using. Having gone through the deployment once by hand, I realized it would be quite painful if I needed to do this again, which I know I will and I did 🙂 I wanted to have a simliar experience to my vGhetto Automated vSphere Lab deployment script which also including setting up the entire vSphere infrastructure along with deploying and configuring NSX-V and extending it to support NSX-T.

Since my original script leverages PowerCLI to access both the vSphere and NSX APIs, I wanted to do the same with NSX-T. Funny enough, the PowerCLI team had just published an update release (6.5.3) which also added support for NSX-T and I thought this was perfect timing to try out the NSX-T APIs, which I had never used before.

UPDATE (01/01/2018) - I have verified the script also works with the latest NSX-T 2.1 which was just released before Christmas. The script has also been updated to create a new Edge Uplink Profile along with an Edge Cluster and automatically associate all Edge VMs to Edge Cluster.

I have created a new Github repository called vghetto-nsxt-automated-lab-deployment which contains detailed instructions along with the PowerCLI script.

Here is what the script is currently performing:

  1. Deploy and configure vCenter Server Appliance 6.5u1
  2. Deploy and configure 3 x Nested ESXi 6.5u1 Virtual Appliance VMs and attaching it to vCenter Server
  3. Deploy NSX-T Manager, 3 x Controllers & 1 x Edge and setup both the Management and Control Cluster Plane
  4. Configure NSX-T with IP Pool, Transport Zone, Add vCenter Server as Compute Manager, Create Logical Switch, Prepare ESXi hosts, Create Uplink Profile & Add configure ESXi hosts as a Transport Node

Similiar to the vSphere version of this script, all deployed VMs will be placed inside of a vCenter vApp construct as shown in the example screenshot below:


Here is an example output of a succesful deployment and you go from nothing to a fully functional NSX-T environment in just 50 minutes, which is pretty awesome if you ask me!?

[Read more...]

Quick Tip - VSAN 6.2 (vSphere 6.0 Update 2) now supports creating all-flash diskgroup using ESXCLI

by // 5 Comments

One of my all time favorite features of VSAN is still the ability to be able to "bootstrap" a VSAN Datastore starting with just a single ESXi node. This is especially useful if you would like to bootstrap vCenter Server on top of VSAN out of the box without having to require additional VMFS/NFS storage. This bootstrap method has been possible and supported since the very first release of VSAN which I have written in great detail here and here.

With the release of VSAN 6.1 (vSphere 6.0 Update 1), an all-flash VSAN configuration was also now possible in addition to a hybrid configuration which uses a combination of SSDs and MDs. One observation that was made by a few folks including myself was that you could not configure an all-flash diskgroup using ESXCLI which was one of the methods that could be used to bootstrap VSAN. If you tried to create an all-flash diskgroup using ESXCLI, you would get the following error:

Unable to add device: Can not create all-flash disk group: current Virtual SAN license does not support all-flash

This turned out to be a bug and the workaround at the time was to add the ESXi host to a vCenter Server which would then allow you to create the all-flash diskgroup. This usually was not a problem but for those wanting to bootstrap VSAN, this would require you to have an already running vCenter Server instance. While setting up my new VSAN 6.2 home lab last night

I found that this issue has actually been resolved in the upcoming release of VSAN 6.2 (vSphere 6.0 Update 2) and you can now create an all-flash diskgroup using ESXCLI which includes do so from the vSphere API as well. For those interested, you can find the list commands required to bootstrap an all-flash VSAN configuration below:

Step 1 - You will need to change the default VM Storage Policy on VSAN to allow "Force Provisioning" since you only have a single node to start with (we will change this back to default once you have deployed vCenter Server):

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"

Step 2 - Ensure that you enable VSAN traffic type on a specific VMkernel interface, in this example, I am using vmk0 by running the following command:

esxcli vsan network ipv4 add -i vmk0

Step 3 - Create a new VSAN Cluster by running the following command:

esxcli vsan cluster new

Step 4 - Run the following command to identify the SSD devices you plan to use for your "Caching" and "Capacity" Tier, specifically you will need to make a note of the device you plan to use for "Capacity" as we will need to tag that device in the next step.

vdq -q

Screen Shot 2016-03-01 at 5.40.25 PM
Step 5 - To tag the specific SSD device as "Capacity", run the following command and substitute the ID of the SSD device from previous step:

esxcli vsan storage tag add -d t10.ATA_____Samsung_SSD_850_EVO_500GB_______________S2RANXAH138511D_____ -t capacityFlash

You can re-run the vdq -q command to verify that the IsCapacityFlash property is now set for the device you just tagged.

Step 6 - Lastly, we can will now create our all-flash VSAN diskgroup by running the following command using the information we found in Step 4:

esxcli vsan storage add -s t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S24BNXAH105839L_____ -d t10.ATA_____Samsung_SSD_850_EVO_500GB_______________S2RANXAH138511D_____

Screen Shot 2016-03-01 at 5.42.23 PM
Step 7 - If everything was configured correctly, we view our new all-flash diskgroup by running the following command:

esxcli vsan storage list

Screen Shot 2016-03-01 at 5.45.17 PM
At this point, you are now ready to provisioning vCenter Server on top of the VSAN Datastore and once that is setup, you can use then use the vSphere Web Client to add your remainder VSAN nodes as you normally would.

Note: If you do NOT plan on running a *single* VSAN node (not recommended but its possible), then remember to change back the original VM Storage Policy settings once you have setup your vCenter Server by running the following ESXCLI command on that first ESXi node used to bootstrap VSAN:

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0))"