WilliamLam.com

Potential ESXi Host Preparation issues with NSX 6.3

by // 16 Comments

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.


There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

[Read more...]

Update on Intel NUC 7th Gen (Kaby Lake) & ESXi 6.x

by // 28 Comments

Intel just started shipping their i3 7th Gen (Kaby Lake) Intel NUCs and no surprise, Florian of virten.net has already gotten his hands on a unit for testing. The Intel NUCs is a very popular platform for running vSphere/vSAN-based home labs, especially for their price point and footprint. Last week, Florian discovered from his testing that the built-in network adapter on the 7th Gen NUC was not being detected by any of the ESXi installers and had published his findings here.

Since the Intel NUC is not an officially supported platform for ESXi, it does not surprise me that these sort of things happen, even if the NUC had a pretty good track record going back to the 5th Gen releases. Nonetheless, I reached out to Florian to see if he can provide me with a vm-support bundle and see if there was anything I could do to help.

A couple of Engineers took a look and quickly identified the issue with the Kabylake NIC (8086:15d8) but before getting to the solution, I did want to clarify something. The Kabylake NIC is actually NOT an officially supported NIC for ESXi and although it currently shows up in the VMware HCL, it is a mistake. I have been told the VMware HCL will be updated shortly to reflect this, apologies for any confusion that this may have caused.

Ok, so the great news is that we do have a solution for getting ESXi to recognize the built-in NIC on the 7th Gen Intel NUCs. The semi-bad news is that we currently do not have a solution in the short term for any released version of ESXi, as the fix will require an updated version of the e1000e Native Driver which will only be available in a future update of ESXi. I can not provide any timelines, but keep an eye on this blog and I will publish more details once they are available.

In the meantime, if you already own a 7th Gen NUC, there is a workaround which Florian has already blogged about here which uses the USB Ethernet Adapter VIB for the initial ESXi installation. If you are planning to purchase the 7th Gen NUC and would like to wait for folks to confirm the fix, then I would recommend holding off or potentially looking at the 6th Gen if you can not wait. Thanks to Florian and others who shared their experiences with the 7th Gen NUC and also to the VMware Engineers who found a quick resolution to the problem.

UPDATE (07/27/17) - The updated e1000e Native Driver that was included in ESXi 6.0 Update 3 is now included in ESXi 6.5 Update 1 which just GA'ed. You should be able to install ESXi without require any additional modifications to the latest Intel NUCs.

UPDATE (02/24/17) - An updated e1000e driver which contains a fix for 7th Gen NUC is now available as part of ESXi 6.0 Update 3, below are several options in how you can consume the driver.

[Read more...]

Using the vSphere API in vCenter Server to collect ESXTOP & vscsiStats metrics

by // 1 Comment

Back in 2013, I wrote two articles here and here on how to use the Service Manager API that was introduced in vSphere 5.1 to remotely collect ESXTOP and vscsiStats metrics. At the time, I was told that this API was only available when connecting directly to an ESXi host using the vSphere API. This was still a huge improvement over the previous method which basically required customers to run the commands locally. For ESXTOP, there was resxtop but it was not available for all platforms and some customers still prefer to use the ESXi Shell's version. When I had learned about this API, I was really hoping I could collect both ESXTOP & vscsiStats metrics using vCenter Server which would remove the need to have direct connectivity to each ESXi host.

Last week, an Engineer came across one of my blog posts related to the Service Manager APIs which helped him with a problem he was trying to solve. In the email conversation, I then came to learn from the Engineer that the the Service Manager API can be used from vCenter Server and going directly to the ESXi host was not necessary. It turns out that the QueryServiceList() method which accepts an array of "location" expects a special keyword prefix appended to the list of ESXi hosts that you wish to use the local Service Manager instances on.

The special keyword prefix is "vmware.host." and this is appended to either the Hostname or IP Address of the ESXi hosts being managed by the vCenter Server. For example, in my environment I have an ESXi host (192.168.1.50) that is managed by my VC and so the location string for that host should be "vmware.host.192.168.1.50". If the method was successfully called, you should get back the two service instances for ESXTOP and vscsiStats for each of the ESXi hosts where you can then perform the metric collection.

I have created two sample pyvmomi scripts which exercises the Service Manager API for ESXTOP and vscsiStats:

Note: For more details on how to use Service Manager API to collect ESXTOP and vscsiStats, please refer to this post here and here.

[Read more...]