ESXi – Page 30

vSphere 6.0 Update 2 hints at Nested ESXi support for Paravirtual SCSI (PVSCSI) in the future

03.14.2016 by William Lam // 6 Comments

Although Nested ESXi (running ESXi in a Virtual Machine) is not officially supported today, VMware Engineering continues to enhance this widely used feature by making it faster, more reliable and easier to consume for our customers. I still remember that it was not too long ago that if you wanted to run Nested ESXi, several non-trivial and manual tweaks to the VM's VMX file were required. This made the process of consuming Nested ESXi potentially very error prone and provide a less than ideal user experience.

Things have definitely been improved since the early days and here are just some of the visible improvements over the last few years:

Prior to vSphere 5.1, enabling Virtual Hardware Assisted Virtualization (VHV) required manual edits to the VMX file and even earlier versions required several VMX entries. VHV can now be easily enabled using either the vSphere Web Client or the vSphere API.
Prior to vSphere 5.1, only the e1000{e} networking driver was supported with Nested ESXi VMs and although it was functional, it also limited the types of use cases you might have for Nested ESXi. A Native Driver for VMXNET3 was added in vSphere 5.1 which not only increased the performance that came with using the optimized VMXNET3 driver but it also enabled new use cases such testing SMP-FT as it was now possible to get 10Gbe interface to Nested ESXi VM versus the traditional 1GBe with e1000{e} driver.
Prior to vSphere 6.0, selection of ESXi GuestOS was not available in the "Create VM" wizard which meant you had to resort to re-editing the VM after initial creation or using the vSphere API. You can now select the specific ESXi GuestOS type directly in the vSphere Web/C# Client.
Prior to vSphere 6.0, the only way to cleanly shutdown or power cycle a Nested ESXi VM was to perform the operation from within the system as there was no VMware Tools support. This changed with the development of a VMware Tools daemon specifically for Nested ESXi which started out as a VMware Fling. With vSphere 6.0, the VMware Tools for Nested ESXi was pre-installed by default and would automatically startup when it detected that it ran as a VM. In addition to power operations provided by VMware Tools, it also enabled the use of the Guest Operations API which was quite popular from an Automation standpoint.

Yesterday while working in my new vSphere 6.0 Update 2 home lab, I needed to create a new Nested ESXi VM and noticed something really interesting. I used the vSphere Web Client like I normally would and when I went to select the GuestOS type, I discovered an interesting new option which you can see from the screenshot below.

It is not uncommon to see VMware to add experimental support for potentially new Guest Operating Systems in vSphere. Of course, there are no guarantees that these OSes would ever be supported or even released for that matter.

What I found that was even more interesting was that when select this new ESXi GuestOS type (vmkernel65) is what was recommended as the default virtual hardware configuration for the VM. For the network adapter, it looks like the VMXNET3 driver is now recommended over the e1000e and for the storage adapter the VMware Paravirtual (PVSCSI) adapter is now recommended over the LSI Logic Parallel type. This is really interesting as it is currently not possible today to get the optimized and low overhead of the PVSCSI adapter working with Nested ESXi and this seems to indicate that PVSCSI might actually be possible in the future! 🙂

I of course tried to install the latest ESXi 6.0 Update 2 (not yet GA'ed) using this new ESXi GuestOS type and to no surprise, the ESXi installer was not able to detect any storage devices. I guess for now, we will just have to wait and see ...

Working USB Ethernet Adapter (NIC) for ESXi

03.01.2016 by William Lam //

As part of upgrading my personal vSphere home lab from an Apple Mac Mini to an Intel NUC (more on this in a future blog), I have been researching to see if there are other alternatives for adding an additional network adapter. The Intel NUC only includes a single built-in ethernet adapter which is similar to the Mac Mini. However, the NUC also lacks additional IO connectors like a Thunderbolt port which the Mac Mini includes and can support a Thunderbolt to Ethernet adapter. I think this is probably the only downside to the Intel NUC platform which has been similar feedback that I have seen from other vSphere home labbers who currently use or would like to use the NUC. Perhaps, with the next update of the NUC platform code named "Skull Canyon", the rumored Thunderbolt 3 / USB-c connector may make things easier as some of the existing vendors who produce Thunderbolt to ethernet adapter also use common drivers like the Broadcom tg3 which have historically worked with ESXi.

One option that has been suggested by many folks over the years was to see if a USB based ethernet adapter could be used to provide additional networking to ESXi? Historically, the answer had been no because there were no known device drivers that would work with ESXi. I had even looked into this a few years ago and although I ran into some folks who seemed to have made it work, I was never able to find the right USB ethernet adapter to personally confirm myself. It was only until last week, I decided to start fresh again and after a bit of Googling I came across an old VMTN thread here where VMTN user AK_____28 mentioned he had success with the StarTech USB 3.0 to Gigabit Ethernet NIC Network Adapter and using a custom compiled driver that was posted over here by another user named Trickstarter.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (03/29/16) - Please have a look at this updated article here which includes ESXi 5.5 and 6.0 driver.

Disclaimer: In case it is not clear and apparent, you should never install any unknown 3rd party software on your ESXi host as it can potentially lead to instability issues or worse open yourself to a security hole. The following solution is not officially supported by VMware, please use at your own risk.

I decided to bite the bullet and give this solution a try and purchased the USB ethernet adapter from Amazon here.

There are two modules that needs to be downloaded, extracted and loaded onto ESXi. I have included the links below for your convenience:

ax88179vz026.gz
usbnetvz026.gz

As the VMTN thread mentioned, you can load using either the vmkload_mod or ESXCLI. Here are the two commands that I used in the following order:

vmkload_mod /vmfs/volumes/mini-local-datastore-1/usbnetvz026
vmkload_mod /vmfs/volumes/mini-local-datastore-1/ax88179vz026

When I tried to initially load either of the modules, I would always get the following error:

vmkwarning.log:2016-02-28T21:54:54.531Z cpu6:374787)WARNING: Elf: 2041: Load of <usbnetvz026> failed : missing required namespace <com.vmware.usb#9.2.1.0>

As you can imagine, I was pretty bummed to see this since I was afraid that something like this would happen. I was not sure if the device I had purchased no longer worked or if was the drivers? I saw that these modules were initially compiled for ESXi 5.1 (at the time, that was the latest version) and the only difference was that I was using a much newer version of ESXi, specifically 6.0 Update 1. I decided to install the latest version of ESXi 5.1 Update 3 and tried the process again and to my surprise, the modules loaded without errors. I suspect that this was a hard dependency on the namespace version which was version 9.2.1.0 and the latest version is now 9.2.3.0.

After successfully loading the two modules, I ran the following command:

esxcfg-nics -l

to verify that ESXi did in fact did claim the USB ethernet device and as you can see from the screenshot below, it did indeed!

Next up, I needed to verify basic connectivity and added the new uplink to my existing vSwitch. You must use the following ESXCLI command (esxcfg-vswitch command does not work apparently for non vmnicX devices)

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Once added, I hopped over to the vSphere C# Client to see if the device is now showing up under the Network Adapters tab, which it is.

Finally, the last test was to make the vsb0 (this is how ESXi names the device) device the active connection while moving my existing vmnic0 to stand-by. Networking connectivity continued to function and I was even able to transfer an ISO image over the USB ethernet adapter without any issues.

So it looks like it is possible to get a USB based ethernet adapter to function with ESXi, at least with the specific model listed above (PCI ID 0b95:1790). The challenge now is to see if there is a way to build an updated version of the drivers targeted at the latest ESXi 6.0 release. From what I have been able to follow on the forum here, it looks like there was also some amount of non-trivial code changes that were required to get the driver to function. If true, without those changes, it can difficult to re-compile the driver. I have reached out to the original author to see if he might be able to share the changes he made to the driver code. In the mean time, if folks are interested in giving the build process a try, Trickstarter did a great two part write up on how to setup your build environment and compile an example driver.

Although the write up is targeted at ESXi 5.x, you can download the equilvenet packages for ESXi 6.0 which includes the ESXi Open Source Disclosure Package as well as the VMware Toolchain which is required and used to compile the source code. I have provided the direct download links below.

You can also find the latest version of the USB ethernet adapter ax88179 ASIX driver here. I have also attempted to compile just the driver but have already ran into some issues. I have not had time to dig further, so not sure how far I will be able to get. Any tips or tricks others may have for compiling against ESXi 6.0, feel free to share them and I will give them a shot when I get some time!

How to limit the number "Logical" CPUs in ESXi?

02.09.2016 by William Lam // 13 Comments

Last week I received a very strange customer inquiry in which they would like to limit the number of physical CPU sockets seen by ESXi. As you can imagine, my interest was piqued as this is usually not the type of request you hear from customers looking to actively reduce the overall computing power of their underlying hardware, especially if they have paid for it. After digging into the details a bit more, it turns out this is related to licensing.

The customer is running an application on a VM which is licensed by the total number of underlying physical CPU sockets of the server, regardless if they are actively being used by the application or not. The vendor shall be left nameless but I am sure some of you can make some educated guesses 🙂 The customer was in the process of performing a hardware refresh where they would be moving from a 4 socket CPU to an 8 socket CPU and they would be negatively impacted by this change from a licensing standpoint. I can understand their concerns, they have now just doubled their application licensing cost without actually benefiting from the actual hardware update.

Unfortunately, after a bit more research, I found that it is not possible to reduce or limit the number of physical CPU sockets from ESXi. The only capability that we do have today around this topic is to limit the number of Logical CPUs that ESXi can see. This capability is exposed as an ESXi Advanced Setting called VMkernel.Boot.maxPCPUS and by default, this is set to unlimited as you would expect. What this setting does is takes the total number of logical CPUs that you wish to expose to ESXi and then evenly distributes that across your physical CPU Sockets as best as it can.

You can change this setting using a variety of methods including the vSphere Web Client, vSphere C# Client, ESXi Embedded Host Client, vSphere API which includes ESXCLI & PowerCLI. One alternative which I have seen some postings online about which is the ability to turn off specific CPU sockets for certain hardware platforms by using the system BIOs, having said that, I have not actually seen any real confirmation that this is in fact possible.

Below are screenshots using the vSphere C# & ESXi Embedded Host.

If you prefer to to use the CLI either locally or remotely, you can run the following ESXCLI commands:

List the current configurations

esxcli system settings kernel list -o maxPCPUS

Set a new configuration

esxcli system settings kernel list -s maxPCPUS -v 4