WilliamLam.com

Functional USB 3.0 Ethernet Adapter (NIC) driver for ESXi 5.5 & 6.0

03.28.2016 by William Lam // 81 Comments

Earlier this month I wrote an article demonstrating a functional USB ethernet adapter for ESXi 5.1. This was made possible by using a custom built driver for ESXi that was created over three years ago by a user named Trickstarter. After having re-discovered the thread several years later, I had tried reaching out to the user but concluded that he/she has probably moved on given the lack of forum activity in the recent years. Over the last few weeks I have been investigating to see if it was possible to compile a new version of the driver that would function with newer versions of ESXi such as our 5.5 and 6.0 release.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (01/22/17) - For details on using a USB-C / Thunderbolt 3 Ethernet Adapter, please see this post here.

UPDATE (11/17/16) - New driver has been updated for ESXi 6.5, please find the details here.

After reaching out to a few folks internally, I was introduced to Songtao Zheng, a VMware Engineer who works on some of our USB code base. Songtao was kind enough to provide some of assistance in his spare time to help with this non-sanction effort that I was embarking on. Today, I am please to announce that we now have a functional USB ethernet adapter driver based on the ASIX AX88179 that works for both ESXi 5.5 and 6.0. This effort could not have been possible without Songtao and I just want to say thank you very much for all of your help and contributions. I think it is safe to say that the overall VMware community also thanks you for your efforts. This new capability will definitely enable new use cases for vSphere home labs that were never possible before when using platforms such as the Intel NUC or Apple Mac Mini for example. Thank you Songtao! I would also like to extend an additional thank you to Jose Gomes, one of my readers, who has also been extremely helpful with his feedback as well as assistance on testing the new drivers.

Now, Before jumping into the goods, I do want to mention there are a few caveats to be aware of and that I think it is important to understand them before making any purchasing decisions.

First and foremost, this is NOT officially supported by VMware, use at your own risk.
Secondly, we have observed there is a substantial difference in transfer speeds between Transmit (Egress) and Receive (Ingress) traffic which may or may not be acceptable depending on your workload. On Receive, the USB network adapter is performing close to a native gigabit interface. However, on Transmit, the bandwidth mysteriously drops by ~50% which includes very inconsistent transfer speeds. We are not exactly sure why this is the case, but given ESXi does not officially support USB based ethernet adapters, it is possible that the underlying infrastructure was never optimized for such devices. YMMV
Lastly, for the USB ethernet adapter to properly function, you will need a system that supports USB 3.0 which kind of makes sense for this type of a solution to be beneficial in the home lab. If you have a system with USB 2.0, the device will probably not work at least from testing that we have done.

Note: For those interested in the required source code changes to build the AX88179 driver, I have published all of the details on my Github repo here.

Disclaimer: In case you some how missed it, this is not officially supported by VMware. Use at your own risk.

Without further ado, here are the USB 3.0 gigabit ethernet adapters that are supported with the two drivers:

StarTech USB 3.0 to Gigabit Ethernet NIC Adapter
StarTech USB 3.0 to Dual Port Gigabit Ethernet Adapter NIC with USB Port
j5create USB 3.0 to Gigabit Ethernet NIC Adapter (verified by reader Sean Hatfield 03/29/16)
Vantec CB-U300GNA USB 3.0 Ethernet Adapter (verified by VMware employee 05/19/16)
DUB-1312 USB 3.0 Gigabit Ethernet Adapter (verified by twitter user George Markou 07/29/16)

Note: There may be other USB ethernet adapters that uses the same chipset which could also leverage this driver but these are the only two that have been verified.

Here are the ESXi driver VIB downloads:

Note: Although the drivers were compiled against a specific version of ESXi, they should also work on the same major version of ESXi, but I have not done that level of testing and YMMV.

Verify USB 3.0 Support

As mentioned earlier, you will need a system that is USB 3.0 capable to be able to use the USB ethernet adapter. If you are unsure, you can plug in a USB 3.0 device and run the following command to check:

lsusb

What you will be looking for is an entry stating "Linux Foundation 3.0 root hub" which shows that ESXi was able to detect a USB 3.0 port on your system. Secondly, look for the USB device you just plugged in and ensure the "Bus" ID matches that of the USB 3.0 bus. This will tell you if your device is being claimed as a USB 3.0 device. If not, you may need to update your BIOS as some systems may have USB 2.0 enabled by default like earlier versions of Intel NUC as desribed here. You may also be running pre-ESXi 5.5 which did not support USB 3.0 as mentioned here, so you may need to upgrade your ESXi host to at least 5.5 or greater.

Install Driver

You can either install the VIB directly onto your ESXi host or by creating a custom ESXi ISO that includes the driver using a popular tool like ESXi Customizer by Andreas Peetz.

To install the VIB, upload the VIB to your ESXi host and then run the following ESXCLI command specifying the full path to the VIB:

esxcli software vib install -v /vghetto-ax88179-esxi60u2.vib -f

Lastly, you will need to disable the USB native driver to be able to use this driver. To do so, run the following command:

esxcli system module set -m=vmkusb -e=FALSE

You will need to reboot for the change to go into effect.

To verify that the USB network adapter has been successfully claimed, run either of the following commands to list your physical NICs:

esxcli network nic list
esxcfg-nics -l

To add the USB uplink, you will need to either use the vSphere Web Client or ESXCLI to add the uplink to either a Virtual or Distributed Virtual Switch.

To do so using ESXCLI, run the following command and specify the name of your vSwitch:

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Uninstall Driver

To uninstall the VIB, first make sure to completely unplug the USB network adapter from the ESXi first. Next, run the following ESXCLI command which will automatically unload the driver and remove the VIB from your ESXi host:

esxcli software vib remove -n vghetto-ax88179-esxi60u2

Note: If you try to remove the VIB while the USB network adapter is still plugged in, you may hang the system or cause a PSOD. Simply reboot the system if you accidentally get into this situation.

Troubleshooting

If you are not receiving link on the USB ethernet adapter, it is most likely that your system does not support USB 3.0. If you find the a similar message like the one below in /var/log/vmkernel.log then you are probably running USB 1.0 or 2.0.

2016-03-21T23:30:49.195Z cpu6:33307)WARNING: LinDMA: Linux_DMACheckConstraints:138: Cannot map machine address = 0x10f5b6b44, length = 2 for device 0000:00:1d.7; reason = address exceeds dma_mask (0xffffffff))

Persisting USB NIC Configurations after reboot

ESXi does not natively support USB NIC and upon a reboot, the USB NICs are not picked up until much later in the boot process which prevents them from being associated with VSS/VDS and their respective portgroups. To ensure things are connected properly after a reboot, you will need to add something like the following in /etc/rc.local.d/local.sh which re-links the USB NIC along with the individual portgroups as shown in the example below.

esxcfg-vswitch -L vusb0 vSwitch0
esxcfg-vswitch -M vusb0 -p "Management Network" vSwitch0
esxcfg-vswitch -M vusb0 -p "VM Network" vSwitch0

You will also need to run /sbin/auto-backup.sh to ensure the configuration changes are saved and then you can issue a reboot to verify that everything is working as expected.

Summary

For platforms that have limited built-in networking capabilities such as the Intel NUC and Apple Mac Mini, customers now have the ability to add additional network interfaces to these systems. This will now open up a whole new class of use cases for vSphere based home labs that were never possible before, especially with solutions such as VSAN and NSX. I look forward to seeing what our customers can now do with these new networking capabilities.

Additional Info

Here are some additional screenshots testing the dual USB 3.0 ethernet adapter as well as a basic iPerf benchmark for the single USB ethernet adapter. I was not really impressed with the speeds for the dual ethernet adapter which I had shared some more info here. Unless you are limited on number of USB 3.0 ports, I would probably recommend just sticking with the single port ethernet adapter.

iPerf benchmark for Ingress traffic (single port USB ethernet adapter):

iPerf benchmark for Egress traffic (single port USB ethernet adapter):

Automating the configuration of new logon banner for the vSphere Web Client in 6.0 Update 2

03.22.2016 by William Lam // 1 Comment

The ability to display a logon banner prior to logging into the vSphere Web Client UI is a new capability that was introduced with the release of vSphere 6.0 Update 2. In addition to the logon banner message, customers can also configure a mandatory consent check box requiring all users to accept prior to logging in. Below is a screenshot of the new logon banner in the vSphere Web Client, prior to login.

The configuration of the logon banner is only available in the Platform Services Controller (PSC) Administrator UI which can be accessed by opening a browser to the following URL: https://[PSC-HOSTNAME]/psc

Although there is currently not an API for administrating the various PSC configurations (being worked on for a future release of vSphere), it is still possible to automate the different aspects of the PSC. There are several options depending on your personal reference as well as environmental configurations.

Option 1: Use the /opt/vmware/bin/sso-config.sh utility on the PSC itself which is the simplest method but it does require either SSH or local shell access.

You first need to create a file that contains the message you wish to display in the logon banner. In this example I have created a filed called banner.txt that contains an example logon message. You can then run the following command which will allow you to set the banner as well as the title and whether to enable the consent check box:

/opt/vmware/bin/sso-config.sh -set_logon_banner /root/banner.txt -title 'Logon Disclaimer' -enable_checkbox N

If you wish to only configure the consent check box, you can just run the following command:

/opt/vmware/bin/sso-config.sh -set_logon_banner -enable_checkbox Y

If you wish to only configure the title, you can just run the following command:

/opt/vmware/bin/sso-config.sh -set_logon_banner -title 'vGhetto Disclaimer'

If you wish to only configure the banner message, you can just run the following command:

/opt/vmware/bin/sso-config.sh -set_logon_banner /root/banner.txt

If you wish to disable the logon banner feature, you can run the following command:

/opt/vmware/bin/sso-config.sh -disable_logon_banner

Option 2: Use the ldapmodify utility either locally on the PSC itself or from a remote system which I have previously written about here.

Disclaimer: Please take extreme caution when connecting to the vmdird database. You should take extreme care in making changes while in the database else you can negatively impact your environment.

In this example, I will be remotely connecting to the PSC from my desktop which is a Mac OS X system, but it can be any system which contains the ldapsearch and ldapmodify commands.

To query for the logon banner configurations, run the following ldapsearch command (specifying your environment details):

/usr/bin/ldapsearch -h 192.168.1.140 -w 'VMware1!' -x -D "cn=Administrator,cn=Users,dc=vghetto,dc=local" -b "cn=Tenants,cn=IdentityManager,cn=Services,dc=vghetto,dc=local" -s sub "objectclass=vmwSTSTenant"

From the output, we can see there are three properties which control the logon banner feature:

vmwSTSLogonBannerEnableCheckbox
vmwSTSLogonBannerTitle
vmwSTSLogonBanner

To configure the new logon banner, we first need to create a file that contains our configuration changes. To do so, create a file called change.ldif which contains the following (replace with your settings):

dn: cn=vghetto.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vghetto,dc=local
changetype: modify
replace: vmwSTSLogonBannerEnableCheckbox
vmwSTSLogonBannerEnableCheckbox: TRUE
-
replace: vmwSTSLogonBanner
vmwSTSLogonBanner: You void all warranty/safety by logging into vGhetto Datacenter, you have been warned :-)
-
replace: vmwSTSLogonBannerTitle
vmwSTSLogonBannerTitle: vGhetto Disclaimer

To apply the configuration, run the following ldapmodify command (specifying your environment details):

/usr/bin/ldapmodify -f change.ldif -h 192.168.1.140 -D "cn=Administrator,cn=Users,dc=vghetto,dc=local" -w 'VMware1!'

To completely disable the logon banner feature, create a new file called change2.ldif which contains the following (replace with your settings):

dn: cn=vghetto.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vghetto,dc=local
changetype: modify
delete: vmwSTSLogonBannerEnableCheckbox
-
delete: vmwSTSLogonBanner
-
delete: vmwSTSLogonBannerTitle

To apply the configuration, run the following ldapmodify command (specifying your environment details):

/usr/bin/ldapmodify -f change.ldif -h 192.168.1.140 -D "cn=Administrator,cn=Users,dc=vghetto,dc=local" -w 'VMware1!'

If you are interested in automating other aspects of the PSC admin configurations, be sure to check out these two articles here and here.

VM serial logging to the rescue for capturing Nested ESXi PSOD

03.21.2016 by William Lam // Leave a Comment

I frequently deploy pre-releases of our software to help test and provide early feedback to our Engineering teams. One piece of software that I deploy some what frequently is our ESXi Hypervisor and the best way to deploy it, is of course inside of a Virtual Machine or commonly referred to as Nested ESXi.

Most recently while testing a new ESXi build in my lab (screenshot below is for demo purposes, not the actual PSOD image), I encountered an ESXi purple screen of death (PSOD) during the bootup of the ESXi Installer itself. Since ESXi had not been installed, there was no place for ESXi to actually store the core dumps which made it challenging when filing a bug with Engineering as screenshots may not always contain all the necessary details.

Luckily, because we are running in a VM, a really neat feature that VMware has supported for quite some time now is configuring a virtual serial port for logging purposes. In fact, one of the neatest feature from a troubleshooting standpoint was the introduction of the Virtual Serial Port Concentrator (vSPC) feature in vSphere 5.0 which allowed a VM to log directly to a serial console server just like you would for physical servers. You of course had few other options of either logging directly to the serial port of the physical ESXi, named pipe or simply to a file that lived on a vSphere Datastore.

Given this was a home lab setup, the easiest method was to simply output to a file. To add a virtual serial port, you can either use the vSphere Web/C# Client or the vSphere APIs. Since this is not something I need to do often, I just used the UI. Below is a screenshot using the vSphere Web Client and once you have added the virtual serial port, you need to specify the filename and where to the store the output file by clicking on the "Browse" button.

If the GuestOS which includes ESXi has been configured to output to a serial port, the next time there is an issue and you can easily captured the output to a file instead of just relying on a screenshot. One additional tip which might be useful is by default, vSphere will prompt whether you want to replace or append to the configured output file. If you wish to always replace, you can add the following VM Advanced Setting and you will not get prompted in the UI.

answer.msg.serial.file.open = "Replace"

Virtual serial ports are supported on both vSphere (vCenter Server + ESXi) as well as our hosted products VMware Fusion and Workstation.