As part of upgrading my personal vSphere home lab from an Apple Mac Mini to an Intel NUC (more on this in a future blog), I have been researching to see if there are other alternatives for adding an additional network adapter. The Intel NUC only includes a single built-in ethernet adapter which is similar to the Mac Mini. However, the NUC also lacks additional IO connectors like a Thunderbolt port which the Mac Mini includes and can support a Thunderbolt to Ethernet adapter. I think this is probably the only downside to the Intel NUC platform which has been similar feedback that I have seen from other vSphere home labbers who currently use or would like to use the NUC. Perhaps, with the next update of the NUC platform code named "Skull Canyon", the rumored Thunderbolt 3 / USB-c connector may make things easier as some of the existing vendors who produce Thunderbolt to ethernet adapter also use common drivers like the Broadcom tg3 which have historically worked with ESXi.
One option that has been suggested by many folks over the years was to see if a USB based ethernet adapter could be used to provide additional networking to ESXi? Historically, the answer had been no because there were no known device drivers that would work with ESXi. I had even looked into this a few years ago and although I ran into some folks who seemed to have made it work, I was never able to find the right USB ethernet adapter to personally confirm myself. It was only until last week, I decided to start fresh again and after a bit of Googling I came across an old VMTN thread here where VMTN user AK_____28 mentioned he had success with the StarTech USB 3.0 to Gigabit Ethernet NIC Network Adapter and using a custom compiled driver that was posted over here by another user named Trickstarter.
UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.
UPDATE (03/29/16) - Please have a look at this updated article here which includes ESXi 5.5 and 6.0 driver.
Disclaimer: In case it is not clear and apparent, you should never install any unknown 3rd party software on your ESXi host as it can potentially lead to instability issues or worse open yourself to a security hole. The following solution is not officially supported by VMware, please use at your own risk.
I decided to bite the bullet and give this solution a try and purchased the USB ethernet adapter from Amazon here.
There are two modules that needs to be downloaded, extracted and loaded onto ESXi. I have included the links below for your convenience:
As the VMTN thread mentioned, you can load using either the vmkload_mod or ESXCLI. Here are the two commands that I used in the following order:
vmkload_mod /vmfs/volumes/mini-local-datastore-1/usbnetvz026
vmkload_mod /vmfs/volumes/mini-local-datastore-1/ax88179vz026
When I tried to initially load either of the modules, I would always get the following error:
vmkwarning.log:2016-02-28T21:54:54.531Z cpu6:374787)WARNING: Elf: 2041: Load of <usbnetvz026> failed : missing required namespace <com.vmware.usb#9.2.1.0>
As you can imagine, I was pretty bummed to see this since I was afraid that something like this would happen. I was not sure if the device I had purchased no longer worked or if was the drivers? I saw that these modules were initially compiled for ESXi 5.1 (at the time, that was the latest version) and the only difference was that I was using a much newer version of ESXi, specifically 6.0 Update 1. I decided to install the latest version of ESXi 5.1 Update 3 and tried the process again and to my surprise, the modules loaded without errors. I suspect that this was a hard dependency on the namespace version which was version 9.2.1.0 and the latest version is now 9.2.3.0.
After successfully loading the two modules, I ran the following command:
esxcfg-nics -l
to verify that ESXi did in fact did claim the USB ethernet device and as you can see from the screenshot below, it did indeed!
Next up, I needed to verify basic connectivity and added the new uplink to my existing vSwitch. You must use the following ESXCLI command (esxcfg-vswitch command does not work apparently for non vmnicX devices)
esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0
Once added, I hopped over to the vSphere C# Client to see if the device is now showing up under the Network Adapters tab, which it is.
Finally, the last test was to make the vsb0 (this is how ESXi names the device) device the active connection while moving my existing vmnic0 to stand-by. Networking connectivity continued to function and I was even able to transfer an ISO image over the USB ethernet adapter without any issues.
So it looks like it is possible to get a USB based ethernet adapter to function with ESXi, at least with the specific model listed above (PCI ID 0b95:1790). The challenge now is to see if there is a way to build an updated version of the drivers targeted at the latest ESXi 6.0 release. From what I have been able to follow on the forum here, it looks like there was also some amount of non-trivial code changes that were required to get the driver to function. If true, without those changes, it can difficult to re-compile the driver. I have reached out to the original author to see if he might be able to share the changes he made to the driver code. In the mean time, if folks are interested in giving the build process a try, Trickstarter did a great two part write up on how to setup your build environment and compile an example driver.
Although the write up is targeted at ESXi 5.x, you can download the equilvenet packages for ESXi 6.0 which includes the ESXi Open Source Disclosure Package as well as the VMware Toolchain which is required and used to compile the source code. I have provided the direct download links below.
You can also find the latest version of the USB ethernet adapter ax88179 ASIX driver here. I have also attempted to compile just the driver but have already ran into some issues. I have not had time to dig further, so not sure how far I will be able to get. Any tips or tricks others may have for compiling against ESXi 6.0, feel free to share them and I will give them a shot when I get some time!
Michael says
Hi! Nice write-up. Would be a great solution to add another interface to 5th gen + Intel NUC's. Are you going to dig in further to compiling a driver for 6.0? If not, any ideas on where to find someone to hire for the task?
Cheers
Jeff says
Thanks for writing this up, I'm in the process of messing around trying to setup a NUC home lab with (4) NUC 5i5RYKs, First step was altering the bios so I could install vcenter... next step is this.
How is the performance on the USB nic? Do you think you are getting USB 3 speeds?
William Lam says
Hi Jeff,
That's interesting to hear that you had to modify the BIOs? I was actually warned by another user who owns NUC and said NOT to update the firmware/bios as it might make the M.2 not detectable :X I left the system as-is and I was able to setup ESXi, VSAN and VC. The only thing I can think of WRT to BIOS is perhaps Intel-VT wasn't enabled and prevented you to power on 64-bit VMs? That would be the only thing but would be surprised if that's disabled by default ...
I've not done any performance tests, it was merely testing its functionality. If I can get it working for other versions of ESXi, I may run a few tests.
Jeff says
@William, you are right about the m.2 not working after flashing the bios. Just discovered this last night. How were you able to get Vcenter to install? ESXI shows a bunch of random characters for me and it was causing vcenter install to hault.
William Lam says
Have you tried to flash it back to its original factory version? The version that's showing up for me is SYSKLi35.86A.0024.2015.1027.2142
I'm not sure what you mean by "bunch of random characters"? You may want to post a screenshot and the steps that you took. If you're new to deploying vCenter Server 6.0, have a look at this deployment guided which steps you through the process https://www.vmware.com/files/pdf/techpaper/vmware-vcenter-server6-deployment-guide.pdf
William Lam says
Looks like Intel just released BIOS v36 which fixes the issue and I've actually flashed mines this weekend to add custom BIOS splash screen, more details here http://www.virtuallyghetto.com/2016/03/adding-custom-vsan-bios-splash-screen-to-the-intel-nuc.html
If you did happen to update using earlier version, the fix is actually simple and you just need to go into the BIOS itself and re-enable the M.2 slot.
Andreas Peetz says
Hi William,
unfortunately the vm-help.com forums are dead. Your yesterday's post was the first since five months, and the user Trickstarter has not visited them for two years.
Back in the ESXi 5.0 days several people who developed custom ESXi drivers (based on the published ESXi Open Source Code) were quite active there, but these times are over.
I tried to reach out to Trickstarter myself a while ago with no success. I hope you are luckier than me.
Andreas
William Lam says
Thanks for the background Andreas. It kind of felt that way when I left my comment, but I wasn't sure it the forum was still visited. I'm also researching this on my end, so we'll see where I get 🙂
Andreas Peetz says
Hi William,
I cannot believe that it is so easy, but I was able to sucessfully load the two binary modules in ESXi 6.0 Update 1, after I replaced every occurence of "9.2.1" with "9.2.3" in them using a Hex Editor.
I'm not able to check if this really works, because I do not have this USB network adapter available, but chances are good ... Can you please try that?
- Andreas
Andreas Peetz says
I have shared the patched files here: https://drive.google.com/folderview?id=0B192QhF1i6YncFJqUW5LcFVjLVU&usp=sharing
Mitchell Barnett says
I used your files on my 6th gen NUC but don't see the USB NIC showing up when I run esxcfg-nics -l.
Same version of ESXi.
[root@esx0:/vmfs/volumes/56a0e218-2922bb59-1358-b8aeed7c3f93] vmware -vl
VMware ESXi 6.0.0 build-3247720
VMware ESXi 6.0.0 Update 1
[root@esx0:/vmfs/volumes/56a0e218-2922bb59-1358-b8aeed7c3f93] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:00:1f.6 e1000e Up 1000Mbps Full b8:ae:ed:7c:3f:93 1500 Intel Corporation Ethernet Connection I219-V
Hmmm...
William Lam says
Hey Andreas,
Nice hack! I just tried and I was able to successfully load the modules and it looks like the USB Ethernet Adapter was claimed. I've not done any further testing, so not sure how stable it might be. However, I'm not a fan of this option as its hard to tell what unknown consequence might crop up and more importantly the stability of the driver given it was compiled against an older release of ESXi. I'm actually in the process of building this out and targeting just a single module where as the current solution is two different modules. I hope to have some news soon, so stay tuned. Thanks for the always creative solutions 🙂
Peter says
Hey, that sounds great... That would help me a lot!
floriangrehl says
I've tried the same hack with the hex editor and it did not work for me. It was possible to load the module and the usb nic showed up, but it never could get a connection. It's just the same behavior as when no drivers are loaded (physically no link). I've also tried to replace with the ESXi's usb binaries, without success...
rsnoble says
This is hugely helpful! Thank you so much!! Is there a way to run the commands on ESXi host startup instead of having to run them each time the host is restarted?
vmkload_mod /vmfs/volumes/mini-local-datastore-1/usbnetvz026
vmkload_mod /vmfs/volumes/mini-local-datastore-1/ax88179vz026
esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0
William Lam says
Yes, for ESXi 5.x, I believe its still /etc/rc.local but ESXi 6.x it moved to /etc/rc.local.d/local.sh and this will run at the very end once all the ESXi bits have booted up
vlab says
Great info, just what I am looking for!
I was a bit confused where you "included the links below for your convenience:" as those links do not seem to target any files, however the other link: http://www.vm-help.com/forum/viewtopic.php?p=15800#p15800 does. Thanks!
piccolo says
Hi,
the vusb0 was loaded successfully , but :
[root@esx-d:~] vmkload_mod /vmfs/volumes/vmfs-ssd240gb/usblan/usbnetvz025
Module /vmfs/volumes/vmfs-ssd240gb/usblan/usbnetvz025 loaded successfully
[root@esx-d:~] vmkload_mod /vmfs/volumes/vmfs-ssd240gb/usblan/ax88179vz026
Module /vmfs/volumes/vmfs-ssd240gb/usblan/ax88179vz026 loaded successfully
[root@esx-d:~]
[root@esx-d:~] esxcli network nic list
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------------ ------------ ----------- ----- ------ ----------------- ---- ------------------------------------------------
vmnic0 0000:00:19.0 e1000e Up Up 1000 Full b8:ae:ed:7c:af:cf 1500 Intel Corporation Ethernet Connection (3) I218-V
vusb0 Pseudo ax88179_178a Up Down 0 Half ce:d4:29:b4:47:72 1500 Unknown Unknown
vusb1 Pseudo ax88179_178a Up Down 0 Half ce:d4:29:b4:47:72 1500 Unknown Unknown
i have also stopped usbarbitrator with /etc/init.d/usbarbitrator stop befor loading the usb.
link does not go up.
thanks
piccolo says
tested on
VMware ESXi 6.0.0 build-3568940
VMware ESXi 6.0.0 Update 1
Piccolo says
The adapter is not the startec, but has the same chipset
Will says
You probably have already been told of this option but I have seen the Jetway Admpeidla Mini PCIe 2X Gigabit LAN used in conjunction with older NUCs.
piccolo says
thanks, but the intel nuc 5th generation has no mini pcie slot. (seen from the intel website http://www.intel.com/content/www/us/en/nuc/nuc-kit-nuc5i3ryh.html
I have orderd the usb giglan adapter from startech, and will try them.
Steve Furniss says
I'm having the same issue as piccolo... I have the same StarTech aadapter as William Lam, have used the files from google drive and get the same issue...
[root@esxi0:/etc/rc.local.d] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:00:1f.6 e1000e Up 1000Mbps Full b8:ae:ed:7c:40:a3 1500 Intel Corporation Ethernet Connection I219-V
vusb0 Pseudo ax88179_178aDown 0Mbps Half 32:d8:16:c2:7f:a3 1500 Unknown Unknown
I have even tried "esxcli network nic up -n vusb0" and it is not brought up.
S says
Does this USB 3 to gigabit adapter work with the new ESXi 6 Update 2?
I'd like to use 3 of these NUCs in a VSAN 6.2 cluster but I don't think it will work (well) with a single NIC if its doing VSAN,vMotion, FT, VM traffic?
I was thinking of using the onbaord NIC for VM traiffic and then using 2 USB3 to Gigabit adapters for vMotion, FT and VSAN traffic.
Great article!! I never considered NUCs before for a VSAN cluster.
William Lam says
Please have a look at the article in more detail. The USB NIC **only** works with vSphere 5.1 at the moment. I'm hoping I'll be able to get it working for other releases and when I have an update, I'll share it with everyone. For now this is mostly an FYI
Sam Kaufman says
Thank you for your effort and i hope to hear updates of this working in 6.0u1 soon would make for a nice improvement in my home lab.
Jose Gomes says
Hi,
First of all, I would like to say that your postings are very good and have helped me immensely in the past…
I have managed to compile the ax88179_178a driver and got it running on ESXi 5.5 (I am not running 6.0 yet) without needing to create a hacked usbnet module. I would be more that happy to provide you with the compiled module as well as the modified source code for you to try. Also, if you have already setup the 6.x toolchain, recompiling the drivers should be very straight forward. As far as I can tell, the only change required to the code would be to set the appropriate namespace version using the following macro:
#define VMKLNX_MY_NAMESPACE_VERSION "9.2.2.0"
VMK_NAMESPACE_REQUIRED("com.vmware.usb", "9.2.2.0");
VMK_NAMESPACE_REQUIRED("com.vmware.usbnet", "9.2.2.0");
Foe ESXi 6, I believe the versions above should all be 9.2.3.0, but haven't checked yet. All modifications I made to the source code are within if statements , so should be easy to spot.
As I said before, if you would like the source code and pre-compiled binary (for 5.5) let me know what would be the best way to get them across to you.
Disclaimer: I am not a programmer and this is my first foray into C code. I have been using the module on my home lab for 5 days now without problems, but I cannot guarantee the stability of the driver...
William Lam says
Jose,
Thanks for sharing. I'm actually surprised to hear that you've been successful in compiling with only changing the macros. I've been working internally with an Engineer who works on our USB code and we've had to make quite a few changes to get it to properly compile and also correctly loaded (at least for 6.x). Given this is not an "officially" supported project, there's been other more pressing matters the Engineer needs to attend to, so hopefully we'll have an update soon on that front once he gets some free time.
I agree that not requiring hacked usbnet module is ideal. I do have a build toolchain available, so would be interested in the steps you took. Feel free to shoot me an email info [dot] virtuallyghetto [at] gmail [dot] com and I can see if I can reproduce for 6.x.
Jose Gomes says
Let me make it clear: when I said just a change to the macro is required I meant for 6.x. I had to deal with a few other issues, such as missing symbols, the skb_clone function (crashes the server after 5 minutes) and also had to copy in the bitreverse function. The overall number of changes is quite small though…
Anyway, I will be sending you the source and binary soon.
Kevin says
So would loading as you did, for ESXi 5.1, create a persistent setting? If not then is there an easy way to make the changes persistent?
Kevin says
Anyone interested in making these drivers and changes available as a VIB so that the changes will stay persistent in ESXi 5.1?
William Lam says
Have a look at the vibauthor tool for creating custom VIBs http://www.virtuallyghetto.com/?s=vibauthor
Another option if you don't feel like creating the VIB is to simply add the commands to /etc/rc.local script which will run after ESXi starts up. Once I'm able to successfully get the new drivers compiled working (still WIP), I'll be creating VIBs for ease of consumption. Until then, you can easily build your own
Jose Gomes says
I did create a VIB for the module — the installation completes OK, but on reboot the modules take ages to load and eventually, when the boot completes, no network adapters have been found. ;-( A second reboot fixes it because the VIB was only added to the altbootbank. Tried a few times, but I can't find any meaningful errors in the log...
Peter says
Hello,
again, great thread...
Do yo think I could succeed with this device: https://www.startech.com/Networking-IO/usb-network-adapters/USB-3-to-Dual-Port-Gigabit-Ethernet-Adapter-NIC-with-USB-Port~USB32000SPT
Seems to use the same chipset?
Best regards
Peter
William Lam says
Peter,
Nice find, I didn't even notice there was a dual-port option. I'm not sure, but I've just purchased one, so hopefully I'll be able to confirm in a few days on ESXi 5.1 to see if it works. BTW, it's actually cheaper on Amazon $54 vs $77 http://www.amazon.com/dp/B00D8XTOD0/ref=twister_B00BQSRKL2?_encoding=UTF8&psc=1
Kevin says
So i loaded up a NUC with ESXi 5.1 and have loaded the drivers for the USB NIC and assigned it to a vswitch. Unfortunately I am unable to maintain network connectivity over the USB NIC. vSphere client connection starts loading inventory then times out. If I load up a VM and try to use it I can ping a host on the network fine, DNS lookups function, and DHCP will (usually) pull an IP, but other than that network connectivity is broken. Any thoughts?
Kevin says
I did notice that the usbnetvz026.gz file linked actually contains a usbnetgz025 file. Would this matter?
William Lam says
Here's a quick update -
Thanks to Jose Gomes, I actually found out the compiled driver that I had built was a working driver for both ESXi 5.5/6.0, however I thought it wasn't functional as I was not able to get valid link when ethernet cable was plugged in. I was also seeing some strange DMA errors spamming the vmkernel logs. It was after talking to Jose, did I realize the only difference between my env and his was that I was testing the USB NIC on my Mac Mini. Once I tried it on the Intel NUC, it immediately showed link (where as before it only claimed the device). So that's some great news and big thanks to Jose for reaching out.
Now, the not so great news is that I ran a basic iPerf test and I found that I was not able to get the full speed of the GigE interface, in fact getting about ~50% of the NIC w/inconsistent speeds. I'm going to run some additional tests and see if Engineering can spot any tweaks we could make to improve the performance, especially as this is running under USB 3.0, so hopefully we can do better.
You can see the details here:
https://twitter.com/lamw/status/712504070188077056
https://twitter.com/lamw/status/712504559902396417
As of right now, I've got VIBs for 5.5u3 and 6.0u2 which should work for earlier 5.5/6.0 builds and will automatically load the module upon installation. If folks are really interested, I can publish them but I was hoping to get better results before sharing the VIB as they may change. Progress! 😀
Steve Furniss says
This is great news, can you publish the vib as is and any updates later on if you manage to get improvements? I'm running ESXi 6 update 2 and using the previous driver can get it to load but not bring any link up ????
sampka says
Amazing work. I would love to get a copy of the VIB as is also but i understand wanting better performance before sharing it.
piccolo says
Awesome news!
Thanks William.
Would you mind to publish the vib as it is ?
William Lam says
Here's the VIBs for both ESXi 5.5/6.0 (should work for earlier versions of the same major release but these were the ones I've personally tested)
ESXi 5.5u3 - https://www.dropbox.com/s/p9gc7cuwsmn8giw/vghetto-ax88179-esxi55u3.vib?dl=0
ESXi 6.0u2 - https://www.dropbox.com/s/b52fyfma08w7ll8/vghetto-ax88179-esxi60u2.vib?dl=0
To install, you'll need to upload the VIB and run the following command:
esxcli software vib install -v /vghetto-ax88179-esxi60u2.vib -f
The driver will automatically be loaded and if you have the USB NIC connected, it'll automatically be claimed which you can verify by running esxcfg-nics -l
If you do want to remove the driver, please ensure you disconnect the USB device before unloading the module and then removing the VIB else you can cause PSOD.
One interesting observation thanks to Jose was that when you send traffic to the USB NIC, you're able to get consistent gigabit speeds. The variation is when traffic exists the USB NIC, is where we see the bandwidth/transfer speeds being sporadic. If we make further progress, I'll update the thread here.
Sam Kaufman says
When i try to install that on 6.0.0, 3029758 i get
[VibFormatError]
Bad VIB archive header
Steve Furniss says
William, this works great for me on ESXi 6 Update 2. The only observations that I would make for people is that the usb nic can only be added to a vswitch via the command line e.g...
esxcli network vswitch standard uplink add -u vusb0 -v vSwitch1
and that as you mentioned earlier with the manual installation, it is not persistent across a host reboot unless you add it to /etc/rc.local.d/local.sh for esxi 6.
Once again great work.
Sam Kaufman says
I figured out my issue i was trying to install it from my Datastore directly instead of moving it to the root
Sam Kaufman says
Got it to install the VIB but sadly i am still having the same issue with it not wanting to come up. I guess ill have to keep digging.
vusb0 Pseudo ax88179_178aDown 0Mbps Half 00:24:9b:0b:0a:fc 1500 Unknown Unknown
Jose Gomes says
Are you sure it's a USB 3.0 port you are using? The driver will not work properly with USB 2.0 ports, and there should be a message similar to this on vmkernel.log:
2016-03-23T15:53:21.098Z cpu3:33075)WARNING: LinDMA: Linux_DMACheckConstraints:138: Cannot map machine address = 0x11a45abbc, length = 2 for device 0000:00:1d.0; reason =
sampka says
When i get home i will check the vmkernel.log, it is plugged into a usb 3.0 port but if esxi see's it as one i am not sure yet. Thank you for the troubleshooting idea
sampka says
you where right i am getting the error
Linux_DMACheckConstraints:138: Cannot map machine address = 0x11a45abbc, length = 2 for device 0000:00:1d.0; reason = address exceeds dma_mask (0xffffffff))
i will have to figure out why that isnt a usb 3.0 port, Thank you so much for the help
William Lam says
Its possible that your BIOS may not have USB 3.0 enabled, this was definitely true for earlier versions of the Intel NUC where you had to explicitly enable 3.0 in the BIOS. For newer versions of the BIOS, USB 3.0 is now enabled by default and you can't even go back to 2.0.
Another way to check is running "lsusb" command and see if your device is connected to 1.0, 2.0 or 3.0 USB HUB. This should also give you some insights but I would probably check to make sure your BIOS is showing 3.0 or that its plugged into a 3.0 port
Sam Kaufman says
Thanks for the helpful ideas and solutions guys. i am 90 percent sure my issue is what is listed in this article
http://www.virten.net/2015/10/usb-3-0-devices-detected-as-usb-2-in-esxi-6-0-and-5-5/
my problem is my lab is on the smallest form factor dell 7010 which doesn't give you the options in the bios to change the xHCI mode so when i plug a usb 3 device in the correct usb 3 port esxi only detects it as a 2.1 I am still trying to find a work around but no luck so far.
Sam Kaufman says
Just to update everyone on what i found related to my issues with xHCI and smart auto vs enabled. I still cant change the setting in my bios from smart auto but once i updated my esxi to 6.0.0, 3620759 it actually started working.
Jose Gomes says
William,
Thank you very much for sharing this.
As you mentioned, there is a fundamental problem with the transmission speed of this driver. My own tests with iPerf were showing an average of 600 Mbits/sec when transmitting and around 912 Mbits/sec receiving. That was with no other traffic and a TCP window size of 128Kb. Larger window sizes actually reduced the speed.
Today, I have tested vMotion operations between two of my NUCs. Both NUCs have identical configuration (CPU, memory and SSD. The first test was moving the VMs from one host using the USB adapter to another using the built-in Intel adapter:
From Intel 82579LM to ASIX AX88179 (5 minutes average to move each VM)
VM1
2016-03-24T12:02:16.538Z cpu0:37849)VMotionRecv: 578: 1458820629918389 D: Estimated network bandwidth 102.221 MB/s during disk copy.
2016-03-24T12:02:36.687Z cpu3:37849)VMotionRecv: 646: 1458820629918389 D: Estimated network bandwidth 108.920 MB/s during pre-copy
2016-03-24T12:02:37.165Z cpu1:37849)VMotionRecv: 2400: 1458820629918389 D: Estimated network bandwidth 112.597 MB/s during page-in
VM2
2016-03-24T12:19:17.836Z cpu1:41059)VMotionRecv: 578: 1458821687053456 D: Estimated network bandwidth 112.436 MB/s during disk copy.
2016-03-24T12:19:36.243Z cpu3:41059)VMotionRecv: 646: 1458821687053456 D: Estimated network bandwidth 114.086 MB/s during pre-copy
2016-03-24T12:19:36.608Z cpu3:41059)VMotionRecv: 2400: 1458821687053456 D: Estimated network bandwidth 108.912 MB/s during page-in
VM3
2016-03-24T12:23:59.644Z cpu1:42338)VMotionRecv: 578: 1458822097784233 D: Estimated network bandwidth 104.313 MB/s during disk copy.
2016-03-24T12:24:32.852Z cpu0:42338)VMotionRecv: 646: 1458822097784233 D: Estimated network bandwidth 114.018 MB/s during pre-copy
2016-03-24T12:24:33.189Z cpu0:42338)VMotionRecv: 2400: 1458822097784233 D: Estimated network bandwidth 110.686 MB/s during page-in
As you can see, the speed was consistent with a Gigabit network. The movement of VMs took an average of 5 minutes, which I know is consistent with other operations performed with the Intel adapters only. I the went on to test moving VMs between hosts both sporting the USB adapters. I was expecting somewhat worse performance, but the results were very disappointing:
From ASIX AX88179 to ASIX AX88179 (20 minutes average to move each VM)
VM1
2016-03-24T13:05:03.570Z cpu2:36496)VMotionRecv: 578: 1458823815072290 D: Estimated network bandwidth 33.694 MB/s during disk copy.
2016-03-24T13:05:54.762Z cpu2:36496)VMotionRecv: 646: 1458823815072290 D: Estimated network bandwidth 42.864 MB/s during pre-copy
2016-03-24T13:05:56.013Z cpu1:36496)VMotionRecv: 2400: 1458823815072290 D: Estimated network bandwidth 48.039 MB/s during page-in
VM2
2016-03-24T13:22:06.459Z cpu1:39622)VMotionRecv: 578: 1458824864780505 D: Estimated network bandwidth 33.535 MB/s during disk copy.
2016-03-24T13:22:49.614Z cpu1:39622)VMotionRecv: 646: 1458824864780505 D: Estimated network bandwidth 48.647 MB/s during pre-copy
2016-03-24T13:22:50.228Z cpu2:39622)VMotionRecv: 2400: 1458824864780505 D: Estimated network bandwidth 78.886 MB/s during page-in
VM3
2016-03-24T13:36:34.523Z cpu3:43424)VMotionRecv: 578: 1458826221582530 D: Estimated network bandwidth 34.071 MB/s during disk copy.
2016-03-24T13:38:03.958Z cpu2:43424)VMotionRecv: 646: 1458826221582530 D: Estimated network bandwidth 44.373 MB/s during pre-copy
2016-03-24T13:38:04.321Z cpu2:43424)VMotionRecv: 2400: 1458826221582530 D: Estimated network bandwidth 69.317 MB/s during page-in
Now, that was shocking! Just for kicks, I then performed a vMotion from USB to the Intel NIC, and the results were equally unimpressive:
From ASIX AX88179 to Intel 82579LM
2016-03-24T14:11:37.090Z cpu0:59030)VMotionRecv: 578: 1458827976536533 D: Estimated network bandwidth 40.222 MB/s during disk copy.
2016-03-24T14:12:17.575Z cpu1:59030)VMotionRecv: 646: 1458827976536533 D: Estimated network bandwidth 51.853 MB/s during pre-copy
2016-03-24T14:12:18.039Z cpu0:59030)VMotionRecv: 2400: 1458827976536533 D: Estimated network bandwidth 76.352 MB/s during page-in
Anyway, I have been trying different changes to the code, but so far have been unable to improve the transmit speed. Unfortunately, there are certain features of the driver that are simply not supported on ESXi, which I think would make a difference. For example, the adapter supports both scatter-gather (SG) and TCP segmentation offload (TSO), but hose features are not supported by usbnet on ESXi. There is also the skb_clone function, which is not supported at all by the vmklinux — it is explicitly marked as not supported in skbuff.h.
On another note, the dual-port adapter from Startech is also claimed by the driver with no issues. Also, unlike Steve's comment below, I was able to add the vusb NIC to a vSwitch both via the command line and the GUI.
Sorry for the long post...
Niclas Borgström says
I was able to add the adapter to the vSwitch using both the embedded host client as well as the web client but not the C # client.
Niclas Borgström says
This is fantastic, thanks for sharing William. I've installed the vib and I get my two USB NICs to show up, done some basic connectivity checking and it seems to work as expected 🙂
I haven't done any performance testing yet though.
To keep the configuration cross reboots the /etc/rc.local.d/local.sh was updated with these two lines:
esxcli network vswitch standard uplink add -u vusb0 -v vSwitch1
esxcli network vswitch standard uplink add -u vusb1 -v vSwitch1
I'm running:
VMware ESXi 6.0.0 build-3620759
VMware ESXi 6.0.0 Update 2
Niclas Borgström says
I can also confirm that it works on my old Intel NUC dc53427hye. Shows up with link speed 10 half using the USB 2.0 ports and 1000 Full for the USB 3.0 port.
William Lam says
Quick Update - Although you can use init scripts to reload the module, its not ideal as it'll have to be custom for each env depending on how you've setup your uplinks.
Thanks to some folks internally, I've been able to update my VIB so that it'll automatically load upon startup based on whatever configuration you've configured. No need to touch your init scripts for this and download URLs should not change and points to the latest version. Give it a shot and let me know if you have any questions
Also, if you have a Mac that *has* USB 3.0, the USB NIC should also work. I've actually been doing most of my testing recently on my Macbook Air so I don't have to keep power cycling my NUC 🙂
Steve Furniss says
Having a bit of a nightmare trying to update to William's latest version of the vib... I have removed the nic from any vswitch, unplugged the nic from the nuc... restarted then tried to update...
I first tried to update the existing vib using the following command:-
esxcli software vib update -v /tmp/vghetto-ax88179-esxi60u2.vib -f
Here is the error that I got...
[ValueError]
Cannot merge VIBs virtuallyGhetto_bootbank_vghetto-ax88179-esxi60u2_6.0.0-1.0.0, virtuallyGhetto_bootbank_vghetto-ax88179-esxi60u2_6.0.0-1.0.0 with unequal filelist attributes: (set(['etc/init.d/vghetto-install-ax88179', 'usr/lib/vmware/vmkmod/ax88179_178a', 'usr/libexec/jumpstart/plugins/usbnic.json']), set(['etc/init.d/vghetto-install-ax88179', 'usr/lib/vmware/vmkmod/ax88179_178a']))
Please refer to the log file for more details.
I then tried:-
esxcli software vib install -v /tmp/vghetto-ax88179-esxi60u2.vib -f
Installation Result
Message: Host is not changed.
Reboot Required: false
VIBs Installed:
VIBs Removed:
VIBs Skipped: virtuallyGhetto_bootbank_vghetto-ax88179-esxi60u2_6.0.0-1.0.0
So then I though that I would remove the existing vib, reboot and install the new package....
esxcli software vib remove --vibname=vghetto-ax88179-esxi60u2
[LiveInstallationError]
Error in running ['/etc/init.d/vghetto-install-ax88179', 'stop', 'remove']:
Return code: 1
Output: Unable to load module /usr/lib/vmware/vmkmod/ax88179_178a: Busy
It is not safe to continue. Please reboot the host immediately to discard the unfinished update.
Please refer to the log file for more details.
Does anyone have any idea what I am obviously doing wrong?
Regards
Steve
Jose Gomes says
Hi Steve,
Before removing the previous VIB, I made sure vusb0 wan't assigned to a vSwitch and then unloaded the module manually:
vmkload_mod -u ax88179_178a
Once that is done, you should be able to uninstall the old VIB and install the new one.
Note: when I was trying various iterations of the driver I also got myself on a situation where the module would not unload. The only way out was to physically power down the server.
Steve Furniss says
Thanks Jose, that fixed my issue and allowed me to replace the original vib with William's new one.
ds maclean says
Thank you! This worked great! ESXi 6 U2
Tom says
Hi all,
Excited to see this. I think I'm having similar problems to someone before, at first I thought it was a usb3 issue, so I changed the setting in bios to xhci enabled. I can firm the adapter is now connected to a usb 3 hub.
Its still showing as down when I do esxcli network nic list. I try esxcli network nic up -n vusb0 and it results in some messages below in the vmkernel.log. Adding to a vswitch, and connecting to the network results in nothing.
Any ideas?
2016-03-29T04:29:33.691Z cpu0:39075)Uplink: 8472: Opening device vusb0
2016-03-29T04:29:33.691Z cpu1:32811)Uplink: 8329: Detected pseudo-NIC for 'vusb0'
2016-03-29T04:29:34.025Z cpu0:39075)Uplink: 6775: enabled port 0x4 with mac MACADDRESS
2016-03-29T04:29:34.025Z cpu1:32811)NetPort: 1632: disabled port 0x4
2016-03-29T04:29:34.025Z cpu1:32811)Uplink: 6775: enabled port 0x4 with mac MACADDRESS
2016-03-29T04:29:34.025Z cpu0:39075)vusb0: register 'ax88179_178a' at usb-0000:00:14.0-3, ASIX AX88179 USB 3.0 Gigabit Ethernet, MACADDRESS
2016-03-29T04:29:34.025Z cpu0:39075)ax88179_178a 2-3:1.0: interface is claimed by ax88179_178a
2016-03-29T04:29:34.025Z cpu0:39075)usb 2-3: device is not available for passthrough
2016-03-29T04:29:34.025Z cpu0:39075)usbcore: registered new interface driver ax88179_178a
2016-03-29T04:29:34.025Z cpu0:39075)Mod: 4780: Initialization of ax88179_178a succeeded with module ID 4192.
2016-03-29T04:29:34.025Z cpu0:39075)ax88179_178a loaded successfully.
2016-03-29T04:29:34.025Z cpu0:39075)WARNING: Elf: 2791: The module 'ax88179_178a' has a module signature that does not come from a trusted certificate. The module is loaded but treated as unsigned.
2016-03-29T04:29:37.848Z cpu3:39103)FSS: 7014: Unmounting file descriptor 4108adc65bf0
2016-03-29T04:29:37.848Z cpu3:39103)VisorFSRam: 925: inode 7221334930893082624
2016-03-29T04:29:37.848Z cpu3:39103)VisorFSRam: 943: ramdisk stagebootbank
2016-03-29T04:29:37.872Z cpu3:39103)VisorFSRam: 1092: for ramdisk stagebootbank
2016-03-29T04:36:28.408Z cpu0:36011 opID=a0e49d82)World: 14302: VC opID hostd-db96 maps to vmkernel opID a0e49d82
2016-03-29T04:36:28.408Z cpu0:36011 opID=a0e49d82)Uplink: 10364: Setting speed/duplex to (0 AUTO) on vusb0.
2016-03-29T04:36:28.409Z cpu0:36011 opID=a0e49d82)Uplink: 10383: Wait for device vusb0 async call failed.
2016-03-29T04:36:41.176Z cpu0:33888 opID=c2c5e2ab)World: 14302: VC opID hostd-abe1 maps to vmkernel opID c2c5e2ab
2016-03-29T04:36:41.176Z cpu0:33888 opID=c2c5e2ab)Uplink: 10364: Setting speed/duplex to (0 AUTO) on vusb0.
2016-03-29T04:36:41.177Z cpu0:33888 opID=c2c5e2ab)Uplink: 10383: Wait for device vusb0 async call failed.
2016-03-29T04:36:46.342Z cpu3:33903 opID=77ac69c)World: 14302: VC opID hostd-8119 maps to vmkernel opID 77ac69c
2016-03-29T04:36:46.342Z cpu3:33903 opID=77ac69c)Uplink: 10364: Setting speed/duplex to (0 AUTO) on vusb0.
2016-03-29T04:36:46.343Z cpu3:33903 opID=77ac69c)Uplink: 10383: Wait for device vusb0 async call failed.
2016-03-29T04:42:34.861Z cpu2:41246)usb passthrough disabled
2016-03-29T04:43:06.797Z cpu2:33898 opID=6e3eba9d)World: 14302: VC opID hostd-cabb maps to vmkernel opID 6e3eba9d
2016-03-29T04:43:06.797Z cpu2:33898 opID=6e3eba9d)Uplink: 10364: Setting speed/duplex to (0 AUTO) on vusb0.
2016-03-29T04:43:06.799Z cpu2:33898 opID=6e3eba9d)Uplink: 10383: Wait for device vusb0 async call failed.
Tom says
Should of said, esxi 5.5u3.
Tom says
Sorry for the double post. Quickly plugged in a different adapter with the same chipset in (not startech) and Its showing as up but getting the messages below in the vmkernal log.
2016-03-29T04:59:04.888Z cpu3:33831)xhci_hcd 0000:00:14.0: ERROR Transfer event TRB DMA ptr not part of current TD
2016-03-29T04:59:08.306Z cpu3:33831)xhci_hcd 0000:00:14.0: ERROR Transfer event TRB DMA ptr not part of current TD
in
Peter says
Hey, just wanted to tell you that the solution works like a charm with the dual port adapter (ESXI 6 U2)! Thank you for that great posting...
Peter
Sean Hatfield says
The J5create (JUE130) version of this adapter is also working with this driver. I tested on an haswell era nuc, D34010wyk. It needed a bios update to expose the xHCI setting, and I had to update from 6.0GA to 6.0U2 before it would show link up.
William Lam says
Thanks for sharing Sean!
William Lam says
FYI - I've just published a new article on the ESXi 5.5/6.0 driver + source code changes we had to make here http://www.virtuallyghetto.com/2016/03/functional-usb-3-0-ethernet-adapter-nic-driver-for-esxi-5-5-6-0.html
I've also moved the downloads from my dropbox, so please refer to that blog post for all the details. If you have any feedback, please direct it there.
Chip Zoller says
You seem to have closed (or not enabled) comments for that new post, William.
William Lam says
Looks like it disabled on the wrong post :X I meant to close the comments here and direct feedback to that post. I've re-enabled, sorry about that