WilliamLam.com

Copying files from a USB (FAT32 or NTFS) device to ESXi

by // 14 Comments

It is not uncommon, especially in troubleshooting scenarios where you might find yourself needing to transfer files to or from an ESXi host using a USB device as it may not be reachable on the network. Another common case for directly attaching a USB device to an ESXi host is to transfer a large amount of Virtual Machines that were exported from another system and rather than streaming the content from your desktop, you may want to connect it directly to ESXi host. In fact, I had this very use case when I was a customer after we had acquired a company and needed to transfer their assets to our infrastructure. The IT admins just copied everything onto a USB device and then shipped us the drive for processing.

Historically, it was understood that ESXi could only access a USB device (requires disabling the USB arbitrator service) if it contains a FAT16 partition which are then automatically mounted under the /vmfs/volumes/ path. The biggest issue with FAT16 is that the size of the partition has to be <=2GB which severely limits its use for larger files. Another alternative that came up in recent years years is that you could run VMFS on a USB device, but that obviously would require you to format the USB device with VMFS and it would only be readable between ESXi hosts. If you were looking for something more generic like FAT32 which supports a larger partition size, it was assumed this was not possible, at least I was under that impression.

It was only recently as part of a project I had been working on where I was re-visiting this topic that I had discovered that other partition types such as FAT32 and even NTFS from a USB device could actually be accessed by ESXi 6.x. The assumption that I and probably others had made was that just because the partitions were not visible or mounted by ESXi, it does not mean the underlying USB device would also not be accessible. To access a FAT32 partition from a USB device in ESXi, you can use the mcopy utility from the ESXi Shell and for accessing an NTFS partition from a USB device in ESXi, you can use ntfscat utility. It actually took me some trial/error to get the correct syntax, but you can see how to use the utilities below.

[Read more...]

Functional USB 3.0 Ethernet Adapter (NIC) driver for ESXi 5.5 & 6.0

by // 81 Comments

Earlier this month I wrote an article demonstrating a functional USB ethernet adapter for ESXi 5.1. This was made possible by using a custom built driver for ESXi that was created over three years ago by a user named Trickstarter. After having re-discovered the thread several years later, I had tried reaching out to the user but concluded that he/she has probably moved on given the lack of forum activity in the recent years. Over the last few weeks I have been investigating to see if it was possible to compile a new version of the driver that would function with newer versions of ESXi such as our 5.5 and 6.0 release.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (01/22/17) - For details on using a USB-C / Thunderbolt 3 Ethernet Adapter, please see this post here.

UPDATE (11/17/16) - New driver has been updated for ESXi 6.5, please find the details here.

After reaching out to a few folks internally, I was introduced to Songtao Zheng, a VMware Engineer who works on some of our USB code base. Songtao was kind enough to provide some of assistance in his spare time to help with this non-sanction effort that I was embarking on. Today, I am please to announce that we now have a functional USB ethernet adapter driver based on the ASIX AX88179 that works for both ESXi 5.5 and 6.0. This effort could not have been possible without Songtao and I just want to say thank you very much for all of your help and contributions. I think it is safe to say that the overall VMware community also thanks you for your efforts. This new capability will definitely enable new use cases for vSphere home labs that were never possible before when using platforms such as the Intel NUC or Apple Mac Mini for example. Thank you Songtao! I would also like to extend an additional thank you to Jose Gomes, one of my readers, who has also been extremely helpful with his feedback as well as assistance on testing the new drivers.

Now, Before jumping into the goods, I do want to mention there are a few caveats to be aware of and that I think it is important to understand them before making any purchasing decisions.

  • First and foremost, this is NOT officially supported by VMware, use at your own risk.
  • Secondly, we have observed there is a substantial difference in transfer speeds between Transmit (Egress) and Receive (Ingress) traffic which may or may not be acceptable depending on your workload. On Receive, the USB network adapter is performing close to a native gigabit interface. However, on Transmit, the bandwidth mysteriously drops by ~50% which includes very inconsistent transfer speeds. We are not exactly sure why this is the case, but given ESXi does not officially support USB based ethernet adapters, it is possible that the underlying infrastructure was never optimized for such devices. YMMV
  • Lastly, for the USB ethernet adapter to properly function, you will need a system that supports USB 3.0 which kind of makes sense for this type of a solution to be beneficial in the home lab. If you have a system with USB 2.0, the device will probably not work at least from testing that we have done.

Note: For those interested in the required source code changes to build the AX88179 driver, I have published all of the details on my Github repo here.

Disclaimer: In case you some how missed it, this is not officially supported by VMware. Use at your own risk.

Without further ado, here are the USB 3.0 gigabit ethernet adapters that are supported with the two drivers:

Note: There may be other USB ethernet adapters that uses the same chipset which could also leverage this driver but these are the only two that have been verified.

usbnic
Here are the ESXi driver VIB downloads:

Note: Although the drivers were compiled against a specific version of ESXi, they should also work on the same major version of ESXi, but I have not done that level of testing and YMMV.

Verify USB 3.0 Support

As mentioned earlier, you will need a system that is USB 3.0 capable to be able to use the USB ethernet adapter. If you are unsure, you can plug in a USB 3.0 device and run the following command to check:

lsusb

usb3nic-0
What you will be looking for is an entry stating "Linux Foundation 3.0 root hub" which shows that ESXi was able to detect a USB 3.0 port on your system. Secondly, look for the USB device you just plugged in and ensure the "Bus" ID matches that of the USB 3.0 bus. This will tell you if your device is being claimed as a USB 3.0 device. If not, you may need to update your BIOS as some systems may have USB 2.0 enabled by default like earlier versions of Intel NUC as desribed here. You may also be running pre-ESXi 5.5 which did not support USB 3.0 as mentioned here, so you may need to upgrade your ESXi host to at least 5.5 or greater.

Install Driver

You can either install the VIB directly onto your ESXi host or by creating a custom ESXi ISO that includes the driver using a popular tool like ESXi Customizer by Andreas Peetz.

To install the VIB, upload the VIB to your ESXi host and then run the following ESXCLI command specifying the full path to the VIB:

esxcli software vib install -v /vghetto-ax88179-esxi60u2.vib -f

usb3nic-1
Lastly, you will need to disable the USB native driver to be able to use this driver. To do so, run the following command:

esxcli system module set -m=vmkusb -e=FALSE

You will need to reboot for the change to go into effect.

To verify that the USB network adapter has been successfully claimed, run either of the following commands to list your physical NICs:

esxcli network nic list
esxcfg-nics -l

usb3nic-2
To add the USB uplink, you will need to either use the vSphere Web Client or ESXCLI to add the uplink to either a Virtual or Distributed Virtual Switch.

usb3nic-3
To do so using ESXCLI, run the following command and specify the name of your vSwitch:

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Uninstall Driver

To uninstall the VIB, first make sure to completely unplug the USB network adapter from the ESXi first. Next, run the following ESXCLI command which will automatically unload the driver and remove the VIB from your ESXi host:

esxcli software vib remove -n vghetto-ax88179-esxi60u2

Note: If you try to remove the VIB while the USB network adapter is still plugged in, you may hang the system or cause a PSOD. Simply reboot the system if you accidentally get into this situation.

Troubleshooting

If you are not receiving link on the USB ethernet adapter, it is most likely that your system does not support USB 3.0. If you find the a similar message like the one below in /var/log/vmkernel.log then you are probably running USB 1.0 or 2.0.

2016-03-21T23:30:49.195Z cpu6:33307)WARNING: LinDMA: Linux_DMACheckConstraints:138: Cannot map machine address = 0x10f5b6b44, length = 2 for device 0000:00:1d.7; reason = address exceeds dma_mask (0xffffffff))

Persisting USB NIC Configurations after reboot

ESXi does not natively support USB NIC and upon a reboot, the USB NICs are not picked up until much later in the boot process which prevents them from being associated with VSS/VDS and their respective portgroups. To ensure things are connected properly after a reboot, you will need to add something like the following in /etc/rc.local.d/local.sh which re-links the USB NIC along with the individual portgroups as shown in the example below.

esxcfg-vswitch -L vusb0 vSwitch0
esxcfg-vswitch -M vusb0 -p "Management Network" vSwitch0
esxcfg-vswitch -M vusb0 -p "VM Network" vSwitch0

You will also need to run /sbin/auto-backup.sh to ensure the configuration changes are saved and then you can issue a reboot to verify that everything is working as expected.

Summary

For platforms that have limited built-in networking capabilities such as the Intel NUC and Apple Mac Mini, customers now have the ability to add additional network interfaces to these systems. This will now open up a whole new class of use cases for vSphere based home labs that were never possible before, especially with solutions such as VSAN and NSX. I look forward to seeing what our customers can now do with these new networking capabilities.

Additional Info

Here are some additional screenshots testing the dual USB 3.0 ethernet adapter as well as a basic iPerf benchmark for the single USB ethernet adapter. I was not really impressed with the speeds for the dual ethernet adapter which I had shared some more info here. Unless you are limited on number of USB 3.0 ports, I would probably recommend just sticking with the single port ethernet adapter.

usb3nic-5
usb3nic-6

iPerf benchmark for Ingress traffic (single port USB ethernet adapter):
usb3nic-7
iPerf benchmark for Egress traffic (single port USB ethernet adapter):
usb3nic-8

Working USB Ethernet Adapter (NIC) for ESXi

by //

As part of upgrading my personal vSphere home lab from an Apple Mac Mini to an Intel NUC (more on this in a future blog), I have been researching to see if there are other alternatives for adding an additional network adapter. The Intel NUC only includes a single built-in ethernet adapter which is similar to the Mac Mini. However, the NUC also lacks additional IO connectors like a Thunderbolt port which the Mac Mini includes and can support a Thunderbolt to Ethernet adapter. I think this is probably the only downside to the Intel NUC platform which has been similar feedback that I have seen from other vSphere home labbers who currently use or would like to use the NUC. Perhaps, with the next update of the NUC platform code named "Skull Canyon", the rumored Thunderbolt 3 / USB-c connector may make things easier as some of the existing vendors who produce Thunderbolt to ethernet adapter also use common drivers like the Broadcom tg3 which have historically worked with ESXi.

One option that has been suggested by many folks over the years was to see if a USB based ethernet adapter could be used to provide additional networking to ESXi? Historically, the answer had been no because there were no known device drivers that would work with ESXi. I had even looked into this a few years ago and although I ran into some folks who seemed to have made it work, I was never able to find the right USB ethernet adapter to personally confirm myself. It was only until last week, I decided to start fresh again and after a bit of Googling I came across an old VMTN thread here where VMTN user AK_____28 mentioned he had success with the StarTech USB 3.0 to Gigabit Ethernet NIC Network Adapter and using a custom compiled driver that was posted over here by another user named Trickstarter.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (03/29/16) - Please have a look at this updated article here which includes ESXi 5.5 and 6.0 driver.

Disclaimer: In case it is not clear and apparent, you should never install any unknown 3rd party software on your ESXi host as it can potentially lead to instability issues or worse open yourself to a security hole. The following solution is not officially supported by VMware, please use at your own risk.

I decided to bite the bullet and give this solution a try and purchased the USB ethernet adapter from Amazon here.

usb-ethernet-adapter
There are two modules that needs to be downloaded, extracted and loaded onto ESXi. I have included the links below for your convenience:

As the VMTN thread mentioned, you can load using either the vmkload_mod or ESXCLI. Here are the two commands that I used in the following order:

vmkload_mod /vmfs/volumes/mini-local-datastore-1/usbnetvz026
vmkload_mod /vmfs/volumes/mini-local-datastore-1/ax88179vz026

When I tried to initially load either of the modules, I would always get the following error:

vmkwarning.log:2016-02-28T21:54:54.531Z cpu6:374787)WARNING: Elf: 2041: Load of <usbnetvz026> failed : missing required namespace <com.vmware.usb#9.2.1.0>

As you can imagine, I was pretty bummed to see this since I was afraid that something like this would happen. I was not sure if the device I had purchased no longer worked or if was the drivers? I saw that these modules were initially compiled for ESXi 5.1 (at the time, that was the latest version) and the only difference was that I was using a much newer version of ESXi, specifically 6.0 Update 1. I decided to install the latest version of ESXi 5.1 Update 3 and tried the process again and to my surprise, the modules loaded without errors. I suspect that this was a hard dependency on the namespace version which was version 9.2.1.0 and the latest version is now 9.2.3.0.

usb-network-adapter-esxi-1
After successfully loading the two modules, I ran the following command:

esxcfg-nics -l

to verify that ESXi did in fact did claim the USB ethernet device and as you can see from the screenshot below, it did indeed!

usb-network-adapter-esxi-2
Next up, I needed to verify basic connectivity and added the new uplink to my existing vSwitch. You must use the following ESXCLI command (esxcfg-vswitch command does not work apparently for non vmnicX devices)

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Once added, I hopped over to the vSphere C# Client to see if the device is now showing up under the Network Adapters tab, which it is.

usb-network-adapter-esxi-4
Finally, the last test was to make the vsb0 (this is how ESXi names the device) device the active connection while moving my existing vmnic0 to stand-by. Networking connectivity continued to function and I was even able to transfer an ISO image over the USB ethernet adapter without any issues.

usb-network-adapter-esxi-5
So it looks like it is possible to get a USB based ethernet adapter to function with ESXi, at least with the specific model listed above (PCI ID 0b95:1790). The challenge now is to see if there is a way to build an updated version of the drivers targeted at the latest ESXi 6.0 release. From what I have been able to follow on the forum here, it looks like there was also some amount of non-trivial code changes that were required to get the driver to function. If true, without those changes, it can difficult to re-compile the driver. I have reached out to the original author to see if he might be able to share the changes he made to the driver code. In the mean time, if folks are interested in giving the build process a try, Trickstarter did a great two part write up on how to setup your build environment and compile an example driver.

Although the write up is targeted at ESXi 5.x, you can download the equilvenet packages for ESXi 6.0 which includes the ESXi Open Source Disclosure Package as well as the VMware Toolchain which is required and used to compile the source code. I have provided the direct download links below.

You can also find the latest version of the USB ethernet adapter ax88179 ASIX driver here. I have also attempted to compile just the driver but have already ran into some issues. I have not had time to dig further, so not sure how far I will be able to get. Any tips or tricks others may have for compiling against ESXi 6.0, feel free to share them and I will give them a shot when I get some time!