With the help from Aquantia, we now have an ESXi driver to enable the built-in 10GbE adaptor for both the Apple 2018 Mac Mini and the iMac Pro. Although this was exciting news for our VMware/Apple customers, I was actually more excited for what this development meant for the larger VMware Community when it comes to 10GbE accessibility.
Many Enterprise customers have already been using Thunderbolt 2/3 to access their 10 Gigabit infrastructure, usually over SFP+ but Ethernet based options also exists such as the Sonnet solution which I had shared last year. This is especially common for VMware customers who virtualize Apple MacOS on vSphere for MacOS/iOS development and the use of Thunderbolt enables ESXi to connect to the underlying storage and networking infrastructure, which traditionally has been either Fibre Channel and/or IP-based storage running over a 10Gig link.
When you start looking at 10GbE accessibility for VMware home labs which could potentially apply to remote office/branch office (ROBO) and Edge/IoT environments, the cost and the complexity of the setup is something that many folks have to consider. There are definitely some creative options out there, most recently Chad Moon shared his solution using a Thunderbolt 3 to PCIe expansion chassis with his Intel NUCs which will run you about $230 per setup or you can be a true hardware hacker like Jack Harvest and use one of the M.2 slots in the Intel Skull Canyon NUC and connect that to PCIe 10GbE SFP+ card with a custom 3D printed chassis to hide everything for just $43.68 🙂
However, I am guessing that most folks including myself, probably prefer a more plug-in-play option that just works out of the box. This is why I was excited for the Aquantia development because there are a number of Thunderbolt 3 (TB3) to 10GbE adapters on the market that can now be enabled with the new ESXi driver. These devices are pretty small and best of all, it enables you to significantly expand your networking capabilities if you have a system with a TB3 port such as an 7th/8th Gen Intel NUC or Skull Canyon NUC or the Hades Canyon NUC which actually has two TB3 ports. For what you get, this is relatively inexpensive, especially if you are setting up a 2-Node vSAN Cluster with Direct Connect!
UPDATE (04/27/20) - Marvell (formally Aquantia) has just released an official Native ESXi Driver for their AQtion based network adaptors which you can find here and for the complete list of supported devices using this driver, please refer to the VMware HCL here. This driver is supported for both ESXi 6.7 and 7.x
I was very fortunate to have been able to get my hands on several of these TB3 to 10GbE devices for testing purposes, a huge thanks goes out to OWC for the Akitio T3-10G and Dirk Hohndel for the OWC TB3 to 10GbE device.
Once you install the Aquantia ESXi Driver which can be downloaded from here, you can plug in the device into a Thunderbolt 3 port and ESXi should automatically detect the network adaptor.
|Confirmed with ESXi
|Thunderbolt 3 10G Ethernet Adapter
|Sonnet Solo 10G
|Sonnet Thunderbolt AVB Adapter
Note: I am still waiting on the arrival of the last two devices, but I should be able to confirm their functionality once they arrive.
UPDATE (04/20/19) - I recently had to setup a new Intel Hades Canyon NUC for testing additional TB3 to 10GbE adaptors as I needed to return the previous unit. I ran into several issues and with the help from some Intel folks, I was able to update the system to the latest Thunderbolt 3 Firmware. Below are the steps for the Hades Canyon, but they should also be applicable for other Intel NUCs with a Thunderbolt 3 port.
Step 1 - Go into the BIOS (F2 during boot up) and make sure BOTH the Thunderbolt Controller and Thunderbolt Boot settings is checked as shown in the screenshot below. These settings are required to use the TB3 ports. Also, if you are using ESXi with the Aquantia Driver, you will need to disable Secure Boot or the system will not boot up ESXi.
Step 2 - Install Windows 10 on the NUC as the firmware utility is only supported on Windows.
Step 3 - Download and install the "legacy" Thunderbolt 3 driver from here. I initially had used the one found on the Hades Canyon download page from Intel, which links to the DCH version pulled down from the Microsoft Store. It turns out, if you are using that version, when trying to run the firmware utility, you will get the following error:
Error: error 0x208 sdk_service_not_found thunderbolt(tm) service doesn't exist
Step 4 - Download and install the Thunderbolt 3 Firmware Utility from here and then flash the device. You may run into an issue if you have something plugged in, so remove any devices and then perform the update operation. Once the system has been updated, you can install ESXi and any of the TB3 to 10GbE adaptors should now automatically be picked up. Also, its a good practice to update to the latest BIOS update, as of writing this, the current BIOS for Hades Canyon is v54.
Lastly, Aquantia also makes a standard PCIe to 10GbE card for $89 which can be added to any white box setup that has a free PCIe slot. Thanks to Stanislav for sharing the good news on getting the card working with his ESXi 6.5 setup.
just tested the driver with PCIe NIC from Aquantia for PC AQC-107 and ESXi 6.5.0 detect this NIC and also detect option of 10Gbit. We will see if I am able to transfer 10gig/s.... once I have other 10gig end 🙂 pic.twitter.com/7QitP6KHuS
— Stanislav Musil (@stmusil) April 14, 2019
In my opinion, this space is only going to get more interesting and with USB 4.0 just around the corner which converges on the Thunderbolt 3 interface, I think it will be exciting to see how connectivity evolves for servers running both within the datacenter as well as on the Edge!