Thunderbolt 3 (TB3) and eventually USB 4 is a really fascinating technology and I believe it still has so much untapped potential, especially when looking at Remote/Branch Office (ROBO), Edge and IoT types of deployments. TB3 was initially limited to Apple-based platforms, but in the last couple of years, adoption has been picking up across a number of PC desktop/laptops including the latest generations of Intel NUCs which are quite popular for vSphere/vSAN/NSX Home Labs. My hope with USB 4 is that in the near future, we will start to see servers with this interface show up in the datacenter 🙂
In the mean time, I have been doing some work with TB3 from a home lab standpoint. Some of you may have noticed my recent work on enabling Thunderbolt 3 to 10GbE for ESXi and it should be no surprise that the next logical step was TB3 storage. Using a Thunderbolt interface to connect to external storage, usually Fibre Channel is something many of our customers have been doing for quite some time. In fact, I have a blog post from a few years back which goes over some of the solutions customers have implemented, the majority use case being Virtualizing MacOS on ESXi for iOS/MacOS development. These solutions were usually not cheap and involved a sizable amount of infrastructure (e.g. storage arrays, network switches, etc) but worked very well for large vSphere/MacOS based environments.
Putting aside the TB interface for a second, another exciting development in the last few years is the introduction of NVMe SSD devices and with the M.2 form factor, this makes for a killer combo in terms of performance, power and footprint. These M.2 NVMe SSDs is almost a defacto standard these days for any vSphere/vSAN/NSX Home Lab and can really give your lab a boost in performance! Last year, I wrote about using a USB-C based enclosure which also support M.2 NVMe devices and this could be used for both traditional VMFS datastore as well as for vSAN. In addition to requiring some tricks to get this working, you do sacrifice the ability to provide pass-through of other USB-based devices but more importantly, you do not get the full benefits of the NVMe SSDs, because the maximum bandwidth of USB 3.1 Gen 2 (USB-C) is 10Gbps, this gives you a transfer rate of ~700-800MB/s. With TB3, you get whopping 40Gbps and a transfer rate of ~2750 MB/s enabling you to take full advantage of what NVMe SSDs can offer you.
With the adoption of TB3 continuing to grow, the rise of TB3 peripherals from the eco-system is also growing and innovating. While doing some research online, I came across a number of TB3 enclosures that finally support customizable storage options using M.2 NVMe SSDs. Historically, TB storage enclosures were fixed capacity that could not be modified, they were very bulky and they also came with an insane price tag. Overall, Thunderbolt-based devices are still pretty expensive and the solution shared here is definitely not focused on cost as a core benefit.
Below are 6 different TB3 storage enclosure (3 of which I have tried) supporting single, dual and quad M.2 NVMe SSDs that you can use with ESXi for both VMFS and vSAN, no additional drives or tweaks required are required. These devices can be consumed purely as external storage or creating a "hybrid" vSAN datastore comprised of NVMe SSDs from both a system like an Intel NUC (both half heigh/full height) or Skull/Hades Canyon and NVMe SSDs from these TB3 enclosures. For those looking for maximum storage performance, extending an initial investment or having a modular and mobile VMFS/vSAN Datastore for travel, this may be an interesting solution.
UPDATE (11/17/20) - Are is an additional Single M.2 NVMe enclosure that is pretty inexpensive compared the ones listed earlier: OWC Envoy Express Thunderbolt 3 ($79 USD)
Single M.2 NVMe Enclosure
Both the Trebleet and TEQK are small and portable, the Trebleet is slightly bigger than the TEQK. Other than the TEQK, all other enclosures do NOT include an M.2. Although the TEQK includes an SSD (Phison), unfortunately it is not recognized by ESXi. You will need to install your own M.2 SSD such as a Samsung or Intel. If you are using Crucial M.2 NVMe, like I was, you may run into issues getting ESXi to recognize the device. If you do, check out this blog post for the workaround. Overall, these are great for providing additional storage and can easily be moved from one host to another. For those wanting a super lower power or mobile environment, you can construct a vSAN datastore using the half-height NUCs!
- Trebleet Thunderbolt 3 enclosure ($169-199 USD)
Dual M.2 NVMe Enclosure
Prior to a couple of weeks ago, I was not even aware that multi M.2 NVMe TB3 enclosures was even a thing until I randomly stumbled onto a company called Netstor who produces a number of TB3 storage solutions including the NA611TB3. The first thing that stood out to me beyond the number of supported M.2 devices was just how small this device was (fits in palm of your hand) which includes built-in cooling. Although it is not cheap, especially after factoring the cost of two M.2 devices, this could be an interesting solution for those wanting to add additional storage or have a completely "remote" vSAN that can easily be attached to an ESXi hosts, great for travel and demo purposes.
I was very fortunate to have been able to get my hands on an evaluational unit directly from Netstor. Below are some pictures of the enclosure and on the back of the unit, it includes two TB3 ports and this is where things get interesting. With TB, you can actually "daisy-chain" multiple TB devices and be able to access all of them from a single system! This Netstor unit can connect up to 6 other TB3 devices, so if you really want to go crazy, you can have up to 12 NVMe SSDs and break that up into multiple diskgroups.
- Netstor NA611TB3 ($299 USD)
Quad M.2 NVMe Enclosure
In case the dual M.2 unit was not enough, both NetStor and OWC also makes a quad M.2 unit! One thing that may standout immediately is the price difference but something that is unique to the Netstor unit which looks to be an industry first is the inclusion of a PCIe switch directly within the enclosure. What this means is that each M.2 device is guaranteed to have PCIe 3.0 x2 bandwidth of up to 1600 MB/s, giving you maximum performance of your NVMe devices. You can carve up the devices however you like, whether that is a couple of VMFS volumes + vSAN or creating two vSAN Diskgroups. I did not get my hands on either of the quad M.2 units but it is expected to just work like the others and both of these enclosures include two TB3 ports which can be used to daisy-chain additional TB3 storage or network devices.