VSAN 6.0 includes a large number of new enhancements and capabilities that I am sure many of you are excited to try out in your lab. One of the challenges with running VSAN in a home lab environment (non-Nested ESXi) is trying to find a platform that is both functional and cost effective. Some of the most popular platforms that I have seen customers use for running VSAN in their home labs are the Intel NUC and the Apple Mac Mini. Putting aside the memory constraints in these platforms, the number of internal disk slots for a disk drive is usually limited to two. This would give you just enough to meet the minimal requirement for VSAN by having at least a single SSD and MD.
If you wanted to scale up and add additional drives for either capacity purposes or testing out a new configurations, you are pretty much out of luck, right? Well, not necessary. During the development of VSAN 6.0, I came across a cool little nugget from one of the VSAN Engineers where USB-based disks could be claimed by VSAN which could be quite helpful for testing in a lab environment, especially using the hardware platforms that I mentioned earlier.
For a VSAN home lab, using cheap consumer USB-based disks which you can purchase several TB's for less than a hundred dollars or so and along with USB 3.0 connectivity is a pretty cost effective way to enhance hardware platforms like the Apple Mac Mini and Intel NUCs.
Disclaimer: This is not officially supported by VMware and should not be used in Production or evaluation of VSAN, especially when it comes to performance or expected behavior as this is now how the product works. Please use supported hardware found on the VMware VSAN HCL for official testing or evaluations.
Below are the instructions on how to enable USB-based disks to be claimable by VSAN.
Step 1 - Disable the USB Arbitrator service so that USB devices can been seen by the ESXi host by running the following two commands in the ESXi Shell:
/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off
Step 2 - Enable the following ESXi Advanced Setting (/VSAN/AllowUsbDisks) to allow USB disks to be claimed by VSAN by running the following command in the ESXi Shell:
esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1
Step 3 - Connect your USB-based disks to your ESXi host (this can actually be done prior) and you can verify that they are seen by running the following command in the ESXi Shell:
vdq -q
Step 4 - If you are bootstrapping vCenter Server onto the VSAN Datastore, then you can create a VSAN Cluster by running "esxcli vsan cluster new" and then contribute the storage by adding the SSD device and the respective USB-based disks using the information from the previous step in the ESXi Shell:
esxcli vsan storage add -s t10.ATA_____Corsair_Force_GT________________________12136500000013420576 -d mpx.vmhba32:C0:T0:L0 -d mpx.vmhba33:C0:T0:L0 -d mpx.vmhba34:C0:T0:L0 -d mpx.vmhba40:C0:T0:L0
If we take a look a the VSAN configurations in the vSphere Web Client, we can see that we now have 4 USB-based disks contributing storage to the VSAN Disk Group. In this particular configuration, I was using my Mac Mini which has 4 x USB 3.0 devices that are connected and providing the "MD" disks and one of the internal drives that has an SSD. Ideally, you would probably want to boot ESXi from a USB device and then claim one of the internal drives along with 3 other USB devices for the most optimal configuration.
As a bonus, there is one other nugget that I discovered while testing out the USB-based disks for VSAN 6.0 which is another hidden option to support iSCSI based disks with VSAN. You will need to enable the option called /VSAN/AllowISCSIDisks using the same method as enabling USB-based disk option. This is not something I have personally tested, so YMMV but I suspect it will allow VSAN to claim an iSCSI device that has been connected to an ESXi host and allow it to contribute to a VSAN Disk Group as another way of providing additional capacity to VSAN with platforms that have restricted number of disk slots. Remember, neither of these solutions should be used beyond home labs and they are not officially supported by VMware, so do not bother trying to do anything fancy or running performance tests, you are just going to let your self down and not see the full potential of VSAN 🙂
Incredible article! I know this isn't really about speeds, as you clearly state. But, I still can't resist asking, can you confirm that the hypervisor/VSAN is actually able to leverage USB 3.0 speeds in this scenario?
If yes, well, wow! With UASP adapters and enclosures these days (for reduced protocol overhead and even better throughputs), the opportunities are pretty wonderful, for the unsupported home lab. If merely USB 2.0 speeds for now, well, then .
vSphere 5.5 was the first release to support USB 3.0 IIRC, so this should already be possible. I've not done any specific test, so it's something you'll want to try out 🙂
"Ideally, you would probably want to boot ESXi from a USB device"
I would argue that ideally while I do want to boot ESXi from a USB device, it would be a smallish (~8 GB or so) memory card of some kind, something that would complement ordinary storage, not compete for it. Once you get logging, etc. off the boot device, ESXi doesn't need to write. This used to be known as an 'embedded' install. For all I know, it still is.
Hooray for USB support! 😀
I'm trying to do a demo VSAN entirely on a NUC. Have a host ESX and three nested ESX ... all would be well except I can't seem to tag the USB attached flash as flash so I can use it in a disk group.
It seems that the USB driver just doesn't support Flash (when I click on the option, I get "Cannot change host configuration").
Thoughts?
The USB disk option is only valid if you plan to run a physical VSAN setup. If you're using Nested ESXi, then this won't work and you'll want to look into setting up regular VMDKs such as http://www.virtuallyghetto.com/2013/07/emulating-ssd-virtual-disk-in-vmware.html or http://www.virtuallyghetto.com/2013/08/quick-tip-marking-hdd-as-ssd-or-ssd-as.html
I FOUND IT! This is my first vmware hack on my own, so i am really proud of myself..
esxcli system settings advanced set -o /Disk/AllowUsbClaimedAsSSD -i 1
This is indeed a handy setting. I actually had shared this a few weeks back on https://www.williamlam.com/2018/10/useful-m-2-nvme-accessories-for-vsphere-vsan-vmfs-home-labs.html
I just installed ESXi 6 on Mac Mini 2012 Late model, everything works well except USB 3.0, the usb 3.0 passthrough devices (harddisk or nic) is functional... however speed is limited to usb2.0 spec. Run "lsusb" from ssh, shown the Host found 2.0 root hub only. Do you know any chance to activate USB3.0 support on Mac Mini ESXi? Thanks alot!
#> lsusb
Bus 002 Device 011: ID 0bda:8153 Realtek Semiconductor Corp.
Bus 002 Device 010: ID 0bda:8153 Realtek Semiconductor Corp.
Bus 002 Device 009: ID 05ac:828a Apple, Inc.
Bus 002 Device 008: ID 05ac:820b Apple, Inc. Bluetooth HID Mouse
Bus 002 Device 007: ID 05ac:820a Apple, Inc. Bluetooth HID Keyboard
Bus 002 Device 006: ID 05ac:8242 Apple, Inc. Built-in IR Receiver
Bus 002 Device 005: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)
Bus 002 Device 004: ID 0424:2512 Standard Microsystems Corp. USB 2.0 Hub
Bus 002 Device 003: ID 0781:5583 SanDisk Corp.
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
#> lsusb -t
Bus# 2
`-Dev# 1 Vendor 0x1d6b Product 0x0002
`-Dev# 2 Vendor 0x8087 Product 0x0024
|-Dev# 11 Vendor 0x0bda Product 0x8153
|-Dev# 10 Vendor 0x0bda Product 0x8153
|-Dev# 3 Vendor 0x0781 Product 0x5583
`-Dev# 4 Vendor 0x0424 Product 0x2512
|-Dev# 5 Vendor 0x0a5c Product 0x4500
| |-Dev# 7 Vendor 0x05ac Product 0x820a
| |-Dev# 8 Vendor 0x05ac Product 0x820b
| `-Dev# 9 Vendor 0x05ac Product 0x828a
`-Dev# 6 Vendor 0x05ac Product 0x8242
Bus# 1
`-Dev# 1 Vendor 0x1d6b Product 0x0002
`-Dev# 2 Vendor 0x8087 Product 0x0024
Virtual SAN with USB Disks is awsome for my Intel NUC based Homelab 🙂
But according to USB3, same problem here. I see an USB 3 hub, but all devices (USB 3 SSD / HDD) are always USB 2.
Bus 001 Device 005: ID 174c:1153 ASMedia Technology Inc.
Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 003: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102 Flash Drive / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
Bus 001 Device 002: ID 8087:8001 Intel Corp.
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Any ideas how to get USB3 to work?
I've found the problem. In the UEFI on my system the default mode for xHCI was set to "Smart Auto" which seems to be an issue with ESXi. After setting xHCI to "Enabled", the 2nd 2.0 root hub is gone, and finally my devices are detected as USB 3:
~ # lsusb
Bus 002 Device 002: ID 174c:1153 ASMedia Technology Inc. <---
Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 002: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102 Flash Drive / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Does the USB sticks still have to be in FAT16 format in the VSAN 6.0 to be recognized ? Thanks for your great articles.
Hi, my Samsung Portable SSD T1 is not recognized as a flash device by ESX.
[root@esxi-2:~] esxcli storage core device list -d mpx.vmhba33:C0:T0:L0
mpx.vmhba33:C0:T0:L0
Display Name: Local USB Direct-Access (mpx.vmhba33:C0:T0:L0)
Has Settable Display Name: false
Size: 238475
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/mpx.vmhba33:C0:T0:L0
Vendor: Samsung
Model: Portable SSD T1
Revision: 0
SCSI Level: 2
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: true
Is SSD: false <-----
Is VVOL PE: false
Is Offline: false
This is annoying as hell to setup a proper VSAN. Any ideas how to trick ESX in thinking this actual USB SSD is a SSD device?
Hold on, I hope you do know that this is not an officially supported method and is meant for learning/lab purposes? If you want to setup a "proper" VSAN setup, you should be using hardware found on the official VMware's HCL.
Getting that out of the way, you can "mark" a device as a an SSD/MD which I've blogged about here (http://www.virtuallyghetto.com/2013/08/quick-tip-marking-hdd-as-ssd-or-ssd-as.html) and this can also now be done using the vSphere Web Client now in vSphere 6.0u1 under the Storage Adapters (https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.troubleshooting.doc%2FGUID-99BB81AC-5342-45E5-BF67-8D43647FAD31.html&resultof=%22mark%22%20%22ssd%22%20).
As a new-found-follower-of-your-knowledge, I have tried that.... in my case:
esxcli storage core device list -d mpx.vmhba32:C0:T0:L0
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d mpx.vmhba32:C0:T0:L0 -o enable_ssd
esxcli storage core claiming reclaim -d mpx.vmhba32:C0:T0:L0
>esxcli storage core device list -d mpx.vmhba32:C0:T0:L0
...
Is SSD: false
...
Is there a VIB for this ?
Much appreciated...
PS it is homegrown LAB with NUC.
The USB device dynamically assigns its hbaXX numbers. So how to get the SATP rules to stick ?
You can create SATP rules based on a variety of params types like vendor, location, etc. Depending on your constraints, you can craft a rule that fits your needs. For more details, take a look at the documentation here http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vcli.examples.doc%2Fcli_advanced_storage.8.2.html&resultof=%22satp%22%20
I'm considering different options for building a lab and stumbled upon this post. With this setup, am I reading correctly that it would only require a single host?
Well, VSAN requires a minimum of 3 ESXi hosts for proper functionality, that's how its designed. You *can* run it on a single node, but if your drives fail, you would lose data. Something to keep in mind 🙂
With VSAN 6.1, you can run VSAN on 2 ESXi hosts w/3rd being a Virtual ESXi VM which can run else where (ideally not on the same VSAN Cluster) as it is your quorum node.
ahh, that's what I thought! much appreciated, thanks!
William, any idea if I can use a Time Capsule as extended datastore for esxi? and can I use the internal switch to connect to my new NUC?
A little off-topic in the sense of USB drives weren't in use for issue I encountered. However when configuring a VSAN cluster recently the flash drives were not being seen. I identified existing vmfs partitions on each drive and removed them, however they were still not able to be assigned to the disk group. Restarting the management agents on the host resolved the issue.
Hi,
Any solution for USB HardDisk (not SSD) with 2TB or more?
ESXi 6.7 and lower doesn't recognizes it.