Useful M.2 NVMe accessories for vSphere (VSAN/VMFS) Home Labs

I recently acquired a new toy for the home lab thanks to Timo Sugliani who shared an article on Twitter a few weeks back for a new USB-based enclosure that supports an NVMe SSD device using the M.2 form factor.

Trying to see if I can get this new toy working 😁 pic.twitter.com/0o4jLng72M

— William Lam (@lamw) September 27, 2018

I was excited to give the accessory a try, especially as the M.2 devices are used regularly for  vSphere home labs running on either the Intel NUCs or Supermicro E200-8D. Most of these platforms only support a single M.2 slot and this is an easy way to add additional high performance storage capacity with a small footprint. The other benefit with an external enclosure is that you now have a portable and reliable storage solution that can easily be moved from system to system, especially for those that have asked about running VMFS on USB-based device.

Being able to run VSAN or VMFS on a USB-based devices (not officially supported by VMware) is not a new concept, I have written about both of these topics here and here before. What is pretty cool about the M2X enclosure is that it supports both USB 3.0 as well as the new USB-C (USB 3.1) connector type, there is a separate cable for each. This probably is sufficient for most vSphere Home Labs and if you wan to really push the boundary on what the NVMe device can do, take a look further below on another accessory that would allow you to take full advantage of the PCIe based device.

Before we can create a VSAN or VMFS based datastore using the USB device, we need to run through some configurations.

Step 1 - Plug in the M2X enclosure using either the USB 3.0 or USB-C connector to your ESXi host. The device will show up as a local USB device with JMicron as the Vendor and "Model USB to PCIE Brid" as the Model.

Step 2 - SSH to the ESXi host and run the following command in the ESXi Shell to disable the USB Arbitrator service (required for USB device to show up):

/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off

Step 3 - We need to enable the following ESXi Advanced Setting so we can mark the USB-based device as an SSD after creating an SATP rule:

esxcli system settings advanced set -o /Disk/AllowUsbClaimedAsSSD -i 1

Step 4 - We will create a new SATP rule to mark our USB device as an SSD. First we need to obtain the device name and the easiest way is run the following command:

vdq -q

The device name should start with t10.JMicron_Generic and we will save that to a variable called DEVICE_NAME by running the following command:

DEVICE_NAME="t10.JMicron_Generic_________0123456789ABCDEF"

Next, run the following command which will create the new SATP and reload the device with our new rule which should enable the device to show up as an SSD:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL --device=${DEVICE_NAME} --option=enable_ssd
esxcli storage core claiming unclaim --type device --device=${DEVICE_NAME}
esxcli storage core claimrule load
esxcli storage core claimrule run

Step 5 - Finally, depending if you plan to use the device for VSAN or VMFS, please follow the datastore specific instructions below:

For VSAN:
To be able to consume the USB device as either a caching or capacity device for VSAN, you also need to enable the following ESXi Advanced Setting:

esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1

If you now run the vdq -q command, we should see our device show up as an eligible disk for VSAN and you can use either the CLI and/or vSphere UI to create a new VSAN Diskgroup using our new device.

For VMFS:
We need to use the ESXi Shell to create a VMFS volume since this is not allowed using the vSphere UI.

First, we need to define a variable that contains the full path of our device which will be under /vmfs/devices/disks and the name of the device (start with t10.JMicron_Generic) by running the following command:

DEVICE_PATH="/vmfs/devices/disks/t10.JMicron_Generic_________0123456789ABCDEF"

Next, run the following commands which will partition the device and then create a VMFS6 datastore on our USB device:

partedUtil mklabel ${DEVICE_PATH} msdos
END_SECTOR=$(eval expr $(partedUtil getptbl ${DEVICE_PATH} | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1)
/sbin/partedUtil "setptbl" "${DEVICE_PATH}" "gpt" "1 2048 ${END_SECTOR} AA31E02A400F11DB9590000C2911D1B8 0"
/sbin/vmkfstools -C vmfs6 -b 1m -S $(hostname -s)-local-usb-ssd-datastore ${DEVICE_PATH}:1

If everything was successful, you should now have a new VMFS6 datastore on our USB based device.

Lastly, if you want to get the full benefit of your M.2 NVMe device and your hardware platform supports a PCIe expansion slot, David Chung (VMware SE) who is well known for his #SDDCinaBox solution has a fantastic recommendation for you.

UPDATE (10/02/18) - David just posted his first blog post about the SDDCinaBox solution, you can follow his new blog series for more details.

With this, you can add 2nd #NVMe drive on #Supermicro E200-8D or any other low profile 1U server. You can run all-NVMe high performance #vSAN #HomeLab in a very small form factor. Picked up few from Amazon and it's running great in my #SDDCinaBox #vmware #HCI #SDDC pic.twitter.com/9pUAC3VK37

— David Chung 🇺🇸 (@dchung615) September 28, 2018

Picture above is an M.2 NVMe to PCIe lower profile expansion adapter (here is the link where he purchased it) which can fit in the popular Supermicro E200-8D or any other platform with an expansion slot. As David mentions, this enables you to have a complete NVMe solution that is high performance and runs in an extremely small form factor, which I am sure also helps with power and cooling.