WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / ESXi / Useful M.2 NVMe accessories for vSphere (VSAN/VMFS) Home Labs

Useful M.2 NVMe accessories for vSphere (VSAN/VMFS) Home Labs

10.01.2018 by William Lam // 8 Comments

I recently acquired a new toy for the home lab thanks to Timo Sugliani who shared an article on Twitter a few weeks back for a new USB-based enclosure that supports an NVMe SSD device using the M.2 form factor.

Trying to see if I can get this new toy working 😁 pic.twitter.com/0o4jLng72M

— William Lam (@lamw) September 27, 2018

I was excited to give the accessory a try, especially as the M.2 devices are used regularly for  vSphere home labs running on either the Intel NUCs or Supermicro E200-8D. Most of these platforms only support a single M.2 slot and this is an easy way to add additional high performance storage capacity with a small footprint. The other benefit with an external enclosure is that you now have a portable and reliable storage solution that can easily be moved from system to system, especially for those that have asked about running VMFS on USB-based device.


Being able to run VSAN or VMFS on a USB-based devices (not officially supported by VMware) is not a new concept, I have written about both of these topics here and here before. What is pretty cool about the M2X enclosure is that it supports both USB 3.0 as well as the new USB-C (USB 3.1) connector type, there is a separate cable for each. This probably is sufficient for most vSphere Home Labs and if you wan to really push the boundary on what the NVMe device can do, take a look further below on another accessory that would allow you to take full advantage of the PCIe based device.

Before we can create a VSAN or VMFS based datastore using the USB device, we need to run through some configurations.

Step 1 - Plug in the M2X enclosure using either the USB 3.0 or USB-C connector to your ESXi host. The device will show up as a local USB device with JMicron as the Vendor and "Model USB to PCIE Brid" as the Model.

Step 2 - SSH to the ESXi host and run the following command in the ESXi Shell to disable the USB Arbitrator service (required for USB device to show up):

/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off

Step 3 - We need to enable the following ESXi Advanced Setting so we can mark the USB-based device as an SSD after creating an SATP rule:

esxcli system settings advanced set -o /Disk/AllowUsbClaimedAsSSD -i 1

Step 4 - We will create a new SATP rule to mark our USB device as an SSD. First we need to obtain the device name and the easiest way is run the following command:

vdq -q

The device name should start with t10.JMicron_Generic and we will save that to a variable called DEVICE_NAME by running the following command:

DEVICE_NAME="t10.JMicron_Generic_________0123456789ABCDEF"

Next, run the following command which will create the new SATP and reload the device with our new rule which should enable the device to show up as an SSD:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL --device=${DEVICE_NAME} --option=enable_ssd
esxcli storage core claiming unclaim --type device --device=${DEVICE_NAME}
esxcli storage core claimrule load
esxcli storage core claimrule run

Step 5 - Finally, depending if you plan to use the device for VSAN or VMFS, please follow the datastore specific instructions below:

For VSAN:
To be able to consume the USB device as either a caching or capacity device for VSAN, you also need to enable the following ESXi Advanced Setting:

esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1

If you now run the vdq -q command, we should see our device show up as an eligible disk for VSAN and you can use either the CLI and/or vSphere UI to create a new VSAN Diskgroup using our new device.


For VMFS:
We need to use the ESXi Shell to create a VMFS volume since this is not allowed using the vSphere UI.

First, we need to define a variable that contains the full path of our device which will be under /vmfs/devices/disks and the name of the device (start with t10.JMicron_Generic) by running the following command:

DEVICE_PATH="/vmfs/devices/disks/t10.JMicron_Generic_________0123456789ABCDEF"

Next, run the following commands which will partition the device and then create a VMFS6 datastore on our USB device:

partedUtil mklabel ${DEVICE_PATH} msdos
END_SECTOR=$(eval expr $(partedUtil getptbl ${DEVICE_PATH} | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1)
/sbin/partedUtil "setptbl" "${DEVICE_PATH}" "gpt" "1 2048 ${END_SECTOR} AA31E02A400F11DB9590000C2911D1B8 0"
/sbin/vmkfstools -C vmfs6 -b 1m -S $(hostname -s)-local-usb-ssd-datastore ${DEVICE_PATH}:1

If everything was successful, you should now have a new VMFS6 datastore on our USB based device.


Lastly, if you want to get the full benefit of your M.2 NVMe device and your hardware platform supports a PCIe expansion slot, David Chung (VMware SE) who is well known for his #SDDCinaBox solution has a fantastic recommendation for you.

UPDATE (10/02/18) - David just posted his first blog post about the SDDCinaBox solution, you can follow his new blog series for more details.

With this, you can add 2nd #NVMe drive on #Supermicro E200-8D or any other low profile 1U server. You can run all-NVMe high performance #vSAN #HomeLab in a very small form factor. Picked up few from Amazon and it's running great in my #SDDCinaBox #vmware #HCI #SDDC pic.twitter.com/9pUAC3VK37

— David Chung 🇺🇸 (@dchung615) September 28, 2018

Picture above is an M.2 NVMe to PCIe lower profile expansion adapter (here is the link where he purchased it) which can fit in the popular Supermicro E200-8D or any other platform with an expansion slot. As David mentions, this enables you to have a complete NVMe solution that is high performance and runs in an extremely small form factor, which I am sure also helps with power and cooling.

More from my site

  • Thunderbolt 3 enclosures with (Single, Dual & Quad) M.2 NVMe SSDs for ESXi
  • ESXi Thunderbolt Driver to Fibre Channel Storage from ATTO
  • Home Labs made easier with VSAN 6.0 + USB Disks
  • Removable M.2 NVMe SSD PCIe enclosure by Icy Dock
  • Quick Tip - Crucial NVMe SSD not recognized by ESXi 6.7 & 7.0

Categories // ESXi, Home Lab, Not Supported, VSAN Tags // M.2, NVMe, usb, USB-c, Virtual SAN, VSAN

Comments

  1. Fereidoun. Fatahiany says

    10/01/2018 at 10:11 am

    If your motherboard's bios sees the drive, you should not have a problem running even in raid 0. ( Fast, really fast.).

    Reply
  2. N/A says

    10/01/2018 at 11:49 pm

    If you really wanna squeeze things in, get an Amfeltec Squid, which comes in PCIe HHHL form factor with a proper PCIe switch chip, allowing driverless 4xM.2 (no BIOS PCIe lane bifurication support necessary). The PCie v2 version will even let you cheat with a 4x lane PCIe slot uplink (PCIe v3 version starts at 8x).

    Reply
  3. Johnny says

    10/16/2018 at 4:49 pm

    I do understand this post quite perfectly, but 1 point still missing or not clear! After you disable the usb arbitrator... Does it mean that you cannot used any other usb ? Or not be able to add any other usb device to a vm ?? As i never see that you start back the usb arbitrator .. after the m2 setup... Thanks

    Reply
  4. Christopher Thorjussen says

    02/25/2019 at 4:10 pm

    Have you heard anything about the latest nvme drivee in 67u1 hides the HP EX920 nvme driver? Just put two in my home lab server on an asus hyper m.2 x16 card (with bifurcation x4x4x4x4) and had to downgrade the driver to the one from 67GA for the disk itself to show up, not just the controller. Found info about it on reddit (https://www.reddit.com/r/vmware/comments/a80r3y/issue_with_hp_nvme_drive_in_esxi/)

    Reply
    • Christopher Thorjussen says

      02/25/2019 at 4:12 pm

      "hides the HP EX920 Nvme drives" it was supposed to say (as in disks)

      Reply
  5. theo says

    06/13/2020 at 12:16 pm

    Your blog is amazing have you come across any pcie cards that allow you to add two or more devices per card the suggested works perfectly thanks

    Reply
  6. runtime says

    08/05/2020 at 12:16 pm

    "esxcli system settings advanced set -o /Disk/AllowUsbClaimedAsSSD -i 1"

    I can't thank you enough for this one line! Your blog _IS_ the VMWare documentation as far as I'm concerned.

    Reply
  7. Darin W. says

    01/16/2021 at 2:01 pm

    Hi, you may want to add a warning that newer versions may not be the same as what you got, it appears that they have changed the chipset and introduced other issues that many are reporting are killing the SSD's attached. I would have the disk show up, allow me to create a VMFS partition, but the second I tried copying any files (large or small) it would effectively disappear (even under Linux).

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Self-Contained & Automated VMware Cloud Foundation (VCF) deployment using new VLC Holodeck Toolkit 03/29/2023
  • ESXi configstorecli enhancement in vSphere 8.0 Update 1 03/28/2023
  • ESXi on Intel NUC 13 Pro (Arena Canyon) 03/27/2023
  • Quick Tip - Enabling ESXi Coredumps to be stored on USB 03/26/2023
  • How to disable the Efficiency Cores (E-cores) on an Intel NUC? 03/24/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...