Sharing a single NVMe device with NVMe Tiering? 

I am a huge fan of the new NVMe Tiering capability within vSphere 8.0 Update 3 and it has been fantastic to hear more users taking advantage of this new feature to see what it can do for their homelab/development setup but also for their various production workloads.

As of right now, enabling NVMe Tiering requires a dedicated NVMe device, which for a production system is probably acceptable as you will probably want to ensure there are no other workloads competing for IO on the NVMe device. However, for a development environment or homelab, this can be challenge due to number of available NVMe devices that can be used.

Thank you to fellow reader Andrea T, for sharing this awesome tidbit with the community and how you CAN actually share a single NVMe device with NVMe Tiering! 😍

For those with small form factor systems with a limited number of NVMe devices like ASUS NUC for example, you can run ESXi off of a USB device, while still being able to use local VMFS datastore and NVMe Tiering and you still technically have a spare slot or two that can be used for vSAN OSA or ESA!

UPDATE (12/16/24) - See this blog post HERE if you would like to run NVMe Tiering, ESXi-OSData and VMFS on a single NVMe device

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Step 1 - Ensure that you have an empty NVMe device, you can not use an existing device with any existing partitions. You can use the vdq -q command to identify and retrieve SSD device name.

Step 2 - Download the calculateSharedNVMeTeiringAndVMFSPartitions.sh shell script to your ESXi host and update the three required variables:

• SSD_DEVICE - Name of the NVMe device from Step 1
• NVME_TIERING_SIZE_IN_GB - Specify the amount storage (GB) that you wish to use for NVMe Tiering
• VMFS_DATASTORE_NAME - Name of the VMFS datastore to create on NVMe device

Ensure the script has executable permission (chmod +x /tmp/calculateSharedNVMeTeiringAndVMFSPartitions.sh) and then run as shown in the screenshot below:

Note: The script will only generate the commands required BUT you will need to run them manually, so make a note of them and should help without having to manually calculate the storage start and end sectors by hand 🙂

Here is an example of running the generated commands for my setup where I have 1TB (913.15GB) NVMe and I am allocating 256GB for NVMe Tiering and then the remainder space will be allocated for VMFS datastore.

Using the ESXi Host Client, we can see the two partitions that we have now created:

Step 3 - Enable the NVMe Tiering feature, if you have not already by running the following ESXCLI command:

esxcli system settings kernel set -s MemoryTiering -v TRUE

Step 4 - Configure the desired NVMe Tiering percentage (25-400) based off of your physical DRAM configuration by running the following command:

esxcli system settings advanced set -o /Mem/TierNvmePct -i 400

Step 5 - Finally, reboot for the NVMe Tiering settings to go into effect and once your ESXi host reboots, you will now have a single NVMe device supporting both NVMe Tiering and for local VMFS datastore for you to use for workloads!