Historically, enabling NVMe Tiering prior to VMware Cloud Foundation (VCF) 9.1 was not very user friendly, as it required users to remember three distinct ESXCLI commands.
# Enable or Disable NVMe Tiering
esxcli system settings kernel set -s MemoryTiering -v TRUE
# Configure the Tiering %
esxcli system settings advanced set -o /Mem/TierNvmePct -i 100
# Configure NVMe device for with NVMe Tiering
esxcli system tierdevice create -d /vmfs/devices/disks/${NVME_TIERING_DEVICE}
By now, most users have probably made a mental note of these commands, with some even incorporating them into their automated ESX Kickstart deployments (example), so it is something you do not have to remember.
VCF 9.1 introduces a new method for enabling and configuring NVMe Tiering that is significantly simpler and no longer requires a system reboot. As a result of these improvements, the ESXCLI commands used in previous releases will no longer properly enable NVMe Tiering in VCF 9.1 and later.
I have seen an increasing number of users, both internally and externally, report that they have enabled NVMe Tiering, only to discover that it was never properly activated due to the use of the legacy ESXCLI commands.
Users will typically share a screenshot from vCenter Server similar to the one above, and there are two dead giveaways that NVMe Tiering was not properly enabled.
- The first is the traditional memory capacity view, which does not reflect the combined memory capacity based on the configured NVMe Tiering ratio.
- The second is the new Memory Tiering widget in the vSphere UI, where the Tier 1 capacity, representing the NVMe Tiering device capacity, shows a value of 0.
To properly enable NVMe Tiering, you will need to use the new ESXCLI memtier namespace. The following one-liner enables NVMe Tiering, selects the NVMe device and configures the desired Tier 1 memory ratio all in a single command!
esxcli memtier enable -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_980_PRO_1TB_________________658AB931B4382500 -r 100
Note: The ESX host must be placed in maintenance mode before you can enable or disable NVMe Tiering, which means if you have running workloads, you will need to first shut them down or have DRS move them to different host.
After applying the correct command and exiting maintenance mode, you will immediately see that your DRAM capacity should now match the configured ratio and Tier 1 capacity is reflecting the ratio amount!
Note: You can also enable NVMe Tiering using the vSphere UI in VCF 9.1 by using vSphere Configuration Profiles (VCP), for more information, you can refer to this blog post.
I’m an oldschool Linux guy and maybe it’s just me but with such changes I’d expect at least a “Warning: Deprecated command, use xyz instead…” when someone tries the old command