I know many in the VMware Community have upgraded to vSphere 8.0 Update 3 to play with the exciting new NVMe Tiering feature, especially for testing various workloads including deploying VMware Cloud Foundation (VCF) for homelab or testing purposes.
A question that has come up a few times now is how do I check the amount of VM memory that has been offloaded to Tier 1 memory (NVMe Tiering) versus using Tier 0 memory (DRAM)?
Today, you can get this information on a specific ESXi host by using the the memstats utility which now has a new report type called vmtier-stats.
Note: In the future, when NVMe Tiering goes out of Tech Preview into generally available (GA), there will be a centralized method for getting this information and going to individual ESXi hosts will not be required.
There are a number of different metrics provided by the vmtier-stats report, but the ones that will be of interests are the following:
| name | Virtual Machine Cartel ID |
|---|---|
| memSize | VM configured memory |
| active | Active guest memory |
| tier0Consumed | Total memory consumed in Tier 0 (DRAM) |
| tier1Consumed | Total memory consumed in Tier 1 (NVMe Tiering) |
One thing you may notice is that memstats uses the VM Cartel ID (e.g. vm.XXXXX) to identify a particular VM, which is not very user friendly. With that said, we can retrieve the VM Cartel ID to VM name by using ESXCLI and then saving that output to a temporarily file, so we can augment the memstats output for ease of consumption.
Step 1 - Run the following command which will output VM Cartel ID to VM name mapping for all current running VMs to a file stored in /tmp/vms
esxcli --formatter csv vm process list | awk -F ',' '{print $2,$5}' > /tmp/vms
Step 2 - Run the following command to retrieve the vmtier-stats and using our VM name mapping file (/tmp/vms), we can replace the VM Cartel ID with the friendly VM name in the output:
memstats -r vmtier-stats -u mb -s name:memSize:active:tier0Consumed:tier1Consumed | sed "$(while read -r name id; do echo "s/vm.$id/$name/g"; done < /tmp/vms)"
From the screenshot above, I have 7 VMs running (VMware Cloud Foundation) on a single ESXi with 96GB of DRAM and NVMe Tiering enabled using 400% ratio. There is ~403GB of configured memory, ~25GB of active memory, ~42GB of consumed DRAM (Tier 0) and ~50GB of memory using NVMe Tiering (Tier 1). What is interesting is that while VCF requires 384GB of memory for deployment, less than half of the physical DRAM is actually being consumed, the overall end user experience is not really impacted based on my experience.
Great Blog. I used your commands with a PowerShell Script to get the results.
Name MemSizeMB ActiveMB Tier0Consumed Tier1Consumed
--------------------------------------------------------------
VCSA8x 16384 3209 14212 7
VAO 20480 2564 13090 21
CP-01 8192 1610 6563 13
vIDM 8192 1445 5870 5
DC-02 2048 206 1986 8
POWERSST 2048 206 1898 38
Results look better when you run the script.
Here is a link to the PS script I used in my GitHub Repository:
https://bit.ly/3MWcxT1
You are Great! Love your articles, so much of stuff, knowledge will get to learn, very very helpful, Thank you so very much for awesomeness
What happens when the NVMe drive in the host fails?
Here is a link to my python script to do the same thing remotely. 🙂
https://github.com/jsconsultancy/tier-mem