Since publishing my NVMe Tiering in vSphere 8.0 Update 3 is a Homelab game changer blog post, the feedback and responses have been absolutely phenomenal!
It will be just a matter of time until we can start using RAID with NVME Tiering !
When it happens, it will be a HUGR game change!
BTW, I'm already using it on my Lab Environment!!
It's F**** awesome! pic.twitter.com/h6Np972RcQ
— Chris ✈️🇧🇷🇵🇹🇺🇸🌍 (@crismsantos) September 4, 2024
In fact, during VMware Explore, I had a number of users share with me in person that they not only updated to vSphere 8.0 Update 3 after learning about the feature but they were extremely happy that they could have their hardware was even more capable with just a software upgrade and workloads varied from general infrastructure VMs to the full VMware Cloud Foundation (VCF) stack.
Right before VMware Explore, I did have a couple of users who reported that after successfully configuring NVMe Tiering and rebooting their ESXi host, they noticed the memory capacity did not change. After sharing the details along with vm-support bundles, Engineering has identified the root cause.
The current implementation of NVMe Tiering requires the vMMR capability, which can be found in both datacenter processors such as Intel Broadwell, Skylake, Cascade Lake, Ice Lake and newer or AMD Milan. For some Intel-based consumer processors, while they may be in the same family, such as Skylake, they actually do NOT contain the vMMR capability like their datacenter peers.
If your CPU processor does not have vMMR, then NVMe Tiering today will not work after enabling. If the following entry is found in the vmkernel log after you have enabled NVMe Tiering, then it means your CPU process is not capable of supporting vMMR:
grep MemHwCounters /var/log/vmkernel
2024-08-11T16:03:18.906Z In(182) vmkernel: cpu2:1048835)MemHwCounters: 501: Matching PCI device not found on this host. Exiting.
2024-08-11T16:03:18.907Z In(182) vmkernel: cpu2:1048835)MemHwCounters: 791: Error status = Not supported
2024-08-11T16:03:18.907Z In(182) vmkernel: cpu2:1048835)MemHwCounters: 1017: Error status = Not supported
The good news is that this limitation will be removed in the future and in fact, this feedback from our community has improve not only the implementation of NVMe Tiering, such that vMMR will not be required but also provide better logging output.
I still have my Intel NUC 10 (Frost Canyon) which is Skylake-based but fortunately, I was able to successfully enabled NVMe Tiering with that kit, so it definitely varies on the consumer Intel CPU on which may or may not have vMMR.
My NUC Coffe lake is working fine too, so you know. UPdated to 8.0.3sb version this morning...
I'm testing this with "Local" NVMe over Fabric devices (32GB Fiber Channel). I've set opp dedicated volumes pr. ESXi host and forced them to be local.
esxcli storage hpp device set --mark-device-local=1 --device=eui.800xxxxx
This allowed me to enable Tiering 😀
I have one question. Is there any way to see how much "tiering" I'm using pr. host ?
I set it up on in a cluster of two Minisforum MS-01 mini PCs. However, live storage vMotion seems not working; the log says:
Migrate: FSR is not yet supported on a system with Software Memory Tiering enabled.
Workaround was to choose "Change both compute resource and storage" and to select the cluster as the compute resource.
Is this issue (FSR not supported with NVMe memory tiering) known?
Yes, there’s a list of known consideration for Tech Preview https://knowledge.broadcom.com/external/article?legacyId=95944 which most will go away when the feature goes GA
William i figured out why the Nested ESXi OVA PANIC exiting with Signal 8 on Memory Tiering enabled Host, that's because it's too new feature, the OVF template need a bump up in the specification of the Guest OS and Compatibility.
Specifically
Guest OS "VMware ESXi 8.0 or later"
line 24
``` ```
and
Compatibility "ESXi 8.0 virtual machine"
line 33
``` vmx-20```
All NestedESXi OVA can benefit from vSphere 8.0u3 Memory Tiering Innovation by updating these 2 lines !
Just out of curiosity i tried the oldest Nested_ESXi6.0u3 which date from 2017 and it run on Memory Tiering enabled Host!
The wanted line are
` `
` vmx-20`
well try again 😀
line 24
OperatingSystemSection ovf:id="104" ovf:version="8" vmw:osType="vmkernel8Guest"
line 33
vmx-20