With the announcement of VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) 9.1, I know many of you are eager to start planning for 9.1 and get it into your lab environment for hands-on experience as the first step toward pre-production validation.
Unless you are among the few with adequate resources, including servers listed on the Broadcom Compatibility Guide (BCG), most lab environments are resource constrained in one way or another.
To help expedite your 9.1 deployment for non-production usage, I have documented the various ESX configuration workarounds that can used for lab deployments, especially for functional testing and/or for educational purpose.
I will continue to update this page with new information, so be sure to bookmark for your single reference!
Physical ESX 9.1 Host
vSAN Compression Algorithm
If you are using vSAN, the default compression algorithm has changed from LZ4 to Zstd in 9.1, which provides a much higher compression ratio. In CPU-constrained environments, you can default to the original LZ4 algorithm. (Reboot not required).
esxcli system settings advanced set -o /VSAN/Vsan2ZdomCompZstd -i 0
AMD Ryzen and CPU Entropy
If you are running on an AMD Ryzen Zen 4/Zen 5 (Consumer) processor, you will probably observe higher than normal CPU utilization due to slow entropy generation, that can severely affect your environment. (Reboot required)
esxcli system settings kernel set -s entropySources -v 1
AMD Ryzen and NVMe Tiering
If you are running on an AMD Ryzen Zen 4/Zen 5 (Consumer) processor, VMs may fail to power on when NVMe Tiering is enabled. (Reboot required)
echo 'monitor_control.disable_apichv ="TRUE"' >> /etc/vmware/config
AMD Ryzen and NSX Edge and Virtual Network Appliance (VNA)
If you are running on an AMD Ryzen Zen 4/Zen 5 (Consumer) processor, deployment of NSX Edges or the new Virtual Network Appliance (VNA) will fail to configure as consumer AMD processors do not support DPDK. Prior to NSX Edge or VNA deployment, run the following command and ensure the string starts with "AMD EPYC" as this is validated from within the NSX Edge and VNAs (Reboot not required)
echo 'cpuid.brandstring = "AMD EPYC Ryzen 9 7945HX"' >> /etc/vmware/config
Intel x710 and LRO/TSO
If you have Intel x710 NICs, especially for Minisforum MS-A2 owners, you probably will have performance degradation due to hardware offload without even realizing it (reboot required)
esxcli system settings advanced set -o /Net/TcpipDefLROEnabled -i 0
esxcli system settings advanced set -o /Net/UseHwTSO -i 0
Fake Server Hardware Vendor for heterogeneous vSphere Lifecycle Manager (vLCM) Clusters
If you are using ESX hosts that have mixed hardware vendors, the VCF Installer will throw an error. See this blog post for more details on applying workaround.
NVMe Tiering for Nested ESX VM
If you have NVMe Tiering enabled on your physical ESX host, the default behavior of the physical ESX host is to NOT tier the memory used by your Nested ESX VM. To allow the Nested ESX VM to get the benefit of NVMe Tiering, you will need to add the following VM Advanced Setting to the Nested ESX VM prior to powering on the VM.
sched.mem.enableNestedTiering = true
Nested vSAN on top of physical vSAN
If you plan to run Nested vSAN on top of a physical vSAN deployment, the following configuration is required. (Reboot not required)
esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1
TRIM/UNMAP for Nested vSAN on top of physical vSAN
If you plan to run Nested vSAN (OSA or ESA) on top of a physical vSAN deployment, ensure the TRIM/UNMAP commands are being passed back up to physical vSAN environment. (Reboot not required, only for Nested vSAN Hosts)
esxcli system settings advanced set -o /VSAN/GuestUnmap -i 1
Nested ESX 9.1 Host
vSAN Optimizations
If you plan to use vSAN in a Nested environment, here are several configurations that have been recommended to ensure optimal setup.
esxcli system settings advanced set -o /LSOM/VSANDeviceMonitoring -i 0
esxcli system settings advanced set -o /LSOM/lsomSlowDeviceUnmount -i 0
esxcli system settings advanced set -o /VSAN/SwapThickProvisionDisabled -i 1
Note: If you use the Nested ESX Virtual Appliance Fling, the vSAN optimization settings are not required as these are already applied by default.
vSAN Compression Algorithm
If you are using vSAN, the default compression algorithm has changed from LZ4 to Zstd in 9.1, which provides a much higher compression ratio. In CPU-constrained environments, you can default to the original LZ4 algorithm by running the following command (reboot not required).
esxcli system settings advanced set -o /VSAN/Vsan2ZdomCompZstd -i 0
AMD Ryzen and CPU Entropy
If you are running on an AMD Ryzen Zen 4/Zen 5 (Consumer) processor, you will probably observe higher than normal CPU utilization due to slow entropy generation, that can severely affect your environment including Nested ESX. Run the following command (reboot required)
esxcli system settings kernel set -s entropySources -v 1
vSAN High pNIC Error Rate Detected Alarms
In resource constrained environments, you may see an increased number of high pNIC alarms that are generated in vCenter Server which is sourced from the vSAN Health Check service. You can either disable the health check or you can increase the buffer as a workaround, the error may return if the buffer limit has been reached again. (Reboot not required)
esxcli network nic ring current set -r 4096 -t 4096 -n vmnic0
Dear William,
First, thank you for your excellent blog post. Your work is invaluable to the community.
I am writing to ask for clarification about one of the workarounds you documented.
My Setup:
Physical ESXi host running VCF 9.1
NVMe Memory Tiering is enabled and active on the physical host (configured using the new esxcli memtier commands)
I am trying to run a nested ESXi 9.1 VM on this physical host
What I tried:
Following your blog post, I added the following advanced parameter to my nested ESX VM:
sched.mem.enableNestedTiering = true
The problem:
When I try to power on the nested ESXi VM, I still receive this error:
"VMware ESXi does not support nested virtualization on this host."
***********************************************************************
[root@Z620:~] esxcli memtier device list
Path Model Type Size Configured Recommended
---------------------------------------------------------------------------------------- ---------------------------------------- ---- ---------- ---------- -----------
/vmfs/devices/disks/t10.NVMe____CT1000P310SSD8__________________________068C704E0175A000 CT1000P310SSD8 NVMe 953869 MiB true
[root@Z620:~] esxcli memtier status get
Status: ENABLED
Devices: /dev/disks/t10.NVMe____CT1000P310SSD8__________________________068C704E0175A000'
Size: 383.85GiBs (200 percent of DRAM)
Encryption: Disabled
License Info: Licensed for 4.00 times of DRAM
[root@Z620:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
1 esx-01b [datastore2] esxi-10p/esxi-10p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
2 esx-02b [datastore2] esxi-11p/esxi-11p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
3 esx-03b [datastore2] esxi-12p/esxi-12p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
5 Router-02 [datastore1] Router-02/Router-02.vmx debian6_64Guest vmx-09
7 esx [datastore2] esx/esx.vmx vmkernel9Guest vmx-22
[root@Z620:~] vim-cmd vmsvc/get.config 7 | grep -A 2 "sched.mem.enableNestedTiering" | grep -E "key|value" | sed 's/^[ \t]*//'
key = "sched.mem.enableNestedTiering",
value = "TRUE"
[root@Z620:~] vim-cmd vmsvc/power.on 7
Powering on VM:
Power on failed: (vim.fault.GenericVmConfigFault) {
faultCause = (vmodl.MethodFault) null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
key = "msg.vhv.ulm",
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
key = "1",
value = "VMware ESXi"
}
],
message = "VMware ESXi does not support nested virtualization on this host.
"
},
(vmodl.LocalizableMessage) {
key = "msg.moduletable.powerOnFailed",
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
key = "1",
value = "VMMon"
}
],
message = "Module 'VMMon' power on failed.
"
},
(vmodl.LocalizableMessage) {
key = "msg.vmx.poweron.failed",
arg = ,
message = "Failed to start the virtual machine."
}
],
reason = "VMware ESXi does not support nested virtualization on this host.
"
msg = "VMware ESXi does not support nested virtualization on this host.
"
}
[root@Z620:~]
**********************************************
Odd ... this works for me as expected. Can you at least power on the VM w/o the VM Advanced Setting, that should also work (it just won't tier the Nested ESX VM memory)
Unfortunately, I have the same error.
FYI, I upgraded my physical ESX host from 8.0 U3 to 9.1. Could this be related to the update? Should I perform a fresh installation?
******************
[root@Z620:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
1 esx-01b [datastore2] esxi-10p/esxi-10p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
2 esx-02b [datastore2] esxi-11p/esxi-11p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
3 esx-03b [datastore2] esxi-12p/esxi-12p.vmx vmkernel65Guest vmx-21 Nested ESXi 8.0U3C Appliance (Build 24414501) http://www.williamlam.com
5 Router-02 [datastore1] Router-02/Router-02.vmx debian6_64Guest vmx-09
7 esx [datastore2] esx/esx.vmx vmkernel9Guest vmx-22
[root@Z620:~] vim-cmd vmsvc/get.config 7 | grep -A 2 "sched.mem.enableNestedTiering" | grep -E "key|value" | sed 's/^[ \t]*//'
[root@Z620:~] vim-cmd vmsvc/power.on 7
Powering on VM:
Power on failed: (vim.fault.GenericVmConfigFault) {
faultCause = (vmodl.MethodFault) null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
key = "msg.vhv.ulm",
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
key = "1",
value = "VMware ESXi"
}
],
message = "VMware ESXi does not support nested virtualization on this host.
"
},
(vmodl.LocalizableMessage) {
key = "msg.moduletable.powerOnFailed",
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
key = "1",
value = "VMMon"
}
],
message = "Module 'VMMon' power on failed.
"
},
(vmodl.LocalizableMessage) {
key = "msg.vmx.poweron.failed",
arg = ,
message = "Failed to start the virtual machine."
}
],
reason = "VMware ESXi does not support nested virtualization on this host.
"
msg = "VMware ESXi does not support nested virtualization on this host.
"
}
Can you provide the vmware.log file for the Nested ESX VM that is failing the power on?
Hello William, thank you for your response.
I reinstalled the host from scratch and enabled memory tiering, but I am still experiencing the same issue.
To obtain clean and reliable logs, I created a new Nested ESX 9.1 VM.
Please find the logs at this share : https://drive.google.com/file/d/10h_tUP-ZsbBlg7KS6jx4DdUu-wpExo_P/view?usp=drive_link.
Let me know if you need any further information.
Hey William, I've got a two node MS-A2 setup. When looking at NVMe tiering using a Samsung_SSD_990_EVO_Plus_1TB, in ESX cli I can certainly configure it and it looks OK. Also on the cluster > summary it shows enabled for 2 hosts.
However if I use desired state, I get incompatability issues, for example "vSAN health test 'NVMe device is VMware certified' reported an issue for cluster 'home-m01-cl01'. Detailed error message: The NVMe device is not listed in the VMware Compatibility Guide (VCG)."
Also when I look at the host, it shows the DRAM of 128GB, where as on 9.0.2 it said 256GB, feels like somethings changed in how it reports it. The new 9.1 config on non-HA needs a hell of a lot of RAM eh? I'm worried my poor dear boxes are gonna burn out. The cluster reports 256GB. Not 512GB which I expected.
Any thoughts?
The fix was simple after reading the new docs. There is an additional command vs 9.0.2. I swear I never did that in the prior version, its now added into my kickstart scrpt.
esxcli memtier enable -d /vmfs/devices/disks/${nvme_tiering_device}
I can then confirm with the below, and the ESX host repors 256Gb again.
esxcli memtier status get
Hope this helps someone.
Here were my notes on the NVME Memory Tiering config with VCF 9.1:
All of the commands have changed to new esxcli memtier commands.
esxcli system maintenanceMode set --enable true
esxcli system tierdevice create -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_9100_PRO_1TB________________0E2A4251A5382500
esxcli memtier enable -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_9100_PRO_1TB________________0E2A4251A5382500 -r 400 -c f
##The -c f turns off crypto/encryption if you wish, I'd love to test what the overhead really is
/sbin/auto-backup.sh
reboot -f
##The /sbin/auto-backup.sh makes sure to persist these changes to /bootbank/state.tgz. I noticed that over and over and over my nvme tiering configs wouldn't take after rebooting, and it had more to do with reboot vs reboot -f. The reboot -f command rebooted before the changes were persisted to the state file. Woops. Adding this little /sbin/auto-backup.sh command fixed that problem immediately.
(The VCF 9.1 lab kickstart scripts will need to be adjusted, as others may have mentioned)
I didn't include this since the change of syntax for NVMe Tiering isn't really a workaround, its more of an optimization instead of having to go to three commands ... you can achieve it with just single command
You do NOT need to run create, you can do it all in one with: esxcli memtier enable -d /vmfs/devices/disks/t10.NVMe____Samsung_SSD_9100_PRO_1TB________________0E2A4251A5382500 -r 400
One of the changes in 9.1 is to remove the need for a system reboot, which is why MM is required prior to enabling or disabling NVMe Tiering
Is anyone else having issues with MS-A2 handling a single VCF-A node? With memory tiering on and dedicating an entire node I can’t get the VM to remain stable. SSH and HTTPS throw timeout errors. The node maxes its CPU, reporting over 100% constantly 🙁
I wrote about my experience here.
https://amayacitta.co.uk/vcf-9-1-two-node-build/#vcf-automation-platform
Have you already applied https://williamlam.com/2026/04/quick-tip-high-cpu-utilization-on-esx-due-to-slow-entropy-from-amd-zen-4-cpus.html which is where I and others were observing higher CPU usage that would affect workloads, including VCFA
I know some folks still see spikes here/there, alternatively you could force CPU limit (15ghz) and the system does behave but I've not had to do that but for some other AMD-kits, it seems this has helped ... so unclear if different processors may still behave oddly
I'll give it a go, didnt realise the Ryxen 9 9955HX is actually based on Zen 5 architecture. I saw those tweaks but ignored them as I didnt think they were relevant. I'll report back 😉 Cheers.
Hello Willam
esxcli network nic ring preset get -n vmnic0
Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Not supported: VSI node (240:VSI_NODE_net_pNics_firmware_ringParams)
I assume you're doing this for Nested ESX VM, is vmnic0 already in use by NSX by chance?