In both Microsoft Windows 10 and 11, Virtualization Based Security (VBS) is enabled by default and this capability uses Hyper-V under the hood, which is a form of Nested Virtualization. If you are using VMware Workstation and you attempt to power on a Nested ESXi VM, you probably will see one of the following error messages based on the CPU vendor:
- Virtualized Intel VT-x/EPT is not supported on this platform
- Virtualized AMD-V/RVI is not supported on this platform
While VMware Workstation has been enhanced to co-exists with Hyper-V through a new Host VBS Mode introduced in VMware Workstation 17.x:
Workstation Pro uses a set of newly introduced Windows 10 features (Windows Hypervisor Platform) that permits the use of VT/AMD-V features, which enables Workstation Pro and Hyper-V to coexist. And because VBS is built on Hyper-V, Windows hosts with VBS enabled can now power on VM in Workstation Pro successfully
This are few limitations as mentioned in the VMware Workstation documentation.
With that said, if you do need to run Nested ESXi under VMware Workstation, you just need to disable Windows VBS, assuming you have administrative privileges on your system.