WilliamLam.com

NVMe Tiering with Nested Virtualization in VCF 9.0

by // 4 Comments

The NVMe Tiering feature was first previewed with the release of vSphere 8.0 Update 3 and the feature is now officially supported with the release of VMware Cloud Foundation (VCF) 9.0! ?

Since the original tech preview, all major limitations and unsupported workload profiles that would prevent an organization from deploying NVMe Tiering in production has been resolved as part of the VCF 9.0 release.

Although Nested Virtualization is NOT officially supported, many users rely on Nested ESXi for testing, development, and learning purposes. With that said, if you have NVMe Tiering enabled on an ESXi 9.0 host and you attempt to power on a Nested ESXi VM or VM that is configured with Virtual Hardware-Assisted Virtualization (VHV), it will fail to power on with the following error message as shown in the screenshot below.


While it was possible to use Nested ESXi with the tech preview of NVMe Tiering, it was just not in scope as part of productizing NVMe Tiering for VCF 9.0.

All hope is not lost, while Nested Virtualization is still NOT officially supported by Broadcom, it is a very useful feature not just for our users but also for internal development purposes and after speaking with the tech lead for NVMe Tiering, this is something that will be addressed in a future update of VCF and this is just a short term limitation for now if you intend to run Nested ESXi workloads.

Enable TRIM/UNMAP from Nested vSAN OSA/ESA to physical vSAN OSA

by // Leave a Comment

If you are running Nested vSAN Original Storage Architecture (OSA) or vSAN Express Storage Architecture (ESA) workloads on top of a physical vSAN OSA setup, you may want to enable the Guest TRIM/UNAMP capability, which would allow your the "inner" (Nested) workloads to pass the TRIM/UNMAP requests down into your physical vSAN OSA deployment for space reclamation.

It was recently observed that storage utilization may continue to increase on a physical vSAN OSA setup even when the Nested vSAN ESA/OSA workloads have already freed up their storage, which is due to the fact that the TRIM/UNAMP commands are not being passed down into the physical vSAN OSA. If you are using vSAN ESA for your physical setup, Guest TRIM/UNMAP is already enabled by default, so this is only applicable if you are using vSAN OSA.

To enable the Guest TRIM/UNMAP capability, you need apply the following ESXi Advanced Setting to your physical vSAN OSA.

[Read more...]

vSAN ESA HCL hardware mock VIB for Nested ESXi

by // 4 Comments

Whether you are configuring vSAN Express Storage Architecture (ESA) directly from vCenter Server or from VMware Cloud Foundation (VCF), the underlying ESXi hardware is automatically validated against the vSAN ESA Hardware Compatibility List (HCL) to ensure that you are using supported vSAN hardware.

In the case of vCenter Server, you can simply ignore the HCL warnings and accept the risks and proceed with the configuration but when using VCF, the operation is blocked to ensure customers have a good experience when selecting vSAN ESA when deploying a VCF Management or Workload  Domain.


With that said, there is workaround where you can create your own custom vSAN ESA HCL JSON based on the hardware that you have and then upload that to either Cloud Builder for setting up a new VCF Management Domain or to SDDC Manager for deploying a new VCF Workload Domain, which I have blogged about HERE and HERE.

The use of Nested ESXi is a very popular way to deploy VCF, especially if you are new to solution and allows you to easily experiment and learn. More recently, I have noticed an uptick in the interests for deploying VCF with vSAN ESA and while you can certainly generate a custom vSAN ESA HCL as mentioned earlier, the process still requires some effort and in some situations the vSAN ESA HCL could get overwritten leading some frustration in debugging.

After helping some folks debug their VCF environments recently, I was thinking about a better experience and leveraging another technique that may not be very well known externally ...

UPDATE (02/03/25) - This solution can also be used for a physical ESXi deployment for use with vSAN ESA and VCF.

[Read more...]