Using Nested ESXi for VMware Cloud Foundation (VCF) Workload Domain fails with vSAN ESA Auto Disk Claim 

As of VMware Cloud Foundation 5.1.0, you can now use the new vSAN Express Storage Architecture (ESA) when deploying a VCF Workload Domain where as in earlier releases, only the vSAN Original Storage Architecture (OSA) was supported, which is typically what I deploy for testing purposes.

While working on some enhancements to my VCF Automated Lab Deployment Script, I needed to go through a VCF Workload Domain deployment using vSAN ESA but I kept running into the following error within SDDC Manager: Perform vSAN ESA Auto Disk Claim

Based on the error message, I had assumed there was an issue with claiming the SSD on the Nested ESXi VM for vSAN ESA but when I logged into the VCF Workload Domain vCenter Server, I was able to claim the devices without any issues. Looking at the SDDC Manager logs /var/log/vmware/vcf/domainmanager/domainmanager.log also did not yield anything useful:

2024-05-27T12:43:44.545+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] Host host-16 does not contribute to the vSAN ESA storage. 2024-05-27T12:43:44.598+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] Host host-19 does not contribute to the vSAN ESA storage. 2024-05-27T12:43:44.650+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] Host host-20 does not contribute to the vSAN ESA storage. 2024-05-27T12:43:44.701+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] Host host-21 does not contribute to the vSAN ESA storage. 2024-05-27T12:43:44.702+0000 DEBUG [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-14] Destroying 2 open views 2024-05-27T12:43:44.722+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] Failed to verify storage pool disks for cluster wld-w01-cl01 2024-05-27T12:43:44.723+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.v.c.f.p.a.i.ClaimDisksForVsanEsaClusterAction,dm-exec-14] com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Found host(s) in clust er wld-w01-cl01 without contributing to storage pool. 2024-05-27T12:43:44.723+0000 ERROR [vcf_dm,66548000bff276b81889b17ab22d34ed,2f33] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-14] [A337BL] VSPHERE_AUTO_DISK_CLAIM_FAILED Failed to enable auto claim on cluster wld-w01-cl01 com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to enable auto claim on cluster wld-w01-cl01 at com.vmware.vcf.common.fsm.plugins.action.impl.ClaimDisksForVsanEsaClusterAction.postValidate(ClaimDisksForVsanEsaClusterAction.java:119) at com.vmware.vcf.common.fsm.plugins.action.impl.ClaimDisksForVsanEsaClusterAction.postValidate(ClaimDisksForVsanEsaClusterAction.java:23) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$1(FsmActionState.java:23) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159) at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:561) at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124) at jdk.internal.reflect.GeneratedMethodAccessor397.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85) at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71) at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Found host(s) in cluster wld-w01-cl01 without contributing to storage pool. at com.vmware.vcf.common.fsm.plugins.action.impl.ClaimDisksForVsanEsaClusterAction.postValidate(ClaimDisksForVsanEsaClusterAction.java:113) ... 17 common frames omitted

After a bit of digging, I was wondering if this was related to a similiar issue that is seen during the VCF Bringup of the VCF Management Domain using vSAN ESA where a custom vSAN HCL JSON was required?

It turns out we also need to update the default vSAN HCL JSON that is located on SDDC Manager with our custom generated vSAN HCL to get around this error, which I wish was part of the error message so it was a bit more specific to issue observed.

Step 1 - You will need to ensure that your Nested ESXi VM only has an NVMe storage controller configured. If you are using my Nested ESXi Virtual Appliance, you will need to add NVMe controller and then update each disk to use the NVMe controller and finally remove the SCSI controller before proceeding further.

Here is a quick PowerCLI snippet if you are interested in automating the reconfiguration of Nested ESXi VM:

$vm = Get-VM [Nested ESXi VM]
$devices = $vm.ExtensionData.Config.Hardware.Device

$newControllerKey = -102

# Reconfigure 1 - Add NVMe Controller & Update Disk Mapping to new controller
$deviceChanges = @()
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$scsiController = $devices | where {$_.getType().Name -eq "ParaVirtualSCSIController"}
$scsiControllerDisks = $scsiController.device

$nvmeControllerAddSpec = New-Object VMware.Vim.VirtualDeviceConfigSpec
$nvmeControllerAddSpec.Device = New-Object VMware.Vim.VirtualNVMEController
$nvmeControllerAddSpec.Device.Key = $newControllerKey
$nvmeControllerAddSpec.Device.BusNumber = 0
$nvmeControllerAddSpec.Operation = 'add'
$deviceChanges+=$nvmeControllerAddSpec

foreach ($scsiControllerDisk in $scsiControllerDisks) {
    $device = $devices | where {$_.key -eq $scsiControllerDisk}

    $changeControllerSpec = New-Object VMware.Vim.VirtualDeviceConfigSpec
    $changeControllerSpec.Operation = 'edit'
    $changeControllerSpec.Device = $device
    $changeControllerSpec.Device.key = $device.key
    $changeControllerSpec.Device.unitNumber = $device.UnitNumber
    $changeControllerSpec.Device.ControllerKey = $newControllerKey
    $deviceChanges+=$changeControllerSpec
}

$spec.deviceChange = $deviceChanges

$task = $vm.ExtensionData.ReconfigVM_Task($spec)
$task1 = Get-Task -Id ("Task-$($task.value)")
$task1 | Wait-Task | Out-Null

# Reconfigure 2 - Remove PVSCSI Controller
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$scsiControllerRemoveSpec = New-Object VMware.Vim.VirtualDeviceConfigSpec
$scsiControllerRemoveSpec.Operation = 'remove'
$scsiControllerRemoveSpec.Device = $scsiController
$spec.deviceChange = $scsiControllerRemoveSpec

$task = $vm.ExtensionData.ReconfigVM_Task($spec)
$task1 = Get-Task -Id ("Task-$($task.value)")
$task1 | Wait-Task | Out-Null

Step 2 - We now need to generate our custom vSAN ESA JSON based on our running Nested ESXi VM and replace that on SDDC Manager before starting the VCF Workload Domain deployment. We will use my create_custom_vsan_esa_hcl_json.ps1 PowerCLI script which just needs to connect to one of the Nested ESXi VM (Connect-VIServer) to generate the custom vSAN HCL JSON.

Step 3 - SSH to SDDC Manager and update /nfs/vmware/vcf/nfs-mount/vsan-hcl/all.json with the contents of the custom vSAN HCL JSON from Step 2

Note: Make sure the ownership of  the file is vcf_lcm:vcf in case you decide to remove and simply re-create the file.

At this point you can now start a VCF Workload Domain deployment using the SDDC Manager UI or API and you should no longer run into the vSAN ESA Auto Disk Claim error 🙂