Virtual NUMA (vNUMA) and CPU Hot-Add support in vSphere 8

While looking for something in the vSphere 8.0 API Reference, I stumbled onto a new VM configuration setting in vSphere 8 called exposeVnumaOnCpuHotadd which looks quite interesting and has the following description:

Capability to expose virtual NUMA when CPU hotadd is enabled. If set to true, ESXi will consider exposing virtual NUMA to the VM when CPU hotadd is enabled. If set to false, ESXi will enforce the VM to have single virtual NUMA node when CPU hotadd is enabled. If unset, the VM continue to follow the behavior in last poweron.

This actually reminded me of question that we got during one of our vSphere 8 Meet the Experts (MTE) sessions at VMware Explore Barcelona and whether there were any new enhancements to vNUMA when CPU hot-add is enabled, which I was not aware of any at the time. The vNUMA and CPU Hot-Add issue is detailed in this blog post by no other than my buddy Frank Denneman and here is a quick summary of the issue:

CPU Hot-Add is not compatible with vNUMA, if hot-add is enabled the virtual NUMA topology is not exposed to the guest OS and this may impact application performance.

Well, this limitation has been lifted as part of vSphere 8 and CPU Hot-Add is now compatible with vNUMA. In addition to being able to use vNUMA with CPU Hot-Add, one additional capability is that a VM can be configured such that ESXi can expose more than a single vNUMA node when a CPU is Hot-Added. This additional setting is only configurable when using the vSphere API, which is the new vSphere 8 API property that is mentioned above.

To take full advantage of the new vSphere 8 vNUMA and CPU Hot-Add capability, here are the following requirements:

• VM must be using latest Virtual Hardware Compatibility 8.0 (vHW 20)
  • Most recent Windows Server and Linux distributions with NUMA kernel support can be used
  • Windows 10/11 do not support CPU Hot-Add
  • Photon 4.0 can be used but requires the real-time linux kernel (tdnf -y install linux-rt)
• VM enabled with CPU Hot-Add (configurable via vSphere UI)
• VM configured with VM Advanced setting numa.allowHotadd = True which will activate vNUMA CPU Hot-Add capability (configurable via vSphere UI)
  • However, this this setting will only enforce a single NUMA node topology
• VM configured with exposeVnumaOnCpuHotadd setting to expose vNUMA for CPU Hot-Add (configurable via vSphere API only)

Since the vSphere API is required to fully enable vNUMA and CPU Hot-Add, here is a quick PowerCLI script that will automatically add the required VM Advanced Setting, Enable CPU Hot-Add and finally enable the new vNUMA CPU Hot-Add vSphere API 8 property to a specific VM:

$vm = Get-VM "Photon-4.0"

$vm | New-AdvancedSetting -Name numa.allowHotadd -Value $true -Confirm:$false

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vNumaSpec = New-Object VMware.Vim.VirtualMachineVirtualNuma
$vNumaSpec.ExposeVnumaOnCpuHotadd = $true
$spec.CpuHotAddEnabled = $true
$spec.VirtualNuma = $vNumaSpec

$vm.ExtensionData.ReconfigVM_Task($spec)

Note: PowerCLI 13.0 will be required to access and configure the new vSphere 8 API property.

After installing the real-time linux kernel on my Photon 4.0 VM and rebooting, I reconfigured the VM from 2 to 12 vCPU and as you can see from the numactl utility, it automatically created two vNUMA nodes rather than the fixed vNUMA node without the new API property.