VCF 9.1.1 - Reducing CPU Utilization on AMD Ryzen Zen4/Zen5 by Masking the RDSEED Instruction

While I am a huge fan of the recent AMD Ryzen Zen4/Zen5-based consumer platforms, including the versatile Minisforum MS-A2, it is well known at this point that these processors can exhibit slower entropy generation, which can lead to significantly higher CPU utilization.

I have published a number of workarounds (HERE, HERE and HERE) to address higher-than-expected CPU utilization across different VCF components, but this has continued to feel like a game of whack-a-mole, with the same underlying entropy issue manifesting itself in different services and workloads.

Right before VMware Explore, I had also been testing VMware Cloud Foundation (VCF) 9.1.1 in my lab while finalizing my presentations and workshops for the conference. After deploying certain VCF components, I noticed that CPU utilization had increased significantly. Long story short, after an extensive debugging session with VCF Engineering, we traced the increased CPU utilization to RDSEED instruction requests originating from application space. The simplest solution turned out to be masking the RDSEED instruction from the guest operating systems!

The latest recommended workarounds for addressing entropy-related issues when running VCF 9.1.x on AMD Ryzen Zen4/Zen5 processors are as follows:

1) Update the ESX entropy source to RDRAND by running the following command:

esxcli system settings kernel set -s entropySources -v 2

2) Mask the RDSEEED instruction for all workloads by running the following command:

echo 'cpuid.7.ebx = "-------------0------------------"' >> /etc/vmware/config

You will need to reboot the system for changes to go into effect.

Coincidentally, over the long U.S. holiday weekend, I saw a note from Daniel Krieger on IN, who shared that AMD had also addressed an RDSEED issue through a firmware update. This fix has been incorporated into the latest BIOS 1.03 firmware update for the Minisforum MS-A2. VCF Engineering had shared that the CVE that was addressed does NOT affect ESX nor does it really help mitigate the observed slowness of RDSEED with Zen4/Zen5 systems, so while there might be some unintentional benefits mentioned by Daniel, the guidance is still to apply the above mentioned.