Quick Tip: Reducing High CPU Utilization in VCF Automation (VCFA) on AMD Zen4/Zen5 CPUs

A couple of weeks ago, I received an interesting email from a reader who had observed high CPU utilization after upgrading VCF Automation (VCFA) to the latest VCF 9.1 release. They were running VCF 9.1 on the popular Minisforum MS-A2 platform, which uses AMD Zen4/Zen5 (Ryzen) consumer processors which are not supported on the Broadcom Compatibility Guide (BCG). As I have covered previously (here and here), these processors have been known to exhibit slower entropy generation that can lead to higher than expected CPU utilization without the appropriate optimizations.

With the help of Claude to analyze the issue, the reader arrived at the following conclusion:

Five JVM-based services were dying repeatedly during startup because FIPS_MODE=strict was slowing down TLS handshakes (both database and internal service-to-service HTTPS) enough to occasionally exceed connection timeouts. Every restart meant a full JVM boot: Spring context initialization, Liquibase migrations, Hibernate, index rebuilding, which is expensive. The crash loop itself became the CPU load. Setting FIPS_MODE=disabled on the affected deployments stopped the crash loop entirely, and CPU usage returned to near-idle.

Interestingly, around the same time, I also heard from an internal colleague who had independently observed that disabling FIPS for specific VCFA services significantly reduced CPU utilization. Since FIPS is enabled by default for VCF 9.1 deployments, and there are known performance implications with FIPS and OpenSSL 3.x, these effects can be amplified on AMD Zen4/Zen5 (Ryzen) consumer processors due to their slower entropy generation, resulting in higher than expected CPU utilization.

Fortunately, both individuals had identified workarounds for the issue, but their recommendations did not completely overlap. Before sharing the guidance more broadly, I wanted to spend some time validating the solution to identify the minimal set of changes required to consistently achieve the same outcome.

Step 1 - SSH to your VCFA deployment using the vmware-system-user account, then elevate to the root user using sudo.

Step 2 - Disable FIPS at the Photon OS level by editing /boot/grub/grub.cfg and removing the fips=1 kernel parameter

Step 3 - Run the following kubectl commands to disable FIPS for the specific service deployments:

kubectl set env deployment/approval-service-app -n prelude FIPS_MODE=disabled
kubectl set env deployment/catalog-service-app -n prelude FIPS_MODE=disabled
kubectl set env deployment/ebs-app -n prelude FIPS_MODE=disabled
kubectl set env deployment/ccs-infra-eas-app -n prelude FIPS_MODE=disabled
kubectl set env deployment/provisioning-service-app -n prelude FIPS_MODE=disabled

Step 3 - Reboot the VCFA appliaance for the changes to go into effect.

Step 4 - Once VCFA is back online, you can run the following command to confirm that FIPS is now disabled for the selective services

kubectl get deployment \
  approval-service-app \
  catalog-service-app \
  ebs-app \
  ccs-infra-eas-app \
  provisioning-service-app \
  -n prelude \
  -o jsonpath='{range .items[*]}{"Deployment: "}{.metadata.name}{"\n"}{"FIPS_MODE: "}{.spec.template.spec.containers[*].env[?(@.name=="FIPS_MODE")].value}{"\n\n"}{end}'

For a VCF 9.1 Fleet using the Simple deployment model that includes VCFA, I typically recommend at least three Minisforum MS-A2 systems. For this testing, I only had two MS-A2 systems available. After disabling FIPS, I was able to successfully configure VCFA with vSphere Supervisor, deploy a vSphere Kubernetes Service (VKS) workload cluster, and the environment remained responsive throughout. VCFA CPU utilization also dropped significantly, averaging around 7 to 9 GHz.

Note: Disabling FIPS in VCFA is not supported by Broadcom and is intended only as a workaround for environments using AMD consumer processors. This workaround is not needed for data center-class AMD processors, which typically provide more CPU cores and higher entropy throughput.