Quick Tip - Workaround for High CPU usage for ccs-k3s-app in VCF 9.0 Automation 

On a few occasions, I have noticed that after the initial deployment of VMware Cloud Foundation (VCF) 9.0 that also includes VCF Automation (VCFA), the VCFA VM can experience a sustained CPU usage spikes exceeding 30 GHz.

Interestingly, VCFA continues to function fine and I am still able to connect to both VCFA Provider Admin/Organization Portals and perform tasks. In fact, I would not have noticed if it was not for the fans on my MS-A2 spinning like crazy and was able to track it down to ESXi host running the VCFA VM.

With the help from VCFA Engineering, they were able to identify scenario where the ccs-k3s-app Kubernetes pod is miss-behaving and would drive up the CPU utilization.

To determine whether this is affecting your high CPU usage for VCFA, you can run through the following:

Step 1 - SSH to VCFA VM using vmware-system-user and the password you had specified during the deployment

Step 2 - Switch to root user with its environment by running the following command:

sudo -i

Step 3 - List the top pods by CPU and check whether the ccs-k3s-app pod is consuming a significant more amount of CPU by running the following command:

kubectl top pods -n prelude --sort-by=cpu

Step 4 - Inspect the ccs-k3s-app pod logs and see if the following message is being "Waiting for API server to become available" repeated:

kubectl -n prelude logs $(kubectl -n prelude get pods -o name | grep ^pod/ccs-k3s-app | head -n1) --tail=10

If you are observing the same behavior, then the workaround is to delete the running ccs-k3s-app pod and a new instance will respawn and hopefully the issue will go away.

You can use the following 1-liner kubectl command to identify the running pod and then perform the pod delete:

kubectl -n prelude delete $(kubectl -n prelude get pods -o name | grep ^pod/ccs-k3s-app | head -n1)

If we re-run Step 3 again after the pod restart, we should now see the ccs-k3s-app pod is using a more reasonable amount of CPU

If we take a look at our VCFA VM in the vSphere UI, we should see the steady state hovering around 7-8Ghz.

If you are still seeing high CPU usage after applying this workaround and you are not observing the same symptoms, please open an SR to troubleshoot further.