WilliamLam.com

Auditing vGPU Profile Reconfigurations in vSphere

by // Leave a Comment

vCenter Server Events are extremely powerful and contains a ton of useful information, especially for auditing and compliance purposes. As of vSphere 8.0 Update 3, there are over 2.1K+ out of the box events with many more through 2nd and 3rd party integrations.

One of my favorite vCenter Server Event is the VmReconfiguredEvent as it provides complete visibility into every VM reconfiguration change as shared in this blog post back in 2015.

I recently saw an interesting inquiry about being able to audit and track vGPU profile reconfigurations for a VM and of course, vCenter Server Events to the rescue!

[Read more...]

NVIDIA GPU with Dynamic DirectPath IO (Passthrough) to Tanzu Kubernetes Grid (TKG) Cluster using vSphere with Tanzu

by // Leave a Comment

When provisioning a Tanzu Kubernetes Grid Cluster (TKC) using vSphere with Tanzu, you can easily request an NVIDIA GPU resource as part of the deployment, which can either be provided by NVIDIA vGPU or using PCIe passthrough with Dynamic DirectPath IO.

vGPU is great for those with a capable NVIDIA GPU, especially if the GPU will not be utilized 100% and you can share its resources amongst several VMs. However, if you do not have a capable GPU that supports vGPU, you can still provide you TKC workloads with a GPU resource using passthrough.


While playing with the Lenovo P3 Ultra, I unfortunately came to learn that NVIDIA RTX A5500 Laptop was NOT the same as an NVIDIA RTX A5500 🙁

Not ideal, but I guess NVIDIA did not want to add this additional device to their test matrix and hence their ESXi graphics drivers would not detect the GPU as vGPU capable. I knew that I could still use the NVIDIA GPU via passthrough but to my surprise, I just needed to get the NVIDIA drivers installed onto the TKC worker nodes.

That was much easier said than done as all the documentation that I could find on both VMware and NVIDIA website had detailed instructions for vGPU configuration but there was little to no documentation on how to use NVIDIA GPU in passthrough mode with vSphere with Tanzu. I came across a number of different NVIDIA solutions when it comes to k8s, but it was not very clear on which would be interoperable with vSphere with Tanzu and I eventually figured it out with the help pointing me in the right direction.

It was actually super easy, once you knew the exact steps! 😅

[Read more...]

GPU Passthrough with Nested ESXi

by // 9 Comments

Advancements in ESXi Nested Virtualization have given us the ability to run ESXi inside of a VM (Nested ESXi) and has allowed us to do just about anything you would with a physical ESXi host for development, testing and learning purposes. In fact, I have shared many tips and tricks for using Nested ESXi and Nested Virtualization over the years on my blog, which is worth bookmarking in case you are trying to do something and run into an issue which more than likely, I have come across.

Today, there is very little you can not do using Nested ESXi and is typically limited to a physical device that can not be virtualized and/or emulated in software.

I bring this up because I recently had a chat with Frank Denneman on an unrelated topic and he brought up the question about being able to double passthrough of a GPU from a physical ESXi host into a Nested ESXi VM which would then be passthrough'ed again to a VM running on that Nested ESXi system. While this was not the first time that I had heard of such a request, it does not come up often, this has only been the second time I have heard of this request. For context, his use case was for testing purposes and I can certainly see some interesting scenarios where you want to run vSphere in a Nested environment and still access all the vSphere capabilities including leveraging a physical GPU within that environment, whether that is AI/ML or other graphics process requirements.

My response to Frank was this will not work for a few reasons, one of which is that the use of Virtual Hardware-Assisted Virtualization (VHV) is not supported with DirectPath I/O and if the GPU is passthrough to a VM, even if it was running ESXi, it would be in control of the GPU, so how could one passthrough it again?

My curiosity got the better of me and given this was the second time I had ever been asked about this, I figured maybe it was worth exploring but before I go down anymore 🐇🕳️, I wanted to get quick sanity check from one of our graphics engineers on the remote feasibility of this ask.

[Read more...]