WilliamLam.com

Troubleshooting ESXi Shutting down firmware services and UEFI Runtime Services (RTS) error message

by // 1 Comment

Several months back, I was helping a customer debug an issue where I needed to install the GA release of ESXi 6.7! Yikes, it certainly has been a minute since I have installed anything older than 7.x but I figured it should still work fine with recent hardware like an Intel NUC systems, right?

After the ESXi installer started to boot up, it eventually halted with the following message:

Shutting down firmware services...

Using 'simple offset' UEFI RTS mapping policy


I decided to give it one more go by using a more recent release of ESXi 6.7 Update 3 and to my surprise, not only did ESXi installed perfectly fine but I did not run into the error message shown abvove!

I then realized that perhaps this has something to do with the ESXi bootloader, like any piece of software, there are fixes and enhancements with newer releases. I also recall a conversation with one of our Engineers that the ESXi bootloader is also designed to be backwards compatible, so that gave me an idea to try replacing the default ESXi 6.7 GA bootloader files with the ones found in ESXi 6.7 Update 3 and now I was able to install ESXi 6.7 GA release!

However, my success did not last very long as I ran into a slightly different message after the initial reboot:

Shutting down firmware services…

UEFI runtime services support is disabled

[Read more...]

How to setup Harbor registry on Synology

by // 6 Comments

With my recent exploration of GenAI and using a private ChatGPT solution with my own blog posts, I quickly realized in the space of AI/ML, the required software dependencies can take up a significant amount of storage, especially for a kubernetes/container-based deployment.

To give you an example, to deploy the private ChatGPT (h2ogpt) application using kubernetes, just the container image itself is a whopping 40GB+! 😲

Unfourntately, this is not a one off scenario but a common theme when working in the AI/ML space that the size of the packages and drivers are extremely large even when using containers. I figure I should probably setup my own container registry instead of pulling directly from the Internet given the size of these images.

I already have a local Harbor instance running in a VM but with my Synology, I have been using it centralize a number of functions and that would be the ideal place to actually run Harbor. While you can run individual containers on the Synology as I have demonstrated HERE with GitLab, the Harbor installation processes relies on Docker Compose, which Synology does not natively support using the Synology DiskStation Manager (DSM) interface.

With a little bit of tinkering and trial/error, I was able to finally get Harbor running on my Synology and centralize all my storage needs including having my own container registry.

[Read more...]

NVIDIA GPU with Dynamic DirectPath IO (Passthrough) to Tanzu Kubernetes Grid (TKG) Cluster using vSphere with Tanzu

by // Leave a Comment

When provisioning a Tanzu Kubernetes Grid Cluster (TKC) using vSphere with Tanzu, you can easily request an NVIDIA GPU resource as part of the deployment, which can either be provided by NVIDIA vGPU or using PCIe passthrough with Dynamic DirectPath IO.

vGPU is great for those with a capable NVIDIA GPU, especially if the GPU will not be utilized 100% and you can share its resources amongst several VMs. However, if you do not have a capable GPU that supports vGPU, you can still provide you TKC workloads with a GPU resource using passthrough.


While playing with the Lenovo P3 Ultra, I unfortunately came to learn that NVIDIA RTX A5500 Laptop was NOT the same as an NVIDIA RTX A5500 🙁

Not ideal, but I guess NVIDIA did not want to add this additional device to their test matrix and hence their ESXi graphics drivers would not detect the GPU as vGPU capable. I knew that I could still use the NVIDIA GPU via passthrough but to my surprise, I just needed to get the NVIDIA drivers installed onto the TKC worker nodes.

That was much easier said than done as all the documentation that I could find on both VMware and NVIDIA website had detailed instructions for vGPU configuration but there was little to no documentation on how to use NVIDIA GPU in passthrough mode with vSphere with Tanzu. I came across a number of different NVIDIA solutions when it comes to k8s, but it was not very clear on which would be interoperable with vSphere with Tanzu and I eventually figured it out with the help pointing me in the right direction.

It was actually super easy, once you knew the exact steps! 😅

[Read more...]