WilliamLam.com

Troubleshooting ESXi Shutting down firmware services and UEFI Runtime Services (RTS) error message

by // 1 Comment

Several months back, I was helping a customer debug an issue where I needed to install the GA release of ESXi 6.7! Yikes, it certainly has been a minute since I have installed anything older than 7.x but I figured it should still work fine with recent hardware like an Intel NUC systems, right?

After the ESXi installer started to boot up, it eventually halted with the following message:

Shutting down firmware services...

Using 'simple offset' UEFI RTS mapping policy


I decided to give it one more go by using a more recent release of ESXi 6.7 Update 3 and to my surprise, not only did ESXi installed perfectly fine but I did not run into the error message shown abvove!

I then realized that perhaps this has something to do with the ESXi bootloader, like any piece of software, there are fixes and enhancements with newer releases. I also recall a conversation with one of our Engineers that the ESXi bootloader is also designed to be backwards compatible, so that gave me an idea to try replacing the default ESXi 6.7 GA bootloader files with the ones found in ESXi 6.7 Update 3 and now I was able to install ESXi 6.7 GA release!

However, my success did not last very long as I ran into a slightly different message after the initial reboot:

Shutting down firmware services…

UEFI runtime services support is disabled

[Read more...]

Support for Virtual Trusted Platform Module (vTPM) on ESXi without vCenter Server?

by // 24 Comments

Starting with vSphere 6.7, users have been able to add a Virtual Trusted Platform Module (vTPM) to a VM, enabling guest operating systems to create and store private keys using a software-based representation of a physical TPM 2.0 chip, that is completely transparent to the underlying OS.

A major benefit of using vTPM is that a physical TPM chip is NOT required in the underlying ESXi host and the vTPM secrets are protected by encrypting the .nvram file, where the secrets are stored.

The encryption keys that are used to encrypt the vTPM is provisioned by a key provider, which can be either be an external Standard Key Provider (SKP) that is KMIP-compliant or using vCenter Server's built-in Native Key Provider (NKP). It is the management of these key providers and their workflows that requires the use of vCenter Server, providing a centralized control plane and a seamless user experience when using the vTPM feature.

Most recently, I saw an influx of inquiries from our field and customers asking about using vTPM with a standalone ESXi host that is NOT managed by vCenter Server, primarily for homelab purposes. While this question has come up in the past, the increased interests might be due to more folks looking to deploy Windows 11, which now has a requirement of a TPM.

While sharing this observation with our lead engineer for VM Encryption, I came to learn that while vCenter Server is highly recommended for a good vTPM user experience, it is technically NOT required for vTPM to function. This sounded very intriguing but surely this solution would NOT be supported right?!

Interestingly, vCenter Server simply uses a set of public vSphere APIs that are available directly on an ESXi host to add or remove encryption keys that is generated from the key provider but the functionality to manage the encryption keys are available on an ESXi host. While this "manual" method is not as seamless as using vCenter Server, you can enable vTPM for a VM using a standalone ESXi host that is not managed by vCenter Server in a completely supported manner!

The lesson here, do not always assume something is NOT supported until you have been told it is NOT supported and always be learning! 😁

[Read more...]

Heads Up - ESXi 8.0 Update 2 requires XSAVE CPU instruction even with allowLegacyCPU=true

by // 23 Comments

As shared back in 2022 in my Homelab Considerations for vSphere 8.x blog post, if you attempt to install ESXi on system where the CPU is no longer being supported by the OEM vendor, who then informs VMware on what to publish on the VMware Hardware Compatibility Guide (VCG), you will see the following error message preventing you from proceeding.


Also mentioned in that article, you can override this default behavior with the following ESXi boot option: allowLegacyCPU=true which would convert this from an error to warning message but allowing you to proceed in an installation or upgrade.

Obviously, this is not officially supported by VMware and its behavior was never guarantee but it certainly was a nice gesture, in my humble opinion from Engineering, as they could have easily not allowed any override based on on our official VCG. For homelab use cases, it might be okay to use a CPU that is 5 to 6 years old, knowing that it should be replaced hopefully in the near future.

I just had a couple of users report issues while upgrading to the latest ESXi 8.0 Update 2 release when using the allowLegacyCPU boot option and saw the following error message

VMB: 716: Unsupported CPU


The ESXi installer immediately halts and prevents the upgrade from proceeding ...

[Read more...]