If you have a supported Trusted Platform Module (TPM) device that has been installed in your ESXi host after the initial installation and you either replace the TPM chip and/or you reset the TPM keys within the system BIOS, you may find several TPM alarms that is raised within your vCenter Server including:
- Host TPM attestation alarm
- TPM Encryption Recovery Key Backup Alarm
- The new host TPM endorsement key doesn't match the one stored in the DB
I recently had to resolve this in my lab after clearing the TPM keys within the system BIOS, this was for some testing I was doing, but I could not figure out how to get vCenter Server to clear the previous endorsement keys associated with the ESXi host.
After a bit of searching, I came across this VMware KB 81446 which outlines a solution to one the scenarios I mentioned above where you would see these TPM alarms, which is replacing the TPM chip, but I came to find out that the workflow is also applicable if you had cleared the TPM keys and new ones were generated prior to re-installing ESXi. The KB was missing a some details, which I have already shared in the feedback and I think there is a more streamline method which I have shared below.
Step 1 - You will need to remove the existing ESXi host from the vCenter Server inventory
Step 2 - SSH to the ESXi host and retrieve the encryption recovery key (96-character) using the following ESXCLI command:
esxcli system settings encryption recovery list
Step 3 - Unlike the VMware KB, which instructs the user to manually type out the 96-character key during the ESXi bootup by pressing SHIFT+O option, a more simplistic method is simply by editing the ESXi boot.cfg and append the encryptionRecoveryKey option and then removing that upon the next boot up, unless you prefer to manually type out the 96-character key, which can lead to a high probability of a typo.
Edit /bootbank/boot.cfg and append encryptionRecoveryKey=[RECOVERY_KEY] from the previous step to the kernelopt line and then save your changes.
Step 4 - Finally, reboot the ESXi host and then remove the entry you made to the /bootbank/boot.cfg and then re-add the ESXi host to the vCenter Server inventory. You may still see the TPM alarm one more time, but go ahead and clear it and on subsequent reboots, the alarm will no longer show up.
You can also confirm everything is working as expected by navigating to the vSphere Cluster and under Monitor->Security, you should now see your ESXi host has successfully been attested by vCenter Server and the previous endorsement key has now been updated
Works only if you did not activate Secure
Boot, which is often the whole point of using a TPM module.
This „removing from vCenter“ is especially nasty in case of a vSAN host, there should be a better way figured out by VMware…
If you place the host into MM along with data migration, why would that be an issue?
Imagine doing the same with a VxRail appliance 😀 let me see what I can find about that and Ill share the content with you so you can share it to the world.
With double or triple digit TB of data per host, that data migration takes a very long time… and then balancing back as well. Just because an error that can’t be cleared otherwise?
Totally fair, just wanted to make sure that it was understood we did have a way to put host into MM w/vSAN (data evacuation), not ideal of course. I'll definitely relay this feedback directly to PM/Engr team and see if we can improve the UX for the future. Thanks for comments
Worked for my Will! Thanks!
Hello William Lam, thanks for this write-up. It's something we run into quite a bit with my work now that more and more people are using secure configuration. Sadly, a very tiny majority ever have the key saved, so most people have to re-install. Given your proximity to this issue, I was wondering if you could provide some further feedback to the process to make it smoother.
1) There should probably be a better notification to make sure that people know to get the key backup before they hit an issue because it can't be easily grabbed once the server has failed.
2) It would be nice if there was an embedded (password protected?) keystore of some kind in vCenter that automatically backs up these recovery keys, and can provide the recovery keys either to the host directly or to an admin with the password.
3) It would be nice if there was some kind of "replace" functionality for a host with a key that isn't recognized after a replacement. Ideally, this would identify that the key is different similar to an SSH key prompt, and then offer the ability to replace the old host with the new host. This would allow it to automatically add it to vCenter without needing to remove from inventory, and then apply any cluster membership, DVS configuration, and host profiles/customization settings like interface aliasing.
I've shared this feedback directly with the Product Manager responsible for TPM management
Hi William, have you heard of any updated procedures to clear the error when replacing a 1.2 TPM with a new 2.0 TPM chip? vCTR error below-
(The new host TPM endorsement key doesn't match the one stored in the DB)
I'd rather not have to remove the vSAN/VxRail host from inventory etc...
thanks
I'm not aware of any specific workflow, best to open an SR and they can pull in Engr if required.
Thanks for this post. Replaced TPM today and followed the existing KBs to confirm new chip. Then I ran into vCenter grumpiness. In my opinion, this is a host issue and vCenter should not require ejecting a host (not a pleasant thing in a 2 node cluster) and shouldn't require DB commands. At most, it should require a vCenter administrator account to acknowledge the new key, just as we acknowledge that the key was backed up.