Recovering ESXi 7.x & 8.x host after forgetting or losing root password

The general guidance and quickest way to recover an ESXi host if you have forgotten or lost the root password is to reset using vSphere Host Profiles if it was managed by vCenter Server or simply reinstall ESXi which would allow you to preserve the existing VMFS volumes along with any workloads that may reside on them.

In the past, it was also possible to reset the ESXi root password by booting the system into Linux and then manually updating the /etc/shadow file, which is simliar to how you could reset the password on a Linux-base system and you can find a number of blog articles outlining the details. With the introduction of the ESXi Configuration Store, the previous methodology no longer works for modern ESXi releases starting from ESXi 7.0 Update 1 and later.

Having said that, I know this is still a topic that comes up frequently, especially in the context of administrators joining a brand new company where the ESXi root password has not been properly documented or an admin being asked to support a random set of standalone ESXi hosts that have no owners. Regardless of the scenario, while a reinstallation is the quickest way to recover, it certainly would be nice to be able to maintain the original configuration, especially if there is no documentation to begin with.

While there has been various snippets of information shared online (here, here and here), which includes information from myself, I figured it might be good to figure out the latest process for recovering an ESXi 7.x or 8.x host without requiring a reinstallation.

Pre-requisite: 

• Access to the physical ESXi installed media (USB or HDD/SSD)
• Linux VM (Ubuntu or Photon OS)
• Nested ESXi VM

To demonstrate the recovery workflow outlined below, I have installed ESXi 8.0 Update 3c onto a USB device with some basic configuration (hostname, networking, SSH MOTD) so that I can confirm the system when I recover the host. I then changed the root password to something completely random and discarded the password, so I that I could not login. The ESXi host where I have "forgotten" the password will be referred to as physical ESXi host and Nested ESXi VM that will aide in the recovery will be referred to Nested ESXi host

Step 1 - Deploy a Nested ESXi VM (download from the VMware Flings site) that matches the version of your physical ESXi host that you wish to recover.

Step 2 - Copy the state.tgz from the physical ESXi host that you wish to recover from. Make sure to save a backup copy in case you make a mistake.

• If your ESXi host is installed on USB, unplug the USB device and connect that to a desktop system and copy the file from the BOOTBANK1 volume.
• If your ESXi host is installed on an HDD/SSD, you will need to boot the physical system using Linux LiveCD (Ubuntu or Knoppnix) and mount partition 5 to access the state.tgz

Step 3 - Copy the state.tgz from your physical ESXi host to your Nested ESXi and place it under /tmp/state.tgz and then run the following command to extract the contents of the file:

tar -zxvf state.tgz
rm -f state.tgz

Step 4 - Login to your Nested ESXi host and run the following commands to extract its state.tgz which will be placed in the /tmp/a directory. We then use the crypto-util to decrypt the Nested ESXi host local.tgz.ve to get a local.tgz and then we simply discard the encrypted file along with the Nested ESXi encryption.info file, so that we can then replace that with our physical ESXi host encryption.info file and we re-create a modified version of the state.tgz that will boot in our Nested ESXi VM which we will then use to decrypt our original state.tgz from our physical ESXi host.

mkdir /tmp/a
cd /tmp/a
tar xzf /bootbank/state.tgz
crypto-util envelope extract --aad ESXConfiguration local.tgz.ve local.tgz
rm -f local.tgz.ve encryption.info
cp /tmp/encryption.info /tmp/a/encryption.info
tar -cvf /tmp/state-mod.tgz encryption.info local.tgz

Once the last command has completed successfully, we need to copy the /tmp/state-mod.tgz to your desktop and then shutdown the Nested ESXi VM.

Step 5 - Mount the first VMDK from your Nested ESXi VM to your Linux VM. In my setup, I am simply using Photon OS which happens to be running my DNS infrastructure.

Step 6 - Confirm that VMDK of your Nested ESXi VM is visible on your linux system by running the following command and we should see the two bootbank partitions (5 and 6) as shown in the screenshot below:

fdisk -l

Step 7 - Transfer the state-mod.tgz from Step 4 to your Linux VM and then we will mount both bootbank partition and replace the state.tgz with our modified version.

mount /dev/sdb5 /mnt
cp ~/state-mod.tgz /mnt/state.tgz -f
chmod 755 /mnt/state.tgz
umount /mnt

mount /dev/sdb6 /mnt
cp ~/state-mod.tgz /mnt/state.tgz -f
chmod 755 /mnt/state.tgz
umount /mnt

Note: This step is required because if you just copy the modified state.tgz directly onto the USB device of the physical ESXI host, you will find that it will restore the original state.tgz even if both partitions contains modified version.

Step 8 - Remove (do not delete) the Nested ESXi VMDK from the Linux VM and then power up the Nested ESXi VM.

Once the Nested ESXi VM has successfully booted up, it is now running with the original encryption.info from our physical ESXi host, which will allow us to recover the original state.tgz file.

Step 9 - Copy the original state.tgz from Step 2 to the Nested ESXi VM and place it under /tmp/state.tgz and run the following command which will now allow us to decrypt the physical ESXi host state.tgz as you can see from the screenshot below!

cd /tmp
tar -zxvf state.tgz
rm -f state.tgz
crypto-util envelope extract --aad ESXConfiguration local.tgz.ve local.tgz
rm -f local.tgz.ve

Step 10 - After decrypting the original state.tgz, we should now have local.tgz which we will extract locally within the /tmp directory by running the following command:

tar -zxvf local.tgz

The following three directories .ssh, etc/ and var/ will be placed in /tmp and /tmp/var/lib/vmware/configstore/backup/current-store-1 is the ESXi Configuration Store for the physical ESXi host which we will need to update and replace the original root password hash with the desired password hash, so that we can login to the system.

To manipulate the ESXi Configuration Store directly, we need to use the sqlite3 utility since the file is stored as a sqlite3 database. We can run the following command on the Nested ESXi VM to check for the current root password hash:

/usr/lib/vmware/sqlite/bin/sqlite3 /tmp/var/lib/vmware/configstore/backup/current-store-1  "select * from config where Component='esx' and ConfigGroup = 'authentication' and Name = 'user_accounts' and Identifier = 'root'"

Step 11 - You will need a new SHA512 password hash, where you know the password and then run the following command and substitute the hash.

/usr/lib/vmware/sqlite/bin/sqlite3 /tmp/var/lib/vmware/configstore/backup/current-store-1  "update config set UserValue='{\"name\":\"root\",\"password_hash\":\"\$6\$s6ic82Ik\$ER28x38x.1umtnQ99Hx9z0ZBOHBEuPYneedI1ekK2cwe/jIpjDcBNUHWHw0LwuRYJWhL3L2ORX3I5wFxKmyki1\",\"description\":\"Administrator\"}' where Component='esx' and ConfigGroup = 'authentication' and Name = 'user_accounts' and Identifier = 'root'"

Note: You will need to properly escape any special characters such as the example above where the password hash contains "$" character. To verify that your hash replacement is correct, you can run the query command above to ensure the output matches the desired password hash as shown in screenshot below.

Step 12 - Now that we have updated the ESXi Configuration Store with our desired root password, we just need to re-create the state.tgz that contains our changes by running the following commands:

rm -f local.tgz
tar -cvf /tmp/local.tgz .ssh/ etc/ var/
tar -cvf /tmp/state-recover.tgz encryption.info local.tgz

Copy the /tmp/state-recover.tgz from the Nested ESXi VM to your Linux VM, which will then use to mount the physical ESXi host media to replace the state.tgz with our recovered version.

Step 13 - Mount the physical ESXi media to your Linux VM. Since my physical ESXi host is installed on USB, I simply passthrough the USB device to my Linux VM.

Again, we can confirm that Linux VM can see the physical ESXi installed media by running the fdisk -l command and we should see the two bootbank partitions (5 and 6) as shown in the screenshot below.

Step 14 - We now just need to mount the bootbank partition and replace the original state.tgz with our modified version (state-recover.tgz).

mount /dev/sdb5 /mnt
cp ~/state-recover.tgz /mnt/state.tgz
chmod 755 /mnt/state.tgz
umount /mnt

Note: Since my physical ESXi host was a brand new installation, there was nothing to replace in the second bootbank but if you find a state.tgz, you should also replace that following the same command but replacing the partition number.

Step 15 - Last and final step is to unmount the physical ESXi host media from the Linux VM and then power up your physical ESXi host and you should now be able to login using the updated root password! 🥳