Nested ESXi installation using HTTPS boot over VirtualEFI in vSphere 8

In vSphere 7.0 Update 2, an enhancement was made to the Virtual Machine's UEFI firmware called VirtualEFI that would enable ESXi to run in a VM (Nested ESXi) and perform an HTTP Boot given the ESXi bootloader URL without requiring any traditional PXE infrastructure.

This was especially useful for anyone testing or developing ESXi automation for use with ESXi Kickstart, where you can quickly prototype your automation without additional infrastructure dependencies and once the automation has been vetted, you can then leverage that exact same automation in your physical ESXi provisioning infrastructure.

The original solution had only supported HTTP and I recently came to learn that we can now also support HTTPS in vSphere 8!

To take advantage of this new capability, you just need to have a physical server running ESXi 8.0 or Workstation 17 and a VM that is configured with the latest vHW20 compatibility.

To configure HTTPS boot, you will need to add the following four VM Advanced Settings:

• networkBootProtocol - httpv4 or httpv6
• networkBootUri - HTTPS URL to the ESXi bootloader (bootx64.efi)
• uefi.tls.CaCertificateDefault.present - TRUE
• uefi.tls.CaCertificateDefault.file0 - DER encoded file of the HTTPS server TLS certificate that is accessible to the VM on the ESXi host

Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware

Step 1 - Download and extract the desired ESXi ISO to your HTTP server. In my example, I am using ESXi 8.0a and have extracted the contents into a directory called esxi80a and placed that into the root of my HTTPS server, which in my setup is /etc/httpd/html

Step 2 - You will need to update the ESXi boot.cfg configuration which is located in efi/boot directory with the following:

• Remove the leading "/" character in front of all module names
• Update the prefix parameter with the base URL of the extracted ESXi image

Here is what my /etc/httpd/html/esxi80a/efi/boot/boot.cfg looks like after applying the two changes mentioned above:

bootstate=0
title=Loading ESXi installer
timeout=5
prefix=https://web.lab.local/esxi80a
kernel=b.b00
kernelopt=runweasel cdromBoot
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- uc_intel.b00 --- uc_amd.b00 --- uc_hygon.b00 --- procfs.b00 --- vmx.v00 --- vim.v00 --- tpm.v00 --- sb.v00 --- s.v00 --- atlantic.v00 --- bcm_mpi3.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- brcmfcoe.v00 --- cndi_igc.v00 --- dwi2c.v00 --- elxiscsi.v00 --- elxnet.v00 --- i40en.v00 --- iavmd.v00 --- icen.v00 --- igbn.v00 --- ionic_en.v00 --- irdman.v00 --- iser.v00 --- ixgben.v00 --- lpfc.v00 --- lpnic.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsi_msgp.v01 --- lsi_msgp.v02 --- mtip32xx.v00 --- ne1000.v00 --- nenic.v00 --- nfnic.v00 --- nhpsa.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- ntg3.v00 --- nvme_pci.v00 --- nvmerdma.v00 --- nvmetcp.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- pvscsi.v00 --- qcnic.v00 --- qedentv.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- qflge.v00 --- rdmahl.v00 --- rste.v00 --- sfvmk.v00 --- smartpqi.v00 --- vmkata.v00 --- vmksdhci.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- bmcal.v00 --- clusters.v00 --- crx.v00 --- drivervm.v00 --- elx_esx_.v00 --- btldr.v00 --- esx_dvfi.v00 --- esx_ui.v00 --- esxupdt.v00 --- tpmesxup.v00 --- weaselin.v00 --- esxio_co.v00 --- loadesx.v00 --- lsuv2_hp.v00 --- lsuv2_in.v00 --- lsuv2_ls.v00 --- lsuv2_nv.v00 --- lsuv2_oe.v00 --- lsuv2_oe.v01 --- lsuv2_sm.v00 --- native_m.v00 --- qlnative.v00 --- trx.v00 --- vdfs.v00 --- vmware_e.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- tools.t00 --- xorg.v00 --- gc.v00 --- imgdb.tgz --- basemisc.tgz --- resvibs.tgz --- esxiodpt.tgz --- imgpayld.tgz
build=8.0.0-1.10.20842819
updated=0

Note: Make sure the extracted ESXi directory has the correct permissions. I simply used chmod 655 -R esxi80a/ to recursively apply the change for all files/directories.

Step 3 - Next, we need to get a copy of the SSL certificate of the HTTPS server (not the private key) but just the certificate file and encode that to DER format. Here is an example if you were installing HTTPS server from scratch, you would need to generate a TLS certificate using something like the following:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt

The apache-selfsigned.crt is the file we are interested in. To convert from the default PEM format to DER, you can run the following command:

openssl x509 -in /etc/ssl/certs/apache-selfsigned.crt -outform der -out apache-selfsigned.der

Once you have a copy of the TLS certificate in the DER format, you will need to transfer that to ESXi host so that the VM has access and can validate when it attempts to boot from the HTTPS server. You can place the file within the VM directory but if you intend to use this for more than a single VM, then I would recommend creating a new directory on one of your ESXi datastores and place the TLS certificate there which can then be referenced by multiple VMs.

In my example, I am placing it on a vSAN Datastore under a directory that I created called uefi-boot-tls-cert and the full path is the following: /vmfs/volumes/sm-vsanDatastore/uefi-boot-tls-cert/apache-selfsigned.der

Step 4 - Create new vHW20 Nested ESXi VM and configure the four required VM Advanced Settings containing the values from your environment:

• networkBootProtocol = httpv4
• networkBootUri = https://web.lab.local/esxi80a/efi/boot/bootx64.efi
• uefi.tls.CaCertificateDefault.present = "TRUE"
• uefi.tls.CaCertificateDefault.file0 = "/vmfs/volumes/sm-vsanDatastore/uefi-boot-tls-cert/apache-selfsigned.der"

Step 5 - Finally, power on the VM and you should see the VM attempt EFI Network boot over HTTPS and you should see the ESXi installer start to boot as shown in the screenshot below.

I did not specify kickstart location, so this will simply boot into the ESXi interactive installer. To do so, just update the kernelopt line to reference the KS URL as described in Step 6 of the official VMware documentation.