Custom UEFI Boot Menu for ESXi 9.0 using rEFInd

Unless I am working with an unreleased version of ESXi, I usually install it by booting from a USB drive that includes an embedded Kickstart configuration file, that will fully automate the installation.

For a single ESXi host deployment, this workflow works great but if you have more than one host with different ESXi Kickstart configuration files, you will either need multiple USB drives or you have to keep updating the embedded Kickstart file reference using a single USB device.

There are a number of different ways to solve this problem, I have covered many of the solutions in my Tech Deep Dive: Automating VMware ESXi Installation at Scale session at VMware Explore.

With that said, since I have been deploying several Minisforum MS-A2 for use with VMware Cloud Foundation (VCF) 9, I wanted to explore a solution that had been requested in the past but I never had the time to look into, which is creating a custom ESXi boot menu that would allow you to select specific ESXi version and/or kickstart configuration files.

In earlier releases of ESXi where BIOS firmware was the predominant deployment type, you could easily create boot menus using SYSLINUX and you can see an example of this in this 2015 blog post. Today, most servers only support UEFI firmware and ESXi 8.0 has also officially dropped support for BIOS firmware, since you will not find modern systems that even have support for it.

There are a number of UEFI boot loader tools that can be used to help create custom boot menus like iPXE and rEFInd to just name a few. In my basic research, I found rEFInd to be easier to get started with and with some trial/error, I was able to create custom menu that allowed me to point to specific ESX Kickstart configuration files as shown in the screenshot below that would completely boot off of a USB drive without relying on any external or network dependencies.

Step 1 - You will need a single USB device that is formatted with a single FAT32 partition with your desired label name. In my example, I have named the partition VCF and all instructions below will be assuming that label name.

As a macOS user, you can use the diskutil command-line tool to partition your USB device using the following (please replace the path to your USB device):

diskutil partitionDisk /dev/disk4 1 MBRFormat "MS-DOS" VCF R

Step 2 - Run the following commands to create the required directories on the USB device:

mkdir -p /Volumes/VCF/EFI/BOOT/
mkdir -p /Volumes/VCF/esx9
mkdir -p /Volumes/VCF/kickstart

• /EFI/BOOT directory will contain the rEFInd bootloader and boot menu configuration
• /esx9 directory will contain the extracted contents from an ESXi 9.0 ISO (you can certainly add other versions by creating additional directories at the top level of the USB device)
• /kickstart directory will contain the unique ESXi kickstart configuration files which is referenced by the specific boot.cfg file that is loaded

Step 3 - Extract the contents of the ESX 9.0 ISO into the /esx9 directory of your USB device

Step 4 - We need to update the /Volumes/VCF/esx9/EFI/BOOT/BOOT.CFG configuration file with the following changes:

1. Update the prefix value to the absolute path of the extracted ESXi binaries which in our example would be located in the /esx9/ directory and you can certainly have multiple versions of ESXi on the USB device
2. Remove the forward slash in front of all the module names or else you will run into a file not found when ESXi bootloader runs

bootstate=0
title=Loading ESXi installer
timeout=5
prefix=/esx9/
kernel=b.b00
kernelopt=
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- uc_intel.b00 --- uc_amd.b00 --- uc_hygon.b00 --- procfs.b00 --- vmx.v00 --- vim.v00 --- tpm.v00 --- sb.v00 --- s.v00 --- atlantic.v00 --- bcm_mpi3.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- cndi_igc.v00 --- dwi2c.v00 --- elxnet.v00 --- i40en.v00 --- iavmd.v00 --- icen.v00 --- igbn.v00 --- intelgpi.v00 --- ionic_cl.v00 --- ionic_en.v00 --- irdman.v00 --- iser.v00 --- ixgben.v00 --- lpfc.v00 --- lpnic.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsi_msgp.v01 --- ne1000.v00 --- nenic_en.v00 --- nenic.v00 --- nfnic.v00 --- nhpsa.v00 --- nipmi.v00 --- nmlx5_cc.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- ntg3.v00 --- nvme_pci.v00 --- nvmerdma.v00 --- nvmetcp.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- pvscsi.v00 --- qcnic.v00 --- qedentv.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- rdmahl.v00 --- rshim_ne.v00 --- rshim.v00 --- sfvmk.v00 --- smartpqi.v00 --- vmkata.v00 --- vmksdhci.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- bmcal.v00 --- clusters.v00 --- crx.v00 --- drivervm.v00 --- btldr.v00 --- dvfilter.v00 --- esx_ui.v00 --- esxupdt.v00 --- tpmesxup.v00 --- weaselin.v00 --- xorg.v00 --- esxio_co.v00 --- infravis.v00 --- loadesx.v00 --- hpv2_hps.v00 --- intelv2_.v00 --- lsiv2_dr.v00 --- nvme_pci.v01 --- oem_dell.v00 --- oem_leno.v00 --- smartpqi.v01 --- native_m.v00 --- podvm_ro.v00 --- qlnative.v00 --- trx.v00 --- vcls_pod.v00 --- vdfs.v00 --- vds_vsip.v00 --- vmware_e.v00 --- vmware_f.v00 --- hbrsrv.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- tools.t00 --- qp_esx_d.v00 --- nsx_adf.v00 --- nsx_cfga.v00 --- nsx_cont.v00 --- nsx_cpp_.v00 --- nsx_esx_.v00 --- nsx_expo.v00 --- nsx_host.v00 --- nsx_ids.v00 --- nsx_moni.v00 --- nsx_mpa.v00 --- nsx_nest.v00 --- nsx_neto.v00 --- nsx_opsa.v00 --- nsx_plat.v00 --- nsx_prot.v00 --- nsx_prox.v00 --- nsx_pyth.v00 --- nsx_pyth.v01 --- nsx_pyth.v02 --- nsx_sfhc.v00 --- nsx_shar.v00 --- nsx_snpr.v00 --- nsx_vdpi.v00 --- nsxcli.v00 --- vsipfwli.v00 --- gc.v00 --- imgdb.tgz --- basemisc.tgz --- resvibs.tgz --- esxiodpt.tgz --- imgpayld.tgz
build=9.0.0-0.24755229
updated=0

Step 5 - Copy your desired ESXi kickstart configuration files and placed them under /kickstart directory and ensure all filenames are capitalized. In my example, I have the following:

• /kickstart/KS-ESX01.CFG
• /kickstart/KS-ESX02.CFG
• /kickstart/KS-ESX03.CFG
• /kickstart/KS-ESX04.CFG

Step 7 - Download the rEFInd bootloader (e.g. refind-bin-0.14.2.zip) and extract refind_x64.efi from refind directory and copy that to /Volumes/VCF/EFI/BOOT/bootx64.efi (note the rename of the file)

Step 8 - Finally, create the rEFInd configuration file under /Volumes/VCF/EFI/BOOT/refind.conf with the following configuration, which will define your custom boot entries to the specific ESXi version to boot and the respective Kickstart configuration file to use by passing it as a boot option

timeout 40
textonly
scanfor manual
showtools shell, reboot

menuentry "ESX 9 Kickstart Host 1" {
    loader /esx9/efi/boot/bootx64.efi
    options "ks=usb:/kickstart/KS-ESX01.CFG"
}

menuentry "ESX 9 Kickstart Host 2" {
    loader /esx9/efi/boot/bootx64.efi
    options "ks=usb:/kickstart/KS-ESX02.CFG"
}

menuentry "ESX 9 Kickstart Host 3" {
    loader /esx9/efi/boot/bootx64.efi
    options "ks=usb:/kickstart/KS-ESX03.CFG"
}

menuentry "ESX 9 Kickstart Host 4" {
    loader /esx9/efi/boot/bootx64.efi
    options "ks=usb:/kickstart/KS-ESX04.CFG"
}

Here is an abbreviated screenshot of what your USB drive structure should look like once you have reached Step 8

You can now unmount the USB drive from your desktop and plug that into a system and when it boots up, you should see rEFInd bootloader with your custom menu items! There is ton of customizations that rEFInd supports, check out the various themes from this rEFInd Theme Collection to get an idea on what can be done.