Automated ESXi Installation with a USB Network Adapter using Kickstart

I have been working with the Project Keswick team for quite some time now, which is an OCTO project is lead by my good friend Alan Renouf, who is doing some really innovative work with ESXi at the edge and application deployment using a desired state engine.

Recently I had met with the team to discuss some of the options for their automated deployment which uses the tried and true ESXi scripted installation aka ESXi Kickstart. One thing that I had shared was just how powerful the %pre section within the kickstart is and can be used to redefine or update the original kickstart based on your installation criteria. For example, you could pull down external configuration files and determine at runtime to decide how you want to configure your networking to even fully bootstrapping a local vSAN datastore and this would all happen prior to ESXi installer starting. I have used the %pre section numerous times as a customer and also demonstrated in my USB-to-SDDC project which has also been an inspiration for the Project Keswick team.

One very cool capability that Project Keswick is enabling is the integration of the popular USB Network Native Driver for ESXi and one challenge they had faced with automating an ESXi installation when only a USB network adapter was available is additional configuration that must be setup before the installer can begin. They shared their solution and thought this would be a good blog post topic, especially as I know many folks use the USB Network Native Driver for ESXi in their homelab and if you wish to automate the installation, the solution shared from the team could help.

After looking at the code example and thinking about all the possible scenarios one could have from system with only USB network adapter or only standard network adapter as well as a combination of the two, I wanted to have a more robust solution that did not require managing multiple kickstart configuration files based on the physical network connectivity.

Another really powerful capability that is part of ESXi Kickstart is the use of the %include section, which is not unique to ESXi but rather general Linux Kickstart. In fact, it was more than a decade plus ago  where I had learned about these tricks when working on our Linux/UNIX automation platform for one of my employers who made extensive use of %include and %pre to perform a number of runtime decisions and apply configurations that would occur prior to the OS installation and then perform other operations using %post.

The use of the %include section would allow me to decide on what type of configuration would be needed based on the physical networking that was identified and creates a placeholder which is then processed within %pre section.

Below is fully functional ESXi Kickstart that would do the "right" thing depending on the type of connection it detected (vusb0 for USB networking vs vmnic0 for standard networking):

vmaccepteula
install --firstdisk=usb --novmfsondisk
rootpw VMware1!
reboot

# Network configuration will be defined dynamically based on detected pNIC in %pre section
%include /tmp/ks_network

%pre --interpreter=busybox

MGMT_USB_NIC_NAME="vusb0"
MGMT_VMKERNEL_NAME="vmk0"
MGMT_VSWITCH_NAME="vSwitch0"
MGMT_PG_NAME="Management Network"

# Define network configuration based on whether we have standard or USB NIC
if [ $(localcli network nic list | grep -vE '(PCI|---)' | awk '{print $1}' | head -1) == "${MGMT_USB_NIC_NAME}" ]; then
    localcli network vswitch standard add -v ${MGMT_VSWITCH_NAME}
    localcli network vswitch standard portgroup add -p "${MGMT_PG_NAME}" -v ${MGMT_VSWITCH_NAME}
    localcli network vswitch standard uplink add -u ${MGMT_USB_NIC_NAME} -v ${MGMT_VSWITCH_NAME}
    localcli network ip interface add -i ${MGMT_VMKERNEL_NAME} -p "${MGMT_PG_NAME}"
    localcli network ip interface ipv4 set -i ${MGMT_VMKERNEL_NAME} -t dhcp

    echo "network --bootproto=dhcp --device=${MGMT_USB_NIC_NAME}" > /tmp/ks_network
else
    echo 'network --bootproto=dhcp --device=vmnic0' > /tmp/ks_network
fi

%post --interpreter=busybox
# Only needed for Intel 12th Gen CPU or later w/hybrid CPU
sed -i 's/autoPartition.*/autoPartition=FALSE cpuUniformityHardCheckPanic=FALSE/g' /vmfs/volumes/BOOTBANK1/boot.cfg

%firstboot --interpreter=busybox

# Ensure hostd is ready
while ! vim-cmd hostsvc/runtimeinfo; do
sleep 10
done

MGMT_USB_NIC_NAME="vusb0"
if [ $(esxcli network nic list | grep -vE '(PCI|---)' | awk '{print $1}' | head -1) == "${MGMT_USB_NIC_NAME}" ]; then
    # Restore USB NIC binding to vSwitch portgroups
    esxcfg-vswitch -R

    esxcli system module parameters set -p "usbBusFullScanOnBootEnabled=1" -m vmkusb_nic_fling
fi

# refresh neworking on DCUI
vim-cmd hostsvc/net/refresh

# enable & start SSH
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

# Suppress ESXi Shell warning
esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1
# Suppress Coredump warning
esxcli system settings advanced set -o /UserVars/SuppressCoredumpWarning -i 1

Note: For USB networking, the use of esxcfg-switch -R command is required to restore the physical network binding upon the initial system reboot after installation as outlined in here. Once ESXi has been installed, we can take advantage of the USB Native Network Driver for ESXi parameter enhancement to ensure the physical network binding is always performed.

In addition to copying the KS.CFG to root of your ESXi installation USB device, make sure to also update efi/boot/boot.cfg file to reference the KS.CFG.

bootstate=0
title=Loading ESXi installer
timeout=5
prefix=
kernel=/b.b00
kernelopt=ks=usb:/KS.CFG cpuUniformityHardCheckPanic=FALSE
modules=/jumpstrt.gz --- /useropts.gz --- /features.gz --- /k.b00 --- /uc_intel.b00 --- /uc_amd.b00 --- /uc_hygon.b00 --- /procfs.b00 --- /vmx.v00 --- /vim.v00 --- /tpm.v00 --- /sb.v00 --- /s.v00 --- /atlantic.v00 --- /bcm_mpi3.v00 --- /bnxtnet.v00 --- /bnxtroce.v00 --- /brcmfcoe.v00 --- /cndi_igc.v00 --- /dwi2c.v00 --- /elxiscsi.v00 --- /elxnet.v00 --- /i40en.v00 --- /iavmd.v00 --- /icen.v00 --- /igbn.v00 --- /ionic_en.v00 --- /irdman.v00 --- /iser.v00 --- /ixgben.v00 --- /lpfc.v00 --- /lpnic.v00 --- /lsi_mr3.v00 --- /lsi_msgp.v00 --- /lsi_msgp.v01 --- /lsi_msgp.v02 --- /mtip32xx.v00 --- /ne1000.v00 --- /nenic.v00 --- /nfnic.v00 --- /nhpsa.v00 --- /nmlx5_co.v00 --- /nmlx5_rd.v00 --- /ntg3.v00 --- /nvme_pci.v00 --- /nvmerdma.v00 --- /nvmetcp.v00 --- /nvmxnet3.v00 --- /nvmxnet3.v01 --- /pvscsi.v00 --- /qcnic.v00 --- /qedentv.v00 --- /qedrntv.v00 --- /qfle3.v00 --- /qfle3f.v00 --- /qfle3i.v00 --- /qflge.v00 --- /rdmahl.v00 --- /rste.v00 --- /sfvmk.v00 --- /smartpqi.v00 --- /vmkata.v00 --- /vmksdhci.v00 --- /vmkusb_n.v00 --- /vmkusb.v00 --- /vmw_ahci.v00 --- /bmcal.v00 --- /clusters.v00 --- /crx.v00 --- /drivervm.v00 --- /elx_esx_.v00 --- /btldr.v00 --- /esx_dvfi.v00 --- /esx_ui.v00 --- /esxupdt.v00 --- /tpmesxup.v00 --- /weaselin.v00 --- /esxio_co.v00 --- /loadesx.v00 --- /lsuv2_hp.v00 --- /lsuv2_in.v00 --- /lsuv2_ls.v00 --- /lsuv2_nv.v00 --- /lsuv2_oe.v00 --- /lsuv2_oe.v01 --- /lsuv2_sm.v00 --- /native_m.v00 --- /qlnative.v00 --- /trx.v00 --- /vdfs.v00 --- /vmware_e.v00 --- /vsan.v00 --- /vsanheal.v00 --- /vsanmgmt.v00 --- /tools.t00 --- /xorg.v00 --- /gc.v00 --- /imgdb.tgz --- /basemisc.tgz --- /resvibs.tgz --- /esxiodpt.tgz --- /imgpayld.tgz
build=8.0.0-1.10.20842819
updated=0

If you are interested in more details for automating ESXi installation on a USB device, please see this previous blog post for more information.