Automated network scripted installation of ESXi-Arm without SD Card

A topic that I have been working on even before the release of the ESXi-Arm Fling is the ability to perform a network scripted installation (Kickstart) of ESXi-Arm for the Raspberry Pi (rPI) but doing so without the need of an SD Card plugged into the rPI itself. The latter point is critical as today, the SD Card is used to house the required UEFI files to allow the rPI to boot the ESXi-Arm installation files which can be served from a local USB device or remotely over the network using HTTP, NFS or even iSCSI boot, which Andrei had recently blogged about.

Running through the SD Card preparation is not a difficult process and if you only have a single ESXi-Arm host, this may not be all that interesting other than learning about how this works and setting up a basic Kickstart environment. However, if you have several rPI and maybe you do not have spare SD Card and you prefer to make it easy to deploy additional ESXi-Arm host, this is a pretty cool solution. The precursor to this work was actually from a blog post I had published a few weeks ago on copying the rPI UEFI files and booting ESXi-Arm off of a USB device.

Once I figured out how that worked, it was simply figuring out the automation required during the %post section of a scripted installation of ESXi-Arm to pull down a copy of the UEFI files which is then copied onto the first partition of the USB device and thus allowing you to completely boot off of the USB device after installation. This took some trial and error playing around with mcopy which is a tool I have written about to help copy files to and from a USB device using the ESXi Shell. The other trick that we are taking advantage of here is that the USB device that you intend to use are mostly the same from a UEFI point of view and by disabling all other boot option, we ensure that after the UEFI files are copied over to ESXi-Arm host, it will boot from USB device rather than over the network.

I will not bore you further with the details, but can take a look at the kickstart script to see what was needed along with the detailed the instructions below so that you can now easily deploy new ESXi-Arm hosts running o rPI without having to go through the manual SD Card setup and creating a bootable ESXi-Arm installer. A HUGE thanks goes out to Cyprien Laplace who was instrumental in helping me piece together the overall end to end solution.

Setup Kickstart Server

Step 1 - Install an x86 Ubuntu VM some where on your network. I happen to have 16.04 lying around in my vSphere Datastore but you can use any recent version. We will be installing dnsmasq which I have blogged about in the past as a PXE/Kickstart host for automating x86 ESXi. In addition to the lightweight setup, it also integrates with your existing network which may already have a DHCP server running. This is always tricky as most tutorials online will have you setup a new DHCP server which generally is not a good idea unless you are ready for some pain.

Step 2 - Apply the latest patches and install the following packages:

apt update
apt install -y dnsmasq apache2 unzip

Step 3 - Create the following dnsmasq configuration under /etc/dnsmasq.conf and replace the dhcp-range with your subnet. In my example, I have 192.168.30.0/24 network. dnsmasq supports a ton of different settings, you can refer to their documentation for more details.

dhcp-range=192.168.30.255,proxy
log-dhcp
enable-tftp
tftp-root=/srv/tftpboot

pxe-prompt="Booting PXE Client", 1

# ESXi AARCH64
dhcp-match=set:aarch64,60,PXEClient:Arch:00011:UNDI:003000
dhcp-boot=tag:aarch64,esxi-arm/efi/boot/bootaa64.efi

# rPI AARCH64
pxe-service=0,"Raspberry Pi Boot"

Step 4 - Create the following directory structure which will be used to host both our UEFI and ESXi-Arm installation files

mkdir -p /srv/tftpboot/
mkdir -p /srv/tftpboot/rpi-uefi-1.20/

Step 5 - Download and upload the ESXi-Arm ISO to the Kickstart server

Step 6 - Mount the ESXi-Arm ISO and copy the installation files to our TFTP directory

mount -o loop VMware-VMvisor-Installer-7.0.0-17068872.aarch64.iso /mnt/
cp -rf /mnt /srv/tftpboot/esxi-arm
umount /mnt

Step 7 - We need to make a few changes to the ESXi-Arm boot.cfg file to tell it where to find our kickstart configuration file (replace the IP Address with the IP of your Kickstart server) along with the path to the files.

Run the following three commands to perform these operations:

sed -i 's#/##g' /srv/tftpboot/esxi-arm/efi/boot/boot.cfg
sed -i 's#prefix.*#prefix=esxi-arm#g' /srv/tftpboot/esxi-arm/efi/boot/boot.cfg
sed -i 's#kernelopt=.*#kernelopt=ks=http://192.168.30.176/ks.cfg#g' /srv/tftpboot/esxi-arm/efi/boot/boot.cfg

Here is what the final boot.cfg should look like:

bootstate=0
title=Loading ESXi installer
timeout=5
prefix=esxi-arm
kernel=b.b00
kernelopt=ks=http://192.168.30.176/ks.cfg autoPartitionOSDataSize=4096
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- procfs.b00 --- vmx.v00 --- vim.v00 --- tpm.v00 --- sb.v00 --- s.v00 --- ena.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- brcmfcoe.v00 --- brcmnvme.v00 --- elxiscsi.v00 --- elxnet.v00 --- i40en.v00 --- i40iwn.v00 --- iavmd.v00 --- igbn.v00 --- iser.v00 --- ixgben.v00 --- lpfc.v00 --- lpnic.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsi_msgp.v01 --- lsi_msgp.v02 --- mtip32xx.v00 --- ne1000.v00 --- nenic.v00 --- nfnic.v00 --- nhpsa.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- ntg3.v00 --- nvme_pci.v00 --- nvmerdma.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- pvscsi.v00 --- qcnic.v00 --- qedentv.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- qflge.v00 --- rste.v00 --- sfvmk.v00 --- smartpqi.v00 --- vmkata.v00 --- vmkfcoe.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- elx_esx_.v00 --- btldr.v00 --- esx_dvfi.v00 --- esx_ui.v00 --- esxupdt.v00 --- tpmesxup.v00 --- weaselin.v00 --- loadesx.v00 --- lsuv2_hp.v00 --- lsuv2_in.v00 --- lsuv2_ls.v00 --- lsuv2_nv.v00 --- lsuv2_oe.v00 --- lsuv2_oe.v01 --- lsuv2_oe.v02 --- lsuv2_sm.v00 --- native_m.v00 --- qlnative.v00 --- vmware_e.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- tools.t00 --- imgdb.tgz --- imgpayld.tgz
build=7.0.0-1.0.17068872
updated=0

Step 8 - Create /var/www/html/ks.cfg which is our Kickstart configuration file and replace the IP Address within the file to reflect the address of your Kickstart server which will be serving out the uefi.zip file that we had copied over earlier.

vmaccepteula
install --firstdisk=usb --overwritevmfs
reboot

rootpw VMware1!
network --bootproto=dhcp --device=vmnic128

%post --interpreter=busybox
mkdir -p /tmp/uefi
wget http://192.168.30.176/uefi.zip -O /tmp/uefi/ufei.zip
unzip /tmp/uefi/ufei.zip -d /tmp/uefi/
rm -f /tmp/uefi/ufei.zip
USB_DEVICE=$(grep "install --firstdisk ==" /var/log/weasel.log | grep -v grep | awk '{print $6}')
mcopy -i "/dev/disks/${USB_DEVICE}:1" -vs /tmp/uefi/* ::/
 
%firstboot --interpreter=busybox
 
# Ensure hostd is ready
while ! vim-cmd hostsvc/runtimeinfo; do
sleep 10
done
 
# enable & start SSH
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
 
# enable & start ESXi Shell
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

Note: The Kickstart configuration above is just a basic example which assumes DHCP for networking and simply enables and starts ESXi Shell/SSH. You can certainly customize the ESXi-Arm further, for more details, you can check out my extensive ESXi Kickstart resources.

Step 9 - Finally, we need to enable and start the dnsmasq and apache2 services

systemctl enable dnsmasq
systemctl start dnsmasq
systemctl enable apache2
systemctl start apache2

At this point, you have completed the setup of the ESXi-Arm Kickstart server. The next section is to prep our rPI and this ONLY needs to be done once. After that, you can deploy additional rPI without going through these initial setup steps again. You just need the serial number of the rPI to update in dnsmasq configuration which we will cover in the section below.

Setup rPI

Step 1 - Download and install the Raspberry Pi Imager Tool for your desktop OS. This is needed as we need to install Raspberry Pi (rPI) OS onto our SD Card so that we can change the default rPI boot order to 0xf241 which attempts booting using the following order: Network Boot, USB and then SD Card. If no bootable devices are found, the sequence is repeated all over again. This order is important because after our network installation, our USB device will contain ESXi installation and we need to ensure it comes after network boot but before SD Card.

Step 2 - . Power up the rPI with the SD Card plugged in, which has rPI OS image. Once rPI OS boots up, open up a terminal and run the following command to apply latest EEPROM and update the boot order, which is only available when using the command-line.

PI_EEPROM_VERSION=pieeprom-2020-09-03
wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-config ${PI_EEPROM_VERSION}.bin > bootconf.txt
sed -i 's/BOOT_ORDER=.*/BOOT_ORDER=0xf241/g' bootconf.txt
sudo rpi-eeprom-config --out ${PI_EEPROM_VERSION}-netboot.bin --config bootconf.txt ${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-update -d -f ./${PI_EEPROM_VERSION}-netboot.bin

Reboot for the changes to go into effect. At this point, you can shutdown the rPI.

Step 3 - Unplug the SD Card and follow Section 3.2 of the rPI ESXi-Arm Fling documentation in preparing the SD Card with the UEFI Image. Once this has step has been completed, plug the SD Card back into rPI along with an empty USB device which will hold the ESXi installation.

Note: This step is very important, please ensure the order is done correctly prior to moving to the next step.

Step 4 - Power up the rPI and you should see a rainbow screen and then the rPI logo. Hit Esc to go into the UEFI configurations. Navigate to Boot Manager->Boot Options->Delete Boot Option and here you should see several boot options covering PXE/HTTP, SD Card and the USB device. If you do NOT see the USB device, please verify that the device is plugged in as this is critical step.

Select ALL boot option except for the USB device as shown in the screenshot and then select Commit Changes and Exit which will now updates the UEFI boot order to only boot off of a USB device and these changes are now saved onto the SD Card.

Step 5 - Unplug the SD Card and plug that into your local computer. Create a folder on your desktop called "uefi" and copy all contents from the SD Card to this folder.

Step 6 - Zip up the contents of the "uefi" folder and not the folder itself. On a Mac, this can be done by change into the folder and running the following command from within the folder itself and name it uefi.zip:

zip -r ../uefi.zip *

Step 7 - SCP the uefi.zip file to our Kickstart server under /srv/tftpboot/rpi-uefi-1.20/

Step 8 - Unzip the contents but leave the uefi.zip file since we also need to place that under our

cd /srv/tftpboot/rpi-uefi-1.20/
unzip uefi.zip

Step 9 - Copy the uefi.zip to the root of our HTTP server, which will be downloaded as part of the post-installation of ESXi and allow ESXi-Arm to boot off of the USB device as it will have a copy of the UEFI files

cp uefi.zip /var/www/html/

Step 10 - We need to obtain the serial number of our rPI as it expects the UEFI files to be placed in a directory with that ID. You can easily do this by just powering on the rPI and serial will be displayed under the "board:" line as shown in the screenshot. In my example below, it is 49a6ff15

Login to Kickstart server and we will just create a symlink for our rPI serial to our UEFI files which is stored in /srv/tftpboot/rpi-uefi-1.20/ by running the following command:

ln -s /srv/tftpboot/rpi-uefi-1.20/ 49a6ff15

This trick allows us to manage the UEFI files from a single directory while allowing for multiple rPI to network boot. This was another nice tidbit I had learned from Cyprien Laplace

Step 11 - Okay, we are finally ready to kickstart our rPI and if the system! If the system has not already cycled through the boot sequence, it should have kicked off and you should see the ESXi-Arm installation happening if everything was configured correctly. You can also monitor the progress if you have a monitor connected up to the rPI which I would recommend if this is your first time setting this up. You can also monitor the logs on the Kickstart server by tailing the /var/log/syslog and one thing to note is that for each file, I have noticed for my setup, it will say failed to send but the next line immediately says it successfully sent, so you can safely ignore.

Once the installation has completed, it should fully boot off of the USB device and no longer perform a network boot and you now have an easy way to manage and deploy your ESXi-Arm hosts on rPI!