Configuring dnsmasq as PXE Server for ESXi 

One really cool thing that I came to learn while setting up the infrastructure to network boot the latest Raspberry Pi 4 was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. In addition to providing DNS services, it can also be configured to run TFTP and provide DHCP capabilities which can then be used to support PXE installations.

Another neat feature of dnsmasq is ability to proxy to an existing DHCP server which is extremely useful for anyone with an existing DHCP infrastructure. Given the simplicity of dnsmasq and having already set this up for the rPI, I figure it would also be useful to take folks through in setting up dnsmasq to also support ESXi installations over PXE, since this still comes up from new folks just getting started with ESXi kickstart automation.

For more details about PXE installation of ESXi, I highly recommend this whitepaper and although it states 6.0, the concepts and configurations are still applicable to the latest ESXi 7.0 release.

Step 1 - Install an Ubuntu VM, I happen to have 16.04 lying around in my vSphere Datastore but you can use any recent version

Step 2 - Apply latest patches and install the following packages:

apt update
apt install -y nfs-kernel-server dnsmasq kpartx unzip

Step 3 - Download the desired ESXi ISO and upload that to root of the Ubuntu VM. In my example, I am using the VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso.

Step 4 - Mount the ESXi ISO and copy the installation files to our TFTP root directory. In my example, I use the name esxi7 but you can use any name you wish and if you decided to change it, make sure all steps after this also matches the name.

mount -o loop VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso /mnt
cp -rf /mnt /srv/tftpboot/esxi7/
umount /mnt

Step 5 - We need to remove the trailing "/" from both boot.cfg as the installation files will be served from the current path.

sed -i 's#/##g' /srv/tftpboot/esxi7/boot.cfg
sed -i 's#/##g' /srv/tftpboot/esxi7/efi/boot/boot.cfg

Step 6 - There are a number of ways you can serve the kickstart configuration file (ks.cfg) to the ESXi host that is being PXE booted. In this example, I will be using NFS as I already had that setup from the Raspberry Pi 4 article. You can certainly adjust this to use other supported options.

mkdir -p /srv/nfs/esxi7
echo "/srv/nfs/esxi7 *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

Below is a very basic ks.cfg that you can use testing purposes and save this to /srv/nfs/esxi7/ks.cfg

vmaccepteula
install --firstdisk --overwritevmfs --novmfsondisk
reboot

rootpw VMware1!
network --bootproto=dhcp --device=vmnic0

%firstboot --interpreter=busybox

# enable & start SSH
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

For the ESXi installer to locate our ks.cfg, we need to update the respective boot.cfg with kernelopt=ks=nfs://<kickstart-server>/srv/nfs/esxi7/ks.cfg and this will be covered in the next section depending if you are going to be using EFI or BIOS firmware.

Step 7 - Create the dnsmasq configuration /etc/dnsmasq.conf which can serve BIOS and UEFI systems. If you intend to support both, please follow BOTH step 8a and 8b section. If not using one of them, please comment it out as dnsmasq will throw an error if those files are not found.

There are a number of dnsmasq configurations that you can tweak, but if you have an existing DHCP server and simply want to use it, simply specify your network broadcast address and "proxy" keyword as shown in the example below.

dhcp-range=192.168.30.255,proxy
log-dhcp
enable-tftp
tftp-root=/srv/tftpboot

pxe-prompt="Booting PXE Client", 1

# ESXi x86 BIOS
dhcp-boot=esxi7/pxelinux
pxe-service=X86PC, "Boot ESXi x86 BIOS Installer", esxi7/pxelinux

# ESXi x86 EFI
dhcp-match=set:efi64,60,PXEClient:Arch:00007
dhcp-boot=tag:efi64,esxi7/efi/boot/bootx64.efi

# ;)

Step 8 -

For BIOS:

Step 8ai - For legacy BIOS, you must use syslinux 3.86 and the following commands will download pxelinux.0

wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/3.xx/syslinux-3.86.zip
unzip syslinux-3.86.zip -d syslinux
rm syslinux-3.86.zip
cp syslinux/core/pxelinux.0 /srv/tftpboot/esxi7/

Step 8aii - Next, we create the expected pxelinux directory structure and simply use the "default" file for testing purposes. In general, you will want to create a file with the ESXi physical MAC Address (e.g. 00-11-22-33-44-55-66) so that you can control which hosts are PXE booted and the version of ESXi they will receive.

mkdir -p /srv/tftpboot/esxi7/pxelinux.cfg
cp /srv/tftpboot/esxi7/isolinux.cfg /srv/tftpboot/esxi7/pxelinux.cfg/default

Step 8aiii - Update the boot.cfg to specify how to download our ks.cfg and replace the IP Address with address of your Kickstart system

sed -i 's#kernelopt=.*#kernelopt=ks=nfs://192.168.30.176/srv/nfs/esxi7/ks.cfg#g' /srv/tftpboot/esxi7/boot.cfg

For UEFI:

Step 8bi - Update the prefix to point to our root esxi7 directory or else it will assume the installation files will be served from efi/boot

sed -i 's/prefix.*/prefix=esxi7/g' /srv/tftpboot/esxi7/efi/boot/boot.cfg

Step 8bii - Update the boot.cfg to specify how to download our ks.cfg and replace the IP Address with address of your Kickstart system

sed -i 's#kernelopt=.*#kernelopt=ks=nfs://192.168.30.176/srv/nfs/esxi7/ks.cfg#g' /srv/tftpboot/esxi7/efi/boot/boot.cfg

Step 9 - Enable and start the services

systemctl enable dnsmasq
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start dnsmasq
systemctl start rpcbind
systemctl start nfs-serverq

Step 10 - Create either a Nested ESXi VM (BIOS or EFI) or use a physical host and power on the system. If everything was configured correctly, you should see something like the following and after the ESXi installation files are loaded, it should mount the NFS volume to retrieve the ks.cfg and start the unattended installation and finally reboot.