Two methods to network boot Raspberry Pi 4

My Raspberry Pi 4 (8GB) model just came last week and after completing my RADIO (VMware's R&D Innovation Offsite) session recording, I wanted to setup my new rPI so I can start playing with it when I had some spare time. I also have the 4GB model but it was running quite hot as I was using the default case (do not recommend) and decided to put that aside for now. I ended up purchasing the 8GB model from Canakit which includes additional heatsinks and nice built-in fan with their custom case.

Look who just arrived to join the rest of 🥧 family! This will be my reward for tomorrow after I finish my #RADIO session recording pic.twitter.com/h9kxWRpM8S

— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) July 1, 2020

After spending some time setting up the rPI OS and applying the various updates, I was not too keen on using the SD Card, especially as some folks on forums mentioned that it can easily be worn out depending on the type of workload. While researching online and some references pointed out by colleagues, I came to learn that in addition to booting from USB which is now possible with rPI, you can also network boot the rPI without an SD Card or any storage for that matter! This immediately resonated with me, especially as I am a huge fan of scripted installations such as Kickstart/Jumpstart.

Here are all the resources that I had used that provided all the heavy lifting. I have optimized some of the commands to make it easy for anyone to simply copy and past who is new to setting up an rPI. The authors below did a fantastic job in outlining all the details, so I highly recommend a read if you would like to get more information. I also will be going over an alternative method at the end for booting the rPI over the network which is not really talked about.

• https://hackaday.com/2019/11/11/network-booting-the-pi-4/
• https://www.ferdinand-keil.com/network-booting-rpi4-from-centos7.html
• https://codestrian.com/index.php/2020/02/14/setting-up-a-pi-cluster-with-netboot/
• https://linuxhit.com/raspberry-pi-pxe-boot-netbooting-a-pi-4-without-an-sd-card/

One really cool thing that I came to learn while setting up the infrastructure to network boot an rPI was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. I may have spent more time playing with dnsmasq than with the rPI itself and I will probably cover this in another blog post on how you can easily setup a PXE/gPXE/iPXE system to enable automated OS installation (e.g. Kickstart) that can be used with ESXi or any other OS for that support network installations via BIOS/UEFI.

Preparing the rPI

Step 1 - Download and install the Raspberry Pi Imager Tool for your OS

Step 2 - Plugin an SD Card (If the card is not formatted, you can use the Imager Tool to format) and then flash it with the rPI OS. Insert the SD Card and power up the rPI

Step 3 - Once rPI OS has booted up, open a terminal to enable SSH access and retrieve the IP Address assigned by running the following commands:

systemctl start ssh
ip add show eth0

Step 4 - SSH to the rPI using username pi/raspberry and then run the following command to update and upgrade the rPI

sudo apt-get update
sudo apt-get upgrade

Step 5 - Run the following command to download and apply the latest rPI BETA EEPROM which supports network booting. We are also updating the default boot order to 0xf241 which will try SD Card , USB  and then Network Boot and then reboot to go through the boot order again.

PI_EEPROM_VERSION=pieeprom-2020-06-15
wget https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-config ${PI_EEPROM_VERSION}.bin > bootconf.txt
sed -i 's/BOOT_ORDER=.*/BOOT_ORDER=0xf241/g' bootconf.txt
sudo rpi-eeprom-config --out ${PI_EEPROM_VERSION}-netboot.bin --config bootconf.txt ${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-update -d -f ./${PI_EEPROM_VERSION}-netboot.bin

Step 6 - Lastly, we need to retrieve the rPI serial number and MAC Address which will be used

cat /proc/cpuinfo | grep Serial | awk -F ': ' '{print $2}' | tail -c 8
ip addr show eth0 | grep ether | awk '{print $2}'

Step 7 - Shutdown the rPI and remove the SD Card

Preparing the Kickstart/Netboot Server

Step 1 - Install an Ubuntu VM, I happen to have 16.04 lying around in my vSphere Datastore but you can use any recent version

Step 2 - Apply latest patches and install the following packages:

apt update
apt install -y nfs-kernel-server dnsmasq kpartx unzip

Step 3 - Download the latest rPI OS and run the following commands to prepare it so we can boot it over the network to our rPI:

wget -O raspbian_lite_latest.zip https://downloads.raspberrypi.org/raspbian_lite_latest
unzip raspbian_lite_latest.zip
kpartx -a -v *.img
mkdir {bootmnt,rootmnt}
mount /dev/mapper/loop0p1 bootmnt/
mount /dev/mapper/loop0p2 rootmnt/

Step 4 - Run the following command and replace it with your rPI serial (last 8 characters), MAC Address and the IP Address of your Ubuntu VM. This will create the necessary filesystem structure and copy the rPI OS from previous step to be able to boot the rPI over the network:

PI_SERIAL=abcdefgh
PI_MAC=dc:a6:ab:cd:ef:gh
KICKSTART_IP=192.168.30.176
mkdir -p /srv/nfs/rpi4-${PI_SERIAL}
mkdir -p /srv/tftpboot/${PI_SERIAL}
cp -a rootmnt/* /srv/nfs/rpi4-${PI_SERIAL}/
cp -a bootmnt/* /srv/nfs/rpi4-${PI_SERIAL}/boot/

Step 5 - We need to replace the default rPI firmware files with the latest version by running the following comamnds:

rm /srv/nfs/rpi4-${PI_SERIAL}/boot/start4.elf
rm /srv/nfs/rpi4-${PI_SERIAL}/boot/fixup4.dat
wget https://github.com/Hexxeh/rpi-firmware/raw/stable/start4.elf -P /srv/nfs/rpi4-${PI_SERIAL}/boot/
wget https://github.com/Hexxeh/rpi-firmware/raw/stable/fixup4.dat -P /srv/nfs/rpi4-${PI_SERIAL}/boot/

Step 6 - Next we setup the NFS export and dnsmasq configuration for the rPI to mount the images over the network. There are a number of dnsmasq configurations that you can tweak, but if you have an existing DHCP server and simply want to use it, simply specify your network broadcast address and "proxy" keyword as shown in the example below.

echo "/srv/nfs/rpi4-${PI_SERIAL}/boot /srv/tftpboot/${PI_SERIAL} none defaults,bind 0 0" >> /etc/fstab
echo "/srv/nfs/rpi4-${PI_SERIAL} *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
cat > /etc/dnsmasq.conf << EOF
dhcp-range=192.168.30.255,proxy
log-dhcp
enable-tftp
tftp-root=/srv/tftpboot
pxe-service=0,"Raspberry Pi Boot"
EOF
mount /srv/tftpboot/${PI_SERIAL}/

Step 7 - In addition to enable SSH and clearing the fstab file so it will not look for filesystem on the SD Card, the cmdline.txt is updated so rPI knows how to reach our NFS mount for serving the rPI OS image:

touch /srv/nfs/rpi4-${PI_SERIAL}/boot/ssh
sed -i /UUID/d /srv/nfs/rpi4-${PI_SERIAL}/etc/fstab
echo "console=serial0,115200 console=tty root=/dev/nfs nfsroot=${KICKSTART_IP}:/srv/nfs/rpi4-${PI_SERIAL},vers=3 rw ip=dhcp rootwait elevator=deadline" > /srv/nfs/rpi4-${PI_SERIAL}/boot/cmdline.txt

Step 8 - Enable and start the services

systemctl enable dnsmasq
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start dnsmasq
systemctl start rpcbind
systemctl start nfs-server

Step 9 - Finally, we are now ready to power on our rPI and you should see rPI OS boot over the network. You can also confirm things are working or debug by tailing the logs of /var/log/syslog on your Ubuntu system.

🥳 Just PXE Booted my rPI4!

Its a shame to enable PXE, you need to flash the EEPOM using rPI OS to change default which is to continuously try SDCARD (0xf34) pic.twitter.com/pAO4Nm02Ee

— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) July 5, 2020

Alternative Network Boot on rPI

Thanks to fellow VMware colleague Andrei Warkentin, I came to learn about another method to network boot the rPI with other OSes that support PXE/iPXE is by using UEFI on the rPI itself via the SD Card which would enable you to boot anything you need over UEFI. The instructions below are additive and assumes you have completed the steps above already.

If you use UEFI in the Pi that *itself* supports PXE and HTTP booting (and also - iSCSI). Keep UEFI on SD and network boot whatever you need via UEFI.

— Andrei Warkentin (@WhatAintInside) July 5, 2020

Step 1 - Download the latest rPI Firmware and extract the contents to your computer, you should have a folder called firmware-master

Step 2 - Download the latest rPI 4 UEFI firmware and extract the contents to your compute you should have a folder called RPi4_UEFI_Firmware_v1.14

Step 3 - Delete all files starting with kernel*.img within firmware-master/boot directory and then copy the entire "boot" directory onto the newly formatted SD Card

If you're on MacOS system, the following commands can be used to perform the operation. In the example below, the SD Card label is called UEFI

rm ~/Desktop/firmware-master/boot/kernel*.img
cp -rf ~/Desktop/firmware-master/boot/* /Volumes/UEFI

Step 4 - Copy all files within the RPi4_UEFI_Firmware_v1.16 directory into the same boot directory on SD Card

If you're on MacOS system, the following commands can be used to perform the operation. In the example below, the SD Card label is called UEFI

cp -rf ~/Desktop/RPi4_UEFI_Firmware_v1.16/* /Volumes/UEFI

Step 5 - Eject the SD Card and plug that into your rPI

Step 6 - Update the /etc/dnsmasq.conf to include the following. In my example below, I am booting the netboot image for Ubuntu 20.04 AARCH64, but you should be able to specify any OS that supports UEFI booting on the rPI.

dhcp-match=set:aarch64,60,PXEClient:Arch:00011:UNDI:003000
dhcp-boot=tag:aarch64,ubuntu20.04/boot/bootnetaa64.efi

Note: If you want to boot rPI OS over the network, you will need to use the first option outlined. rPI OS does not used UEFI and hence you will not be able to use the alternative approach

Step 7 - Power on your rPI and you should see UEFI boot from the SD Card (rainbow screen) and then shortly after, you should see an image get served from the Kickstart/Netboot server.