Playing with Zero Touch Provisioning (ZTP) in vSphere 9.1 using an ASUS NUC

Introduced in VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) 9.1, Zero Touch Provisioning (ZTP) is the next evolution of Auto Deploy. Like Auto Deploy, it automates both ESX installation and host configuration. Instead of relying on the traditional PXE/TFTP boot process and Host Profiles, ZTP uses modern UEFI HTTP Boot along with vSphere Configuration Profiles (VCP) to provision a fully configured ESX host that automatically registers itself into the desired vSphere Cluster.

While ESX has supported UEFI HTTP Boot for some time, adoption has been gradual as organizations continue to rely on existing PXE/TFTP-based deployment infrastructure. At the same time, the hardware ecosystem has steadily transitioned from legacy BIOS firmware to UEFI, making UEFI HTTP Boot more broadly available. Auto Deploy's iPXE implementation was a nice bridge to support traditional PXE/TFTP infrastructure while still getting the benefits of HTTP boot, especially when it comes to installing ESX over the network compared to a TFTP-based approach.

While cleaning out some old lab hardware, I came across a spare ASUS NUC 15 Pro that I was no longer using and wondered whether it supported UEFI HTTP Boot and could be repurposed for ZTP testing? 🤔

Fortunately, the ASUS NUC 15 Pro supports UEFI HTTP Boot, and several earlier generations do as well. Unfourntately for Minisforum MS-A2 owners, it does not look like these systems are capable of native UEFT HTTP Boot.

Requirements:

• Verify that your NUC or simliar system has support for UEFI HTTP Boot support
• Native VLAN networking support since NUC do NOT support configuring VLAN tagging, especially for those setup 802.1Q trunk for VCF networking

Note: If you do not have a physical host that is capable of HTTP boot, you can still play with ZTP using Nested ESX and leveraging VirtualEFI capability introduced in vSphere 8, which can be another great way to get hands experience without any hardware dependencies.

Step 1 - Download the VMware Certificate Authority (VMCA) root certificate (/var/lib/vmware/vmca/root.cer) from your vCenter Server, and then convert it to DER format using the following command:

openssl x509 -outform der -in root.cer -out root.der

Copy the root.der file to a USB storage device that has been formatted with a FAT filesystem. Plug the USB storage device into your NUC or other supported system, and then power it on.

Step 2 - Enter the system BIOS by pressing F2 key and then navigate to Boot tab and under Network Boot select HTTP boot.

Next, navigate to the Advanced tab and scroll down to Add-in Config setting to enroll our VMCA root certificate into the NUC and configure the ZTP boot URL.

Step 3 - Select the Tls Auth Configuration and then Enroll Cert Using File

You will be prompted to select a storage device, which will include onboard storage devices and the USB storage device that you connected earlier. The device label and capacity are shown to help you identify the correct device. Select the USB storage device until you see the contents of the drive, including the root.der file and then user arrow keys to hit OK button.

You should now be taken back to the certificate enrollment screen and see root.der file listed on main page, which means the previous step was completed successfully. To complete the enrollment, select Commit Changes and Exit.

If a valid certificate is provided, you will be able to complete the workflow. Otherwise, an error will be displayed, indicating that the certificate is not compatible or was not formatted correctly. I found the overall certificate enrollment workflow on the NUC to be less intuitive than the other configuration options.

Step 4 - Login to your vCenter Server and in the left-hand navigation, select Auto Deploy and enable the Auto Deploy and Image Builder service. You will need to select a vSphere Cluster that you wish to enable ZTP.

Next, retrieve the ZTP Boot URL which will have the following format: https://FQDN/vmw/brs/vbmx64.efi located under the Configure tab

Step 5 - To configure the NUC to boot from ZTP Boot URL, head back to Add-in Config menu and select the network adapter that ends with HTTP Boot Configuration. Enter a description which will show up as a boot menu option and then enter the ZTP Boot URL from previous step.

Note: While vCenter Server provides the ZTP Boot URL using just the IP Address of the vCenter Server, you can use either IP or FQDN, the latter is what I ended up testing with.

At this point, you have completed all the required configurations on the NUC, go ahead and save your settings and exit to reboot. Use the F10 key to access the boot menu and you should now see your new boot description as one of the options to boot.

If everything has been configured correctly, and your NUC has obtained a DHCP address and can reach the ZTP Boot URL, you should see the NUC contact the Boot Routing Service (BRS) on your vCenter Server.

Shortly after, you should see the NUC is now booting an ESX image that is being served over HTTPS from our vCenter Server.

⚠️ Once ESX begins booting on the NUC, you will encounter an expected PSOD due to Intel Consumer Hybrid CPUs, which use non-uniform CPU cores, which ESX does not expect.

Step 6 (Applicable only to consumer Intel non-uniform processors) While there is a workaround for the PSOD, we do require the ESX host to attempt the initial boot as it will create the required boot.cfg file within vCenter Server, which will allow us to apply the workaround.

SSH to vCenter Server and look for a new UUID for the booted ESX host under /storage/updatemgr/nbs/bootbanks/ directory and traverse into the next UUID directory until you see image-0 folder that contains the ESX boot.cfg configuration file which we will need to edit.

For my setup, the boot.cfg configuration file is located at /storage/updatemgr/nbs/bootbanks/521f6cfb-1ea8-5ef2-1ff5-b1773fb4cbf6/52a253fc-68de-f90e-557b-5654e9ad5d01/image-0/boot.cfg and we just need to edit the file and append cpuUniformityHardCheckPanic=FALSE on the kernelopt line (leave the weasel string and make sure to add space). Once the changes have been saved, we can now reboot the ESX host and it should now successfully boot with a screen like the following, since we have not created any deployment rules.

We can also see our booted ESX host under the Discovered Hosts tab in vCenter Server, which you can create deployment rules using any of the host attributes.

Step 7 - Create a deployment rule by navigating over to the Deploy Rules tab based on your requirement. Make sure to activate your rule and when the host reboots, it will now perform an ESX installation and apply the configurations based on your deployment rule including adding it to vCenter Server Inventory.

⚠️ After ESX successfully installs and the NUC reboots, you will encounter another expected PSOD due to Intel Consumer Hybrid CPUs, which use non-uniform CPU cores, which ESX does not expect. The reason for the second occurrence is that boot.cfg override only applies to the initial boot of ESX installer, it does not persist post-installation and needs to be reapplied directly on the ESX host by appending the workaround by hitting SHIFT+O. You can refer to this blog post that contains an instructional video if you have never done this before.

If you have direct console access to your ESX host, you might be able to catch the reboot message that is displayed after ESX has successfully been installed, which will allow you to apply the change as part of the reboot after installation and you may not run into the PSOD issue. Once the ESX host has successfully booted, you should persist the configuration by setting the required kernel advanced setting as outlined in the blog post so you do not have to do this again.

Finally, once the ESX host has fully booted, it will retrieve its inventory information from the vCenter Server and automatically join the configured vSphere Cluster.