Deploying an ESXi scripted installation aka Kickstart running within a VM (Nested ESXi) has a number of benefits, especially for testing and development purposes. This was something I did regularly as a customer, especially with new releases of ESXi to ensure our existing automation scripts and processes continued to work before rolling out into production. ESXi kickstart itself is pretty straight forward, but the required supporting infrastructure (PXE Server, DHCP, TFTP, etc) that needs to be configured, especially for a greenfield deployment can often be challenging for new comers.
Even with an existing PXE infrastructure, it can often be difficult to configure or troubleshoot depending on your level of access which does not add any value in actually testing or automating the ESXi scripted installation process. In ESXi 7.0 Update 2, an enhancement was made to the Virtual Machine's UEFI firmware called VirtualEFI that would enable ESXi to perform an HTTP Boot given the ESXi bootloader URL and without requiring any of the traditional PXE infrastructure.
To take advantage of this new capability, you just need to have a physical server running ESXi 7.0 Update 2 and a VM that is configured with the latest vHW19 compatibility. To configure HTTP boot, you will need to add the following two VM Advanced Settings:
- networkBootProtocol - httpv4 or httpv6
- networkBootUri - HTTP URL to the ESXi bootloader (bootx64.efi)
Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware
Step 1 - Download and extract the desired ESXi ISO to your HTTP server. In my example, I am using ESXi 7.0 Update 1c and have extracted the contents into a directory called esxi70u1c and placed that into the root of my HTTP server which is /etc/httpd/html
Step 2 - You will need to update the ESXi boot.cfg configuration which is located in efi/boot directory with the following:
- Remove the leading "/" character in front of all module names
- Update the prefix parameter with the base URL of the extracted ESXi image
Here is what my /etc/httpd/html/esxi70u1c/efi/boot/boot.cfg looks like after applying the two changes mentioned above:
bootstate=0
title=Loading ESXi installer
timeout=5
prefix=http://192.168.30.6/esxi70u1c
kernel=b.b00
kernelopt=cdromBoot runweasel
modules=jumpstrt.gz --- useropts.gz --- features.gz --- k.b00 --- uc_intel.b00 --- uc_amd.b00 --- uc_hygon.b00 --- procfs.b00 --- vmx.v00 --- vim.v00 --- tpm.v00 --- sb.v00 --- s.v00 --- bnxtnet.v00 --- bnxtroce.v00 --- brcmfcoe.v00 --- brcmnvme.v00 --- elxiscsi.v00 --- elxnet.v00 --- i40en.v00 --- i40iwn.v00 --- iavmd.v00 --- icen.v00 --- igbn.v00 --- iser.v00 --- ixgben.v00 --- lpfc.v00 --- lpnic.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsi_msgp.v01 --- lsi_msgp.v02 --- mtip32xx.v00 --- ne1000.v00 --- nenic.v00 --- nfnic.v00 --- nhpsa.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nmlx5_co.v00 --- nmlx5_rd.v00 --- ntg3.v00 --- nvme_pci.v00 --- nvmerdma.v00 --- nvmxnet3.v00 --- nvmxnet3.v01 --- pvscsi.v00 --- qcnic.v00 --- qedentv.v00 --- qedrntv.v00 --- qfle3.v00 --- qfle3f.v00 --- qfle3i.v00 --- qflge.v00 --- rste.v00 --- sfvmk.v00 --- smartpqi.v00 --- vmkata.v00 --- vmkfcoe.v00 --- vmkusb.v00 --- vmw_ahci.v00 --- crx.v00 --- elx_esx_.v00 --- btldr.v00 --- esx_dvfi.v00 --- esx_ui.v00 --- esxupdt.v00 --- tpmesxup.v00 --- weaselin.v00 --- loadesx.v00 --- lsuv2_hp.v00 --- lsuv2_in.v00 --- lsuv2_ls.v00 --- lsuv2_nv.v00 --- lsuv2_oe.v00 --- lsuv2_oe.v01 --- lsuv2_oe.v02 --- lsuv2_sm.v00 --- native_m.v00 --- qlnative.v00 --- vdfs.v00 --- vmware_e.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- tools.t00 --- xorg.v00 --- gc.v00 --- imgdb.tgz --- imgpayld.tgz
build=7.0.1-0.25.17325551
updated=0
Note: Make sure the extracted ESXi directory has the correct permissions. I simply used chmod 655 -R esxi70u1c/ to recursively apply the change for all files/directories.
Step 3 - Create new vHW19 Nested ESXi VM and configure the two required VM Advanced Settings which specifies the boot protocol (IPv4/IPv6) and the ESXi bootloader URL:
- networkBootProtocol = httpv4
- networkBootUri = http://192.168.30.6/esxi70uc1/efi/boot/bootx64.efi
Step 4 - Power on the VM and you should see the VM attempt EFI Network boot
and if everything was setup correctly, we should the ESXi image booting over the network from our HTTP server. In my example above, I did not specify kickstart location, so it will just boot into the interactive installer. To do so, just update the kernelopt line to reference the KS URL as described in Step 6 of the official VMware documentation.
After all these years working with ESXi kickstart, I never thought this could be simplified even further when running inside of a VM form factor, huge props to the VMware Engineers who challenged the status quo and provided an even better user experience when it comes to automating Nested ESXi deployments!
How can, I do the same on Windows Server as HTTP?
I will extract the files to a folder on Windows Server 2019 and then try this method for http server - https://ryanblunden.medium.com/create-a-http-server-with-one-command-thanks-to-python-29fcfdcd240e
This should work for any OS bootloader that supports HTTP boot, I'm not familiar enough with Windows bootloader on whether this is possible or not. Again, this optimization was for running Nested ESXi and its behaviors may or may not work for other OSes. Let me know how your testing goes
A welcome improvement, but I still feel this is a lot easier in Microsoft land. I’d like to see the ability to make a kickstart available on removable media and for the installer to automatically pick it up (kind of like how Windows works). This would remove the requirement to fiddle around making custom ISOs. Or how about some PowerCLI CMDlets that would let me bake my own ISO with custom kickstart. I can do this natively with PowerShell for Windows, but for ESXi we often need to end up calling out to CLI tooling like mkisofs or Ultraiso.
If you need to "embed" a Kickstart file, you don't *need* to author a brand new ISO. Simply create the bootable media with the existing ISO AND then update the BOOT.CFG file afterwards to point to local KS.CFG which you would include. See https://www.williamlam.com/2019/07/automated-esxi-installation-to-usb-using-kickstart.html for more details. You can certainly create a new ISO depending on frequency, but at the end of the day it's using any of the ISO-based tools that allow you to construct a new ISO with some updated files.
It tried to use Windows Server for HTTP. Unfortunately it is not working. I am getting this error - Failed - A specified parameter was not correct: config.extraConfig["networkBootProtocol "]
When, I try to add configuration parameters - it is not accepting networkBootProtocol configuration which is http://192.168.0.2/esx/efi/boot/x64boot.efi - I did extract the ISO to esx folder and then modified boot.cfg file
Few things to check which is mentioned in blog post
1) Make sure you can manually download this file and there isn't a permission issue.
2) Make sure you've got vSphere 7.0 Update 2 running (both VC/ESXi)
3) Make sure you've got vHW19 VM and its configured for EFI
I have uploaded the document, I created to test this functionality - unfortunately it is not working for me. Appreciate, if anybody can take a look and see why VM is not accepting above configuration parameter.
https://easyupload.io/oglqdb
It looks like you have started your python http server on port 8000 but your boot.cfg file is using port 80. You are also serving the files in the root of your folder, not a sub folder called esx
You need to update your boot.cfg file to use http://192.168.0.2:8000
Your VM config needs to use http://192.168.0.2:8000/efi/boot/x64boot.efi
Make sure to allow port 8000 through your Windows firewall
Your fix resolved VM creation. I even tested to make sure, I can download boot.cfg from 192.168.0.2:8000 through web browser. The file is there but when, I power on the VM, I can see the VM is able to reach 192.168.0.2 but python http server is giving code 404 message file not found - Head /efi/boot/x64boot.efi
The VM config should be http://192.168.0.2:8000/efi/boot/bootx64.efi. When I power on the vm EFI starts and python http server reports HEAD /efi/boot/bootx64.efi http/1.1 200. Still the esxi installer won't start. Not sure where is the mistake?
Screen shot - https://easyupload.io/zqss5e
2. I am running 70u2 on esxi and vc
3. Yes VM created with hw19 and boot option is set to EFI and unchecked UEFI
1. The esx folder permission within Windows is set to FULL access
Hi, in my .iso the efi folder consist of
-rw-r-xr-x. 1 root root 1564 Mar 27 13:22 BOOT.CFG
-rw-r-xr-x. 1 root root 167360 Dec 15 07:45 BOOTX64.EFI
-rw-r-xr-x. 1 root root 98896 Dec 15 07:45 SAFEBOOT.EFI
are you sure that the file is x64boot.efi and not BOOTX64.EFI
I am using the same .iso , Thanks William
You're correct, what happens when you go off memory on writing the blog post 🙂
Perhaps, I'm missing something, but what makes this more significant as opposed to just sticking to an existing working PXE environment? I've built a well-oiled machine, should I consider adding this to it? Is the Kickstart file different? or can I just use the same Kickstart on my Cobbler PXE server?
As mentioned in the blog post, not everyone has an existing PXE infra 🙂
There's still many folks who are just getting started and/or for Dev/Test purposes, you don't need a PXE infra or easily influence changes. If you have an existing PXE Infra, this won't change anything. However, if you do not, you can then get the benefits of PXE Infra for Nested ESXi w/o relying on an actual PXE Infra. Hope that makes sense
Thank you for this great info William, does anybody know by any chance, how to configure the Advanced Settings for a stateful install when working in a nested environment? I tried a lot of different settings, but it keeps using stateless... I'm running out of idea :/
On bare-metal I've had success installing ESXi via UEFI HTTP boot of the unmodified ISO. I can specify the URI in BIOS or I can hand out the URI via DHCP option 67. I didn't need to modify anything within the ISO. Could this also be made to work within a VM using VirtualEFI?
So this just creates an ESXI host with out any other configuration like Cluster, distributed switch ect, like the powershell script does? Can i use that script and deploy esxi 7 like it does for 6.5 and 6.7?