WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple

How to prompt for user input during an interactive or scripted install of ESXi?

10.28.2015 by William Lam // 24 Comments

A question that continues to pop up from time to time is whether it is possible to prompt for user input during an interactive or scripted installation of ESXi? This is actually something I have written about before using PXE boot options as a workaround to provide for a semi-interactive automated installation of ESXi. The most recent request for this was not actually from a customer but rather someone internally working at VMware. The individual noted that he had already read my blog and a few other references but was still hopeful for a solution. I remember when I had looked into this problem several years back, I was not able to find anything and the inquiries to VMware (which I was still a customer at the time) came up empty.

UPDATE (01/10/19) - For ESXi 6.5 or greater, please take a look at this blog post for an updated solution

After replying back to the individual with the information that I had, it actually got me thinking which is usually not good 😉 Having just recently finished building a new Kickstart environment to test UEFI PXE boot for ESXi 6.0, I figure I might as well take another look at this topic again. I wanted to see if there was something that could be done with one of the /dev/ttyl (teletype) interfaces while the ESXi Installation was running. I did a couple of Google searches and to my surprise, there was someone on the VMTN Community forum who had already solved this problem and posted a solution almost 1 year before my article, not sure how I could have missed that?!

In ESXi, there are only two TTY interfaces: TTY1 (/dev/tty1) and TTY2 (/dev/tty2) and during the installation of ESXi, /dev/tty2 is used by the interactive installer and /dev/tty1 is used by the ESXi Shell login for troubleshooting/debugging purposes. If you try to overlay another menu system on top of the existing one, what you will find is that although it is possible to prompt for user input, the user keystrokes would not be properly being sent due to having two menus as mentioned by VMTN Community user mossko. The clever solution that was identified by cacheman was to simply disable the console login and free up TTY1 which could then be used to present a custom menu and prompt for user input. This was accomplished by creating a custom TGZ file called "extras.tgz" (you can name it anything you want) which would contain a modified copy of the /etc/inittab file which has the following line commented out:

#tty1::respawn:/bin/initterm.sh tty1 /bin/techsupport.sh

You would then append this TGZ to the ESXi's boot.cfg file which would then overwrite the existing inittab and prevent ESXi Shell from running on TTY1. I thought this was very cool way of solving this problem, nice work and thanks for sharing cacheman!

I wanted to verify that this solution still works with the latest release of ESXi 6.0 Update 1 and luckily it still does and here is a sample menu prompt that I created which prompts users for the networking settings for the ESXi host which will then be used by the ESXi Kickstart script. This is just one example of user prompts that can be created, this is only limited by your imagination and the amount of input you wish to put your users through 😉 If you are interested in the ASCII art, go to the very bottom for the full details.

prompt-user-input-during-interactive-or-scripted-install-of-esxi
Here are the exact instructions along with the sample menu that was used in the screenshot above.

Step 1 - Create a temporary directory which will be used to store our modified inittab file by running the following command:

mkdir -p temp/etc

Step 2 - Change into the "temp" directory

Step 3 - Copy the /etc/inittab file into the etc directory and you should have the following: temp/etc/inittab

Step 3 - Edit the etc/inittab file and comment out the following line:

#tty1::respawn:/bin/initterm.sh tty1 /bin/techsupport.sh

Step 4 - Next, we need to create a TGZ file of the etc directory by running the following command:

tar -czvf extras.tgz etc

Step 5 - Copy the extras.tgz to root of your ESXi installation if you are PXE booting or if you are re-authoring a new ESXi ISO

Step 6 - Edit the ESXi's boot.cfg configuration file and append the following after imgpayld.tgz:

--- extras.tgz

Step 7 - Finally, you will add your custom menu prompt into the %pre section of an ESXi Kickstart file. Below is an example Kickstart that I used to build the menu shown in the screenshot above.

accepteula
install --firstdisk --overwritevmfs
rootpw vmware123
reboot

%include /tmp/networkconfig

%pre --interpreter=busybox

exec < /dev/tty1 > /dev/tty1 2>&1
chvt 1
HOSTNAME=""
IPADDR=""
NETMASK=""
GATEWAY=""
DNS=""

while [[ "$HOSTNAME" == "" ]] || [[ "${IPADDR}" == "" ]] || [[ "${NETMASK}" == "" ]] || [[ "${GATEWAY}" == "" ]] || [[ "${DNS}" == "" ]] ; do
 echo
 echo " *** Please enter the following details: *** "
 echo
 read -p "Hostname: " HOSTNAME
 read -p "IP Address: " IPADDR
 read -p "Netmask: " NETMASK
 read -p "Gateway: " GATEWAY
 read -p "DNS: " DNS
done
clear

echo "network --bootproto=static --addvmportgroup=true --device=vmnic0 --ip=${IPADDR} --netmask=${NETMASK} --gateway=${GATEWAY} --nameserver=${DNS} --hostname=${HOSTNAME}" > /tmp/networkconfig

echo -e "Applying the following configuration: \n"
echo "Hostname = ${HOSTNAME}"
echo "IP Address = ${IPADDR}"
echo "Netmask = ${NETMASK}"
echo "Gateway = ${GATEWAY}"
echo -e "DNS = ${DNS}\n"
sleep 5
chvt 2

%firstboot --interpreter=busybox

sed -i 's/#tty1/tty1/g' /etc/inittab
/sbin/auto-backup.sh

vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

Lines 10-11 - Redirects the output to /dev/ttyl and changes to tty1 virtual console.

Lines 12-16 - Define a couple of empty string variables that will be used to store user input.

Lines 18-28 - Prompt user input and loop until all variables are not empty.

Lines 30 - Using the user input, generate the networking configuration for the ESXi Kickstart.

Line 32-38 - Display the user input to the user and sleep for 5 seconds before switching back to the ESXi installer.

Line 39 - Change to /dev/tty2 virtual console which will show the regular ESXi installer menu.

Line 43-44 - Modify the /etc/inittab to re-enable ESXi Shell for /dev/tty1. For the changes to go into effect, you will need reboot and you can either do this when you need the capability (which is not ideal) or you can just automatically issue a second reboot in the %firstboot section of the Kickstart, so the ESXi Shell is available when you need it.

For those interested in how I added the ASCII art, I used this site here to generate the text. Once you are happy with the text and how it is displayed, which may require several rounds of testing, you will then want to save it to a file. In my setup I just called it vg-ascii.cfg and this should be placed on either your HTTP Server where the Kickstart is being pulled or embedded into the ESXi ISO if you are doing local installs only. You will then need to add the following lines to the "pre" section of the Kickstart right before the loading of the menu prompt. In this example, I am downloading the ASCII file from my web server which is also serving my kickstart and then clearing the screen before iterating through the file to display the text.

wget http://192.168.1.180/esxi60u1/vg-ascii.cfg -O /tmp/vg-ascii.cfg
clear
IFS=$'\n';for i in $(cat /tmp/vg-ascii.cfg);do echo $i;done

Categories // Automation, ESXi Tags // /dev/tty, boot.cfg, esxi, inittab, kickstart, ks.cfg, tty1, tty2

How to create custom ESXi boot menu to support multiple Kickstart files?

06.11.2015 by William Lam // 27 Comments

I recently received a question from one of my readers who was looking to migrate from ESXi 4.1 to newer version and one of the challenges they faced was around their ESXi scripted installs, better known as ESXi Kickstart. Previously, they had relied on using a custom syslinux boot menu to be able to select a specific Kickstart configuration file that resided locally on a bootable ESXi Image (USB, ISO or CDROM) as a PXE/DHCP environment was not allowed in their environment. There was a small change to how ESXi boot files were reference between ESXi 4.x and ESXi 5.x/6.x and a new boot.cfg configuration is now used which I had written about here with respect to scripted installs when ESXi 5.0 was first released.

Luckily, even with these changes one can still use a custom menu with ESXi 5.x/6.x and be able to select a specific Kickstart configurations based on user input. Here is a screenshot example of a custom ESXi Image that I built providing three different install options that could be selected which would map to three different Kickstart configurations which can be either local to the boot media or can also be retrieved remotely.

bootable-esxi-image-with-multiple-kickstart-option
The first thing you should be aware of if you plan to boot the custom ESXi Image from local media such as USB, CDROM or ISO is that the path to the Kickstart file must be in all UPPER CASE which is mentioned in this VMware KB 1026373. The next caveat that I found in my testing is that if you plan to store the local Kickstart files inside of a directory within the ESXi Image, the name of the directory can not be too long. I would recommend using "ks" as "kickstart" apparently was too long.

After you have extracted the contents of an ESXi ISO which you have downloaded, you will want to create a root directory called "ks" which will contain the different Kickstart configuration files. Here is an example of what structure look like:

ks
├── ks1.cfg
├── ks2.cfg
└── ks3.cfg

Next, you will need to edit the isolinux.cfg file which comes by default within the ESXi ISO. This is where you will add the different Kickstart options that a user will be able to select from. In this first example, we will look at referencing the Kickstart files locally on the media which can be either USB or CDROM and you will need to ensure you specify the right boot option as shown here in the VMware documentation. The path to the Kickstart file needs to be appended to the line that contains boot.cfg reference and you must ensure you include "+++" at the end of that line.

Here is an example of referencing a Kickstart file that lives on a USB device under this path /ks/ks.cfg:

APPEND -c boot.cfg ks=usb:/KS/KS.CFG +++

Here is an example of my isolinux.cfg for the boot menu that I have shown above which provides three different options mapping to three different Kickstart configuration files:

DEFAULT menu.c32
MENU TITLE vGhetto Custom ESXi 6.0 Boot Menu
NOHALT 1
PROMPT 0
TIMEOUT 80
LABEL Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS1.CFG +++
  MENU LABEL ^1 Ghetto Install
LABEL A bit More Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS2.CFG +++
  MENU LABEL ^2 A bit More Ghetto Install
LABEL Super Ghetto ESXi Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS3.CFG +++
  MENU LABEL ^3 Super Ghetto ESXi Install
LABEL hddboot
  LOCALBOOT 0x80
  MENU LABEL ^Boot from local disk

As I mentioned earlier, the Kickstart configuration file can either be retrieved locally or it can also be retireved remotely using one of the following supported protocols: http, https, ftp & nfs as shown here in the VMware documentation.

Here is an example of isolinux.cfg for a boot menu which references both a local kickstart as well as one that remotely lives on a web server:

DEFAULT menu.c32
MENU TITLE vGhetto Custom ESXi 6.0 Boot Menu
NOHALT 1
PROMPT 0
TIMEOUT 80
LABEL Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS1.CFG +++
  MENU LABEL ^1 Ghetto Install
LABEL A bit More Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=http://172.30.0.108/ks/ks2.cfg +++
  MENU LABEL ^2 A bit More Ghetto Install
LABEL Super Ghetto ESXi Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=http://172.30.0.108/ks/ks3.cfg +++
  MENU LABEL ^3 Super Ghetto ESXi Install
LABEL hddboot
  LOCALBOOT 0x80
  MENU LABEL ^Boot from local disk

For additional ESXi Kickstart resources and example, be sure to check out my pages here.

Categories // Automation, ESXi, vSphere 5.5, vSphere 6.0 Tags // boot.cfg, esxi, esxi 5, esxi 5.5, esxi 6.0, kickstart, ks.cfg, pxelinux

How to automate VM deployment from large USB keys using ESXi Kickstart?

10.08.2014 by William Lam // 8 Comments

During VMworld US, I had the opportunity to speak with several customers to learn about their VMware environment and some of the challenges they were facing. In some scenarios, I was able to offer a solution or a different way of solving the problem. For others, it was primarily feedback on how we can better improve some of our capabilities/features or specific feature requests they would like to see get added.

One interesting challenge that arose from a class of customers who manages hundreds of remote sites is the ability fully automate the provisioning of an ESXi host as well as set of Virtual Machines as part of the initial deployment. The provisioning is all done through Kickstart (unattended installation of ESXi) and usually from a USB device but it could also be from a custom ISO. One ask that kept coming up was the support for larger USB key support within ESXi so that it could be used to include additional payload.

As some of you may or may not know, ESXi can only access USB devices within the ESXi Shell formatted using the FAT16 filesystem which allows for a maximum file size of 2GB for each partition. However, this limitation is only for the ESXi Shell itself and for the size of the ESXi installation media, this is more than sufficient. If you wish to leverage larger USB keys which has increased significantly in recent years from 32GB, 64GB and even 128GB, you can directly pass that into any guest OS through the USB Arbitrator Service (enabled by default) and there you will be able to consume the entire capacity of the USB device. The challenge is how do you go about bootstrapping ESXi as well as the initial set of Virtual Machines with these limitations and completely automated using an ESXi Kickstart?

Over the years I have seen some really creative solutions to solving this problem and funny enough, right before VMworld I had several folks reach out asking similar questions. I decided to take a look and also build upon some earlier work done by a fellow VMware SE (Tim S) to come up with a completely automated solution that would scale to any size USB device and hopefully make it easy to extend if needed.

For this project, I used a 64GB USB key which I received from the folks over at Micron who I visited in the Solution Exchange during VMworld US (these guys are doing some really awesome stuff with VSAN and an All-Flash array, be sure to check them out).

automate-esxi-kickstart-and-servicevm-usb-3
Here is a diagram of the partition structure for the 64GB USB key which I will explain further:

esxi-usb-partition
The first partition is 2GB using a FAT16 filesystem and this is used to store the actual ESXi media along with an embedded ESXi Kickstart configuration file. You can easily reference a remote Kickstart if you wish, but for simplicity purposes and to support some of the requested use cases from customers, I have embedded it.

The second partition is also 2GB using a FAT16 filesystem and this is used to store a tiny VM which I am calling a "Service VM". This VM needs to be small enough to fit the partition and will be used to read the remainder capacity of the USB device which will be using a more capable filesystem type. I have decided to store a pre-configure vMA appliance which is tarred up to reduce the disk footprint.

The third and final partition will consume the remainder capacity of the USB device, in this case it would be 60GB and using a FAT32 partition which can support up to 2TB for a single volume. This is where additional Virtual Machines would be stored and accessed by the "Service VM".

As you can probably guess, the idea is to install ESXi as you normally would to a local disk or directly onto the USB device in which case an additional partition would be required. As part of the installation, the "Service VM" would be boot strapped as it would be visible within the ESXi Shell and registered and powered on during first bootup. A first boot script could then be included in the guestOS which can receive some details about the ESXi deployment which could be hard coded (not recommended) or dynamically discovered as I have implemented it. The USB device would then be passed directly to this "Service VM" to mount and then it would be able to deploy the remainder Virtual Machines which would be stored in this larger partition.

Here is the complete ESXi Kickstart which implements what has been discussed so far and I have also included a break down of the kickstart below:

vmaccepteula
install --firstdisk --overwritevmfs
rootpw vmware123
reboot

network --bootproto=static --ip=192.168.1.200 --netmask=255.255.255.0 --gateway=192.168.1.1 --hostname=mini.primp-industries.com --nameserver=192.168.1.1 --addvmportgroup=1

%post --interpreter=busybox

# stop USB Arbitrator service to access USB device in ESXi Shell
/etc/init.d/usbarbitrator stop

# copy service VM to local VMFS datastore
cp /vmfs/volumes/SERVICEVM/vMA.tar.gz /vmfs/volumes/datastore1
tar -zvxC /vmfs/volumes/datastore1 -f /vmfs/volumes/datastore1/vMA.tar.gz
rm -f /vmfs/volumes/datastore1/vMA.tar.gz

# add guestinfo property for ESXi IP Address for adv. VM deployment
ESXI_IP=$(localcli network ip interface ipv4 get | grep vmk0 | awk '{print $2}')
echo "guestinfo.esxi_ip = ${ESXI_IP}" >> /vmfs/volumes/datastore1/vMA/vMA.vmx

%firstboot --interpreter=busybox

# Ensure hostd is ready
while ! vim-cmd hostsvc/runtimeinfo; do
sleep 10
done

# enable & start SSH
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

# Suppress ESXi Shell warning
esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

# rename datastore1
vim-cmd hostsvc/datastore/rename datastore1 mini-local-datastore-1

# Register VM
vim-cmd solo/registervm /vmfs/volumes/mini-local-datastore-1/vMA/vMA.vmx

# connect USB device via passthrough
USB_DEV_NAME="Alcor Micro Corp"
USB_DEV_BUSID=$(lsusb | grep "${USB_DEV_NAME}" | awk '{print $2}' | cut -c 2)
vim-cmd vmsvc/device.connusbdev 1 "path:${USB_DEV_BUSID}/0/1 version:2"

# power on VM
vim-cmd vmsvc/power.on 1

Line12 - Need to disable the USB Arbitrator Service so the USB device can be seen by ESXi since it is by default made ready to be exposed to a VM. The service will be automatically re-enabled after the installation of ESXi which will allow for the VM to connect to the USB device.

Line15-17 - Copy the "Service VM" from USB device to local VMFS datastore1. In the example, I have pre-configured the vMA appliance tarred up the VMX and its respective VMDK.

Line20-21 - Extract the ESXi IP Address and sets a custom guestInfo property so the "Service VM" knows where to deploy the additional VMs to

Line26-29 - This checks to ensure hostd is up and running before continuing on

Line45 - Register the "Service VM" within ESXi

Line49-50 - Identify the USB device ID which will be required to mount to the "Service VM". You will need to update USB_DEV_NAME based on the USB device you are using

Line51 - Connect the USB Device to "Service VM"

Line54 - Power on the "Service VM"

At this point, you should be able to access the USB device from within the "Service VM". We can easily verify this by running the following command:

sudo fdisk -l

automate-esxi-kickstart-and-servicevm-usb-0
As seen in the screenshot above, we can see our three partitions and third is the one with our FAT32 partition which contains a couple of Virtual Machines that I want to deploy. Of course, this partition can contain anything you wish to store, so the sky is the limit!

To mount the USB device and the specific partition, we will create a temporarily directory and issue the mount command by running these two commands:

sudo mkdir -p /mnt/USB;sudo mount /dev/sdb3 /mnt/USB

automate-esxi-kickstart-and-servicevm-usb-1
For my USB key, I have stored both the VCSA and NSX Manager OVA which can then be deployed using ovftool. The last part to be able to make this as seamless and automated as possible is to be able to identify the ESXi host information. If you recall earlier, we had set a custom guestInfo property within our "Service VM". This custom property can then be read by the guestOS leveraging VMware Tools and provides the IP Address to the guest. You can easily set other metadata information but to be able to deploy these additional OVA's, we would need to know the IP Address of the ESXi host and this makes it so you do not need to hard code anything (perhaps ESXi host credentials).

To retrieve this custom property, you will need to run the following command:

vmtoolsd --cmd "info-get guestinfo.esxi_ip"

automate-esxi-kickstart-and-servicevm-usb-2
With these last few guestOS commands, you will be able to create a firstboot script which will automatically mount the appropriate USB partition and deploy these additional Virtual Machines. This is just one of the many possibilities on how you can deploy additional VMs as part of your ESXi Kickstart deployment. Hopefully this solution provides a base in which you can easily customize based on your own requirements.

Categories // Automation, ESXi, vSphere Tags // esxi, fat16, fat32, kickstart, ks.cfg, sd, usb, vSphere

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • Next Page »

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Blocking vSphere HTML5 VM Console and allowing only Standalone VM Remote Console (VMRC)? 02/08/2023
  • Quick Tip - Inventory core count for vSphere+, vSAN+ & VCF+ Cloud Service 02/07/2023
  • How to automate adding a license into vCenter Server with custom label?  02/06/2023
  • Automated ESXi Installation with a USB Network Adapter using Kickstart 02/01/2023
  • How to bootstrap ESXi compute only node and connect to vSAN HCI Mesh? 01/31/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...