ks.cfg – Page 4

Automating ESXi 5 Kickstart Tips & Tricks

07.13.2011 by William Lam // 58 Comments

There are some minor changes with kickstarting ESXi 5.0 but the majority of your existing ESXi 4.1 kickstart configurations can be re-used with a few modifications. One of my goals during the vSphere 5.0 beta was to automate as much of the configurations of an ESXi host as possible within the kickstart process. I also converted as many of the legacy esxcfg-* commands as I could over to the enhanced esxcli namespaces as the esxcfg-* commands will eventually be deprecated in favor of esxcli. Hopefully the tips & tricks and the example kickstart configuration file will be useful in aiding the transition to ESXi 5.0.

As usual, before diving in and creating an ESXi 5.0 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide. If you would like to get a sense for what a ESXi 5.0 kickstart could look like, please jump to the bottom of this post to get a complete working example displaying the various types of configurations.

Tip #1

If you want to have your ESXi pxelinux configuration boot up to a kickstart configuration file, you have two methods of specifying this with ESXi 5.0.

The first is using the defualt installer method which specifies a new boot.cfg configuration file to contain all the boot parameters.

Here is an example of what the pxelinux default configuration file would look like:

Here is an example of what the boot.cfg looks like:

As you can see the default boot.cfg that is included in the ESXi 5.0 installer in the same directory as all the modules needed to boot up ESXi.

The second method is specifying the kickstart configuration file in the actual pxeboot file versus in a seperate boot.cfg file. This is similar to the old method of doing things, but you will have to also include all the entries that are in the boot.cfg if you decide to go down this route.

Here is an example of specifying ks.cfg in pxelinux default file:

Notice the "pxebooting ks=" stanza used to specify the ks.cfg configuration file and list of modules in boot.cfg. If you need to append additional parameters such as "IPAPPEND 1", you will need to add "+++" (three plus characters) at the very end of the module list separated with a new line for your additional entries. If you forget the "+++" symbols, you will not be able to successfully boot up the installer and an error will be thrown.

The default method of relying on the boot.cfg is the recommended approach. You can still append custom kernel parameters such as "IPAPPEND 1" which will still be in the pxelinux file, but your kickstart entry will now be part of the boot.cfg configuration file. Below is an example of the above configuration but leveraging the additional boot.cfg configuration file.

Here is an example of what the pxelinux file should look like:

As you can see, we still need to specify the "+++" to include additional parameters, but the pxelinux file is much cleaner now.

Here is an example of what the boot.cfg should look like:

As you can see, all we needed to do is append the following line "kernelopt=ks=http://172.30.0.108/esxi5/ks.cfg" which specifies the source to the kickstart configuration file.

Tip #2

This is not really a new tip, but definitely make use of the "dryrun" mode in the ks.cfg and reviewing the esxi_install.log and hostd.log for any errors. The logs will only persist upon the first reboot, if you specify a secondary reboot for whatever reason, these logs will be lost. If you would like to automate the copying of the installation logs to help debug/troubleshoot, you can copy them to a local VMFS. Here is a quick snippet that can be used (This assumes your local VMFS will have the name with the following format $(hostname -s)-local-storage-1):

cp /var/log/hostd.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-hostd.log"
cp /var/log/esxi_install.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-esxi_install.log"

Tip #3

VMware has once again and hopefully for the last time, changed what was known as Busybox Console, Unsupported Mode, Tech Support Mode to ESXi Shell. In doing so, the old vim-cmd to enable both local (TSM) and remote (SSH) to ESXi Shell needs to be updated.

To enable local ESXi Shell (previously known as Tech Support Mode) you will need to run the following commands to enable and start ESXi Shell.

vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

To enable remote ESXi Shell (previously known as TSM SSH) you will need to run the following commands to enable and start remote ESXi Shell for SSH support.

vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

Note: During the beta, there was actually an intermediate name for this which was known as ESX Shell but through our feedback, we had recommended ESXi Shell so that users do not confuse old classic ESX with ESXi and this may or may not change from the actual vim-cmd's

Tip #4

The --level XX that would specify when a %firstboot script would execute is no longer supported and deprecated in ESXi 5.0. By default all %firstboot script will automatically execute after all default ESXi start up scripts have finished running. The location of the %firstboot scripts are in /etc/rc.local.d/001.fireboot_001

Tip #5

You should be able leverage the new changes in esxcli to configure majority of your ESXi 5.0 hosts, the are various examples below in the example kickstart. There are still a few things that are currently not included in esxcli such as configuring NTP, host licensing, ESXi Shell, SSH, and a few others that rely on vim-cmd or some minor hacks. VMware recommends you start to get familiar with esxcli, as the old esxcfg-* will eventually be deprecated and removed in future vSphere releases and be completely replaced with esxcli.

Tip #6

You now have the ability to configure multiple syslog hosts with ESXi 5.0 but also the ability to control individual loggers such as for vmkernel, hostd, vpxa, fdm, etc. Here is an example of changing the default syslog rotation from 10 to 20 and specifying two syslog hosts:

esxcli system syslog config set --default-rotate 20 --loghost syslog1.primp-industries.com:514,syslog2.primp-industries.com:1514

Note: The new syslog facility supports tcp,udp and tcps but there is a syntax bug on how to specify normal tcp, udp and or tcps.

By default, TCP is assumed and you can specify that using the following formats:

syslog1.primp-industries.com:514
tcp://syslog1.primp-industries.com:514

To specify UDP, you will need to use the following syntax:

udp://syslog1.primp-industries.com:514

To specify TCPS, you will need to use the following syntax:

ssl://syslog1.primp-industries.com:514

To view the individual syslog types, you can run the following command:

esxcli system syslog config logger list

To update the individual syslog types, you can run the following command:

esxcli system syslog config logger set --id=fdm --rotate=20 --size=2048

Tip #7

In ESXi 4.1, to enable the SSH security banner, you had to make some minor hacks. With ESXi 5, there is not an official sshd_config and you can configure the security banner by editing /etc/issue. You can also edit the motd under /etc/motd.

Tip #8

To create custom firewall rules, take a look at the blog post here.

Tip #9

Here is post on how to create custom bootable ESXi ISO along with specifying static IP Address using kernel boot options.

Tip #10

Here is a post on how you can automate host cache configuration.

Tip #11

If you are going to SSH out of an ESXi host at any point (%post, %firstboot) ensure that you enable sshClient via the ESXi firewall else you will get a connection denied. To enable the ESXi firewall use the following command:

esxcli network firewall ruleset set --ruleset-id sshClient --enabled yes

Note: If you're doing this during %post section, hostd is not running and you will need to use "localcli" to enable the firewall which by-passes hostd.

Tip #12
Here is a post on how to Disable LUN During ESXi Installation.

Tip #13

Here is a post on Removing Previous Local Datastore Label for Reinstall in ESXi 5

Tip #14

Here is a post on How to Deploy an OVF/OVA in the ESXi Shell

Tip #15

How to persist custom firewall rules in ESXi 5, take a look at these two articles here and here

Tip #16

Here is a post on Configuring ESXi Power Management Policy Using the CLI

Tip #17

Here's an article about creating custom VIB for ESXi 5.0

Here is a complete working example of an ESXi 5.0 kickstart that can help you convert your existing ESX(i) 4.x to ESXi 5.x

# Sample kickstart for ESXi 5.1
# William Lam
# www.virtuallyghetto.com
#########################################

accepteula
install --firstdisk --overwritevmfs
rootpw vmware123
reboot

%include /tmp/networkconfig

%pre --interpreter=busybox

# extract network info from bootup
VMK_INT="vmk0"
VMK_LINE=$(localcli network ip interface ipv4 get | grep "${VMK_INT}")
IPADDR=$(echo "${VMK_LINE}" | awk '{print $2}')
NETMASK=$(echo "${VMK_LINE}" | awk '{print $3}')
GATEWAY=$(localcli network ip route ipv4 list | grep default | awk '{print $3}')
DNS="172.30.0.100,172.30.0.200"
HOSTNAME=$(nslookup "${IPADDR}" "${DNS}" | grep Address | grep "${IPADDR}" | awk '{print $4}')

echo "network --bootproto=static --addvmportgroup=false --device=vmnic0 --ip=${IPADDR} --netmask=${NETMASK} --gateway=${GATEWAY} --nameserver=${DNS} --hostname=${HOSTNAME}" > /tmp/networkconfig

%firstboot --interpreter=busybox

# enable VHV (Virtual Hardware Virtualization to run nested 64bit Guests + Hyper-V VM)
grep -i "vhv.enable" /etc/vmware/config || echo "vhv.enable = \"TRUE\"" >> /etc/vmware/config

# enable & start remote ESXi Shell  (SSH)
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell (TSM)
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

# supress ESXi Shell shell warning - Thanks to Duncan (http://www.yellow-bricks.com/2011/07/21/esxi-5-suppressing-the-localremote-shell-warning/)
esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

# ESXi Shell interactive idle time logout
esxcli system settings advanced set -o /UserVars/ESXiShellInteractiveTimeOut -i 3600

# Change the default ESXi Admins group "ESX Admins" to a custom one "Ghetto ESXI Admins" for AD
vim-cmd hostsvc/advopt/update Config.HostAgent.plugins.hostsvc.esxAdminsGroup string "Ghetto ESXi Admins"

# Users that will have full access to DCUI even if they don't have admin permssions on ESXi host
vim-cmd hostsvc/advopt/update DCUI.Access string root,william,tuan

# Block VM guest BPDU packets, global configuration
esxcli system settings advanced set -o /Net/BlockGuestBPDU -i 1

# copy SSH authorized keys & overwrite existing
wget http://air.primp-industries.com/esxi5/id_dsa.pub -O /etc/ssh/keys-root/authorized_keys

# disable SSH keys - uncomment the next section
# sed -i 's/AuthorizedKeysFile*/#AuthorizedKeysFile/g' /etc/ssh/sshd_config

# rename local datastore to something more meaningful
vim-cmd hostsvc/datastore/rename datastore1 "$(hostname -s)-local-storage-1"

# assign license
vim-cmd vimsvc/license --set AAAAA-BBBBB-CCCCC-DDDDD-EEEEE

## SATP CONFIGURATIONS ##
esxcli storage nmp satp set --satp VMW_SATP_SYMM --default-psp VMW_PSP_RR
esxcli storage nmp satp set --satp VMW_SATP_DEFAULT_AA --default-psp VMW_PSP_RR

###########################
## vSwitch configuration ##
###########################

#####################################################
# vSwitch0 : Active->vmnic0,vmnic1 Standby->vmnic2
#       failback: yes
#       faildectection: beacon
#       load balancing: portid
#       notify switches: yes
#       avg bw: 1000000 Kbps
#       peak bw: 1000000 Kbps
#       burst size: 819200 KBps
#       allow forged transmits: yes
#       allow mac change: no
#       allow promiscuous no
#       cdp status: both

# attach vmnic1,vmnic2 to vSwitch0
esxcli network vswitch standard uplink add --uplink-name vmnic1 --vswitch-name vSwitch0
esxcli network vswitch standard uplink add --uplink-name vmnic2 --vswitch-name vSwitch0

# configure portgroup
esxcli network vswitch standard portgroup add --portgroup-name VMNetwork1 --vswitch-name vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name VMNetwork1 --vlan-id 100
esxcli network vswitch standard portgroup add --portgroup-name VMNetwork2 --vswitch-name vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name VMNetwork2 --vlan-id 200
esxcli network vswitch standard portgroup add --portgroup-name VMNetwork3 --vswitch-name vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name VMNetwork3 --vlan-id 333

# configure cdp
esxcli network vswitch standard set --cdp-status both --vswitch-name vSwitch1

### FAILOVER CONFIGURATIONS ###

# configure active and standby uplinks for vSwitch0
esxcli network vswitch standard policy failover set --active-uplinks vmnic0,vmnic1 --standby-uplinks vmnic2 --vswitch-name vSwitch0

# configure failure detection + load balancing (could have appended to previous line)
esxcli network vswitch standard policy failover set --failback yes --failure-detection beacon --load-balancing portid --notify-switches yes --vswitch-name vSwitch0

### SECURITY CONFIGURATION ###
esxcli network vswitch standard policy security set --allow-forged-transmits yes --allow-mac-change no --allow-promiscuous no --vswitch-name vSwitch0

### SHAPING CONFIGURATION ###
esxcli network vswitch standard policy shaping set --enabled yes --avg-bandwidth 100000 --peak-bandwidth 100000 --burst-size 819200 --vswitch-name vSwitch0

#####################################################
# vSwitch1 : Active->vmnic3,vmnic4 Standby->vmnic5
#       failback: no
#       faildectection: link
#       load balancing: mac
#       notify switches: no
#       allow forged transmits: no
#       allow mac change: no
#       allow promiscuous no
#       cdp status: listen
#       mtu: 9000

# add vSwitch1
esxcli network vswitch standard add --ports 256 --vswitch-name vSwitch1

# attach vmnic3,4,5 to vSwitch0
esxcli network vswitch standard uplink add --uplink-name vmnic3 --vswitch-name vSwitch1
esxcli network vswitch standard uplink add --uplink-name vmnic4 --vswitch-name vSwitch1
esxcli network vswitch standard uplink add --uplink-name vmnic5 --vswitch-name vSwitch1

# configure mtu + cdp
esxcli network vswitch standard set --mtu 9000 --cdp-status listen --vswitch-name vSwitch1

# configure portgroup
esxcli network vswitch standard portgroup add --portgroup-name NFS --vswitch-name vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name FT_VMOTION --vswitch-name vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name VSPHERE_REPLICATION --vswitch-name vSwitch1

### FAILOVER CONFIGURATIONS ###

# configure active and standby uplinks for vSwitch1
esxcli network vswitch standard policy failover set --active-uplinks vmnic3,vmnic4 --standby-uplinks vmnic5 --vswitch-name vSwitch1

# configure failure detection + load balancing (could have appended to previous line)
esxcli network vswitch standard policy failover set --failback no --failure-detection link --load-balancing mac --notify-switches no --vswitch-name vSwitch1

### SECURITY CONFIGURATION ###
esxcli network vswitch standard policy security set --allow-forged-transmits no --allow-mac-change no --allow-promiscuous no --vswitch-name vSwitch1

# configure vmkernel interface for NFS traffic, FT_VMOTION and VSPHERE_REPLICATION traffic
VMK0_IPADDR=$(esxcli network ip interface ipv4 get | grep vmk0 | awk '{print $2}')
VMK1_IPADDR=$(echo ${VMK0_IPADDR} | awk '{print $1".51."$3"."$4}' FS=.)
VMK2_IPADDR=10.10.0.2
VMK3_IPADDR=10.20.0.2
esxcli network ip interface add --interface-name vmk1 --mtu 9000 --portgroup-name NFS
esxcli network ip interface ipv4 set --interface-name vmk1 --ipv4 ${VMK1_IPADDR} --netmask 255.255.255.0 --type static
esxcli network ip interface add --interface-name vmk2 --mtu 9000 --portgroup-name FT_VMOTION
esxcli network ip interface ipv4 set --interface-name vmk2 --ipv4 ${VMK2_IPADDR} --netmask 255.255.255.0 --type static
esxcli network ip interface add --interface-name vmk3 --mtu 9000 --portgroup-name VSPHERE_REPLICATION
esxcli network ip interface ipv4 set --interface-name vmk3 --ipv4 ${VMK3_IPADDR} --netmask 255.255.255.0 --type static

# Configure VMkernel traffic type (Management, VMotion, faultToleranceLogging, vSphereReplication)
esxcli network ip interface tag add -i vmk2 -t Management
esxcli network ip interface tag add -i vmk2 -t VMotion
esxcli network ip interface tag add -i vmk2 -t faultToleranceLogging
esxcli network ip interface tag add -i vmk3 -t vSphereReplication

# Configure VMkernel routes
esxcli network ip route ipv4 add -n 10.20.183/24 -g 172.30.0.1
esxcli network ip route ipv4 add -n 10.20.182/24 -g 172.30.0.1

# Disable IPv6 for VMkernel interfaces
esxcli system module parameters set -m tcpip3 -p ipv6=0

### MOUNT NFS DATASTORE ###
esxcli storage nfs add --host 172.51.0.200 --share /volumes/Primp/primp-6 --volume-name himalaya-NFS-primp-6

### ADV CONFIGURATIONS ###
esxcli system settings advanced set --option /Net/TcpipHeapSize --int-value 30
esxcli system settings advanced set --option /Net/TcpipHeapMax --int-value 120
esxcli system settings advanced set --option /NFS/HeartbeatMaxFailures --int-value 10
esxcli system settings advanced set --option /NFS/HeartbeatFrequency --int-value 20
esxcli system settings advanced set --option /NFS/HeartbeatTimeout --int-value 10
esxcli system settings advanced set --option /NFS/MaxVolumes --int-value 128

### SYSLOG CONFIGURATION ###
esxcli system syslog config set --default-rotate 20 --loghost vcenter50-3.primp-industries.com:514,udp://vcenter50-3.primp-industries.com:514,ssl://vcenter50-3.primp-industries.com:1514,udp://vcenter50-3.primp-industries.com:514,udp://vcenter50-3.primp-industries.com:514,ssl://vcenter50-3.primp-industries.com:1514,ssl://vcenter50-3.primp-industries.com:1514

# change the individual syslog rotation count
esxcli system syslog config logger set --id=hostd --rotate=20 --size=2048
esxcli system syslog config logger set --id=vmkernel --rotate=20 --size=2048
esxcli system syslog config logger set --id=fdm --rotate=20
esxcli system syslog config logger set --id=vpxa --rotate=20

### NTP CONFIGURATIONS ###
cat > /etc/ntp.conf << __NTP_CONFIG__
restrict default kod nomodify notrap noquerynopeer
restrict 127.0.0.1
server 0.vmware.pool.ntp.org
server 1.vmware.pool.ntp.org
__NTP_CONFIG__
/sbin/chkconfig ntpd on

### FIREWALL CONFIGURATION ###

# enable firewall
esxcli network firewall set --default-action false --enabled yes

# services to enable by default
FIREWALL_SERVICES="syslog sshClient ntpClient updateManager httpClient netdump"
for SERVICE in ${FIREWALL_SERVICES}
do
 esxcli network firewall ruleset set --ruleset-id ${SERVICE} --enabled yes
done

# backup ESXi configuration to persist changes
/sbin/auto-backup.sh

# enter maintenance mode
esxcli system maintenanceMode set -e true

# copy %first boot script logs to persisted datastore
cp /var/log/hostd.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-hostd.log"
cp /var/log/esxi_install.log "/vmfs/volumes/$(hostname -s)-local-storage-1/firstboot-esxi_install.log"

# Needed for configuration changes that could not be performed in esxcli
esxcli system shutdown reboot -d 60 -r "rebooting after host configurations"

Semi-Interactive automated ESXi installation

05.09.2011 by William Lam // 18 Comments

There was a recent thread in the VMTN community forums where a user wanted to provide user input prior to an automated ESXi installation. This may sound odd at first, especially when we are talking about an unattended installation, the last thing you want is any type of user interaction. The reason for this requirement was to maintain a generic kickstart configuration file and based on the site specific input (IP Address, Netmask, Gateway, Hostname and DNS Servers), the ESXi installation would be able to deploy and configure itself the same across multiple locations.

UPDATE (01/10/19) - For ESXi 6.5 or greater, please take a look at this blog post for an updated solution

UPDATE (10/28/15) - Please take a look at this blog post on how to prompt for user input during an interactive or scripted installation of ESXi.

The other reason for this requirement is that all hosts in the user's environment must be configured with a static IP Address, this is not an uncommon requirement for many production environments to not have DHCP enabled networks. I can only assume the initial network the host is being built is either a private build network or booting off of local media such as a USB or CD-ROM device.

Whether you are booting off of local media or via PXE over the network, you have the ability to specify some boot parameters which includes things like IP Address, Netmask, Gateway and DNS Servers. These "bootstrap" options are fully supported and documented in the ESXi Installation Section. You can also specify advanced VMkernel boot parameters which may not be officially supported by VMware, but you can take a look here for more details.The concept of specifying these boot parameters is nothing new and has been supported since the early days of classic ESX and other popular UNIX/Linux distros for PXE boot/installations.

When you boot the ESXi installer, you may have seen a screen similar to the following in which you have a few seconds to hit the "tab" key to edit the boot options.

Once you hit the "tab" key, you will be able to see what the default boot options are and if you are PXE booting, you will also see some IP information appended towards the end of the string. This is where you can append or update additional parameters and later read in by your kickstart script.

Here is an example of an ESXi installation being PXE booted over the network and I have added 4 supported boot parameters and 2 custom ones.

If you take a look at the screenshot, the "+++" IP information towards the end is what was given by the DHCP server but I am interested in specifying a different IP Address for the ESXi installer to boot from. I provided new entries for ip, netmask, gateway and nameserver. You can also see that I introduced two new variables called hostname and dc, these will be used to specify the hostname of the ESXi host and also the name of the datacenter which will be used later in the kickstart script to rename a datastore.

Note: If you redefine the IP information, you do not need to change the IP information found after the "+++", these will just be over-written with the new IP information.

When specifying these boot parameters, you need to make sure it is after the "vmkboot.gz" but before "--- vmkernel.gz". To help make this clear, I have colorized the section that was appended to the default options.

vmkboot.gz dc=ghettoDC hostname=vesxi41-2.primp-industries.com ip=172.30.0.200 netmask=255.255.255.0 gateway=172.30.0.1 nameserver=172.30.0.100 ks=http://172.30.0.108/esxi41u1/ks.cfg --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

Note: You can create any custom variables, the key in using the custom variables will be parsing from the boot command line using vsish which will be defined in your kickstart script. This will work also work for local media, but you MUST use a kickstart to perform the actual installation or inject the kickstart script into a custom ISO.

After you have provided all the input, you will then boot the installer and the the following kickstart configuration file is used to parse the boot options using vsish.

accepteula
autopart --firstdisk --overwritevmfs
rootpw vmware
install url http://172.30.0.108/esxi41u1
reboot

%include /tmp/networkconfig

%pre --unsupported --interpreter=busybox

CMDLINE_FILE=/tmp/cmdline
ESXI_INSTALL_LOG=/var/log/esxi_install.log

# extract boot options
vsish -e get /system/bootCmdLine > ${CMDLINE_FILE}

# extract and set variables
HOSTNAME=$(cat ${CMDLINE_FILE} | grep hostname | sed -e 's/.*hostname=\([^ ]*\).*/\1/')
IPADDR=$(cat ${CMDLINE_FILE} | grep ip | sed -e 's/.*ip=\([^ ]*\).*/\1/')
NETMASK=$(cat ${CMDLINE_FILE} | grep netmask | sed -e 's/.*netmask=\([^ ]*\).*/\1/')
GATEWAY=$(cat ${CMDLINE_FILE} | grep gateway | sed -e 's/.*gateway=\([^ ]*\).*/\1/')
NAMESERVER=$(cat ${CMDLINE_FILE} | grep nameserver | sed -e 's/.*nameserver=\([^ ]*\).*/\1/')
DC=$(cat ${CMDLINE_FILE} | grep dc | sed -e 's/.*dc=\([^ ]*\).*/\1/')

# create networkline based on boot options
echo "network --bootproto=static --hostname=${HOSTNAME} --ip=${IPADDR} --netmask=${NETMASK} --gateway=${GATEWAY} --nameserver=${NAMESERVER} --addvmportgroup=0" > /tmp/networkconfig

# persist custom variables in ESXi install log for %post section
echo "GHETTO_CUSTOM_VARIABLE-DC ${DC}" >> ${ESXI_INSTALL_LOG}

%firstboot --unsupported --interpreter=busybox --level=9999

#extract custom variables in ESXi install log
DC=$(grep "^GHETTO_CUSTOM_VARIABLE-DC" /var/log/esxi_install.log | awk '{print $2}')

vim-cmd hostsvc/datastore/rename datastore1 "${DC}-datastore1"

vim-cmd hostsvc/maintenance_mode_enter
vim-cmd hostsvc/enable_remote_tsm
vim-cmd hostsvc/start_remote_tsm
vim-cmd hostsvc/enable_local_tsm
vim-cmd hostsvc/start_local_tsm

The script does the following:

Reads the boot options using vsish and temporarily stores the output to /tmp/cmdline for later use
Extracts all the relevant boot parameters in /tmp/cmdline and stores them in variables to be used in the script.
Creates the "network" stanza for static IP assignment of the ESXi host which includes: Hostname, IP Address, Netmask, Gateway and Nameserver
Writing out the custom variable "dc" into /var/log/esxi_install.log because this will be persisted through the reboot and can be later read in for any %firstboot operations.
Upon the reboot, %firstboot will execute and read from /var/log/esxi_install.log to extract the "dc" variable in which it will use to rename the local datastore

Note: The above is just an example of what you can do with custom parameters. You can easily add as many as you need for site specific configurations and then use those input in your post configure your ESXi host. Also note that you do not need to make sure of ip,netmask,gateway,dns parameters to use custom variables, these are all optional.

You can add as many custom entries as you would like, but this can easily get error prone due to the amount of typing required. One thing you can do to minimize the amount of typos is to pre-specify the custom variables in your PXE/TFTP configuration file.

To do so, your PXE/TFTP configuration file would look something like this:

IMEOUT 300 #30 seconds
PROMPT 1
DEFAULT menu.c32
SAY -
SAY vga - Install esxi410u1 (vga console)
SAY -

LABEL www.virtuallyghetto.com - Semi-Interactive ESXi Install
KERNEL mboot.c32
APPEND vmkboot.gz dc= hostname= ip= netmask= gateway= nameserver= ks=http://172.30.0.108/esxi41u1/ks.cfg --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz
IPAPPEND 1

This not only reduces the amount of typing but also lets the user know what variables must be defined in order for the installation to proceed.

I totally understand where the user is coming from and I have had this requirement in the past but I think having a static DHCP entry and maintaining a simple configuration file can easily solve this problem. This not only makes the deployment process hands-off as it should be in the first place but also removing the human factor out of the equation. Can you imagine deploying 100-300 ESXi hosts in a day? I sure can not if I had to manually type out all those addresses by hand.

Automating ESXi 4.1 Kickstart Tips & Tricks

09.11.2010 by William Lam // 25 Comments

While testing the new kickstart functionality in ESXi 4.1, I ran into a few issues trying to convert a classic ESX 4.x deployment to ESXi 4.1. I thought I share some of the tips and tricks I have learned, so others will not encounter the same issues.

Before diving in and creating an ESXi 4.1 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide.

UPDATE: For ESXi 5, Check out ESXi5 Kickstart Tips & Tricks

Tip #1

If you are going to specify a ks.cfg (kickstart configuration file) in your pxelinux file, make sure that the kickstart entry is appended after the *vmkboot.gz* but before *vmkernel.gz* entry as highlighted in green in the screenshot. If you place it anywhere else in the boot line option, you will receive an error that is not easy to diagnose. Also you want to make sure you add triple dashes (---) after the kickstart line following the required syntax for the boot options as highlighted in orange in the screenshot.

Tip #2

While developing and testing your ks.cfg, you may want to use the new dryrun parameter which parses your kickstart configuration file looking for syntax and formatting errors. In dryrun mode, no installation will be performed but you will be provided with a log of whether your ks.cfg had any errors, warnings or was successful in being validated.

The following screenshot shows a warning where I purposely left out --hostname entry which is generally recommended within the "network" portion of the ks.cfg:

If there are other errors or warnings, they will be displayed within this screen and you can login to the host to view the log for more details (esxi_install.log):

To login to the host, you will press "enter" and you will be prompted for login (press Alt+F1 to go to login screen). The username by default will be "root" and the password is blank, just press enter for the password:

Once logged in, you will want to take a look at the esxi_install.log for more details on how your ks.cfg is being processed and if there are any errors or warnings discovered by the parser:

Tip #3

If you want to enable both local and remote (SSH access) Tech Support Mode on your ESXi host, you now have the ability to do this via host services. You can use the vim-cmd (vimsh) utility to enable these services and both local and remote TSM is disabled by default.

Note: If you want to enable either local and/or remote TSM, you need to make sure you enable and start the service for you to actually be able to SSH into your ESXi host.

Tip #4

With classic ESX, if you needed to transfer additional packages or files to your host, you could easily mount an NFS volume, with ESXi, an NFS client is not available. If need to transfer files for configuration purposes, you can utilize the wget utility.

The syntax for wget is the following:

wget http://webserver/file -O /tmp/file

Tip #5

I have been told by support that you could not configure syslog for your ESXi host without relying on external tools such as vCLI, PowerCLI or vSphere Client. I have found that you actually can configure syslog configurations, though you have to dig a little bit into vim-cmd (vimsh) as it is not available using any of the local esxcfg-* commands. There only three syslog options as provided via vSphere Client in the Advanced Host Configurations: Syslog.Remote.Hostname, Syslog.Remote.Port and Syslog.Local.DatastorePath

Here is the syntax for the syslog options:

vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string syslog.primp-industries.com
vim-cmd hostsvc/advopt/update Syslog.Remote.Port int 514
vim-cmd hostsvc/advopt/update Syslog.Local.DatastorePath string "[datastoreName] /logfiles/hostName.log"

Note: Currently you can only configure one syslog server for your ESXi host to forward logs to.

Tip #6

Another new new kickstart parameter introduced with ESXi 4.1 is --level that is used in conjunction with %firstboot stanza. This parameter specifies the specific order in which the kickstart firstboot configurations should run with respect to the other startup scripts when your ESXi host first boots. By default, if you leave this out, VMware will automatically create a script called firstboot_001 and number it 999 which will be the very last script to execute. It is a good idea to move any post configurations to the very end, since most of post configuration may rely on specific VMware CLIs and services which must be started up before executing. You of course can change level, but be careful about moving it too early in the boot process.

Here is an example of changing the level to 998:

Once the host has booted up, you can login to see the script that was created from your %firstboot stanza under /etc/vmware/init/init.d

Note: As you can see, the firstboot script has now changed to 998. You will also notice two other scripts set at level 999 that handles updating the password if you decide to set a root password from the default blank, which you should. These custom scripts are generated after the initial build and upon the next reboot, these will be automatically removed.

Tip #7

You may have noticed in Tip #6, we changed the --level to 998, by default all three of these init scripts are set to boot order 999. This was actually done on purpose, the reason being as described earlier, the root password is blank by default. One issue that I found while testing is the inability to enable "Management Traffic" for a VMkernel interface. You can easily enable vMotion and FT Traffic for a VMkernel interface using vim-cmd (vimsh), but you can not for Management Traffic. One way I solved this problem is creating a python script which connects to the local ESXi MOB and enables Management Traffic on a particular VMkernel interface. I have shared this specific script on the on the VMTN communities which can be found here. The script is actually based on modified version that was initially created by Justin Guidroz who blogged about it here.

Here is the snippet that would be included in the %firstboot in which does not require you to expose the root password as it is empty by default:

ESXi 4.1

import sys,re,os,urllib,urllib2
 
# connection info to MOB
url = "https://localhost/mob/?moid=ha-vnic-mgr&method=selectVnic"
username = "root"
password = ""
 
#auth
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
#execute method
params = {'nicType':'management','device':'vmk0'}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params)
page = urllib2.urlopen(req).read()
__ENABLE_MGMT_INT__
 
python /tmp/enableVmkInterface.py

ESXi 4.1 Update 1 ( Requires CSRF code update)

cat > /tmp/enableVmkInterface.py << __ENABLE_MGMT_INT__
import sys,re,os,urllib,urllib2
 
# connection info to MOB
url = "https://localhost/mob/?moid=ha-vnic-mgr&method=selectVnic"
username = "root"
password = ""
 
# Create global variables
global passman,authhandler,opener,req,page,page_content,nonce,headers,cookie,params,e_params
 
#auth
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
# Code to capture required page data and cookie required for post back to meet CSRF requirements  ###
req = urllib2.Request(url)
page = urllib2.urlopen(req)
page_content= page.read()
 
# regex to get the vmware-session-nonce value from the hidden form entry
reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')
nonce = reg.search(page_content).group(1)
 
# get the page headers to capture the cookie
headers = page.info()
cookie = headers.get("Set-Cookie")
 
#execute method
params = {'vmware-session-nonce':nonce,'nicType':'management','device':'vmk0'}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params, headers={"Cookie":cookie})
page = urllib2.urlopen(req).read()
__ENABLE_MGMT_INT__
 
python /tmp/enableVmkInterface.py

As you can see, we first create the python script and then we execute it. This allows us to call other utilities within the Busybox console without having to specify the interpreter to be python, we can just use busybox as the interpreter.

Tip #7a Here is an alternative solution to enable management traffic type on ESXi - Another way to enable management traffic on ESXi

Tip #8

If you tried to configure NTP by echoing your NTP servers into /etc/ntpd.conf and restarting ntpd, you will notice that the changes do not take effect. The only way I have been able to get this to work is by issuing another reboot which is specified at the very end of the %firstboot which will then be picked up upon boot up by the host.

Tip #9

If you would like customize the DCUI Welcome Screen, take a look at my blog post How to add a splash of color to ESXi DCUI Welcome Screen.

Tip #10

If you want to update the default datastore name from "datastore1" to something more useful such as [hostname]-local-storage-1, you can use vim-cmd (vimsh) to do so. Here is the syntax for the command if you want to use the short hostname and append "-local-storage-1" (this should be done in the %firstboot section of your ks.cfg): vim-cmd hostsvc/datastore/rename datastore1 "$(hostname -s)-local-storage-1"

Tip #11

SNMP is another one of those configurations that can not be configured and started up via normal services as you would have done in classic ESX. You can make the appropriate edits to the configuration file and you will need to reboot the host for the changes to take affect just like NTP configurations. You will need to edit /etc/vmware/snmpd.xml and add that to your firstboot section. Here is an example of snmpd.xml file:

<config>
  <snmpsettings> 
    <communities>public1;private1</communities> 
    <enable>true</enable> 
    <port>163</port> 
    <targets>192.168.1.5 public1;192.168.1.6@163 private1</targets
  </snmpsettings>
</config>

Tip #12

A VMTN user recently posted an issue when applying patches during firstboot, that the init scripts were not being removed and the scripts continue to execute upon every reboot. The problem was that the boot.cfg were not being properly updated under /vmfs/volumes/Hypervisor{1,2}. I did some testing and found that if you created a second customization script and perform the patching as the very last step and issued a reboot, that you would not run into this problem. Here is a small snippet of what your ks.cfg would like look with 2 custom init scripts, one configured at 998 and the other configured at 9999:

%firstboot --unsupported --interpreter=busybox --level=998
# do your customization
# in this section
 
%firstboot --unsupported --interpreter=busybox --level=9999
# do your patching
# in this section
 
#issue one final reboot
reboot

Tip #13

Here is a post on Automating Active Directory Domain Join in ESXi Kickstart

Tip #14

Here is a post on Automating Active Directory User Management in ESXi Kickstart As you can see, there are quite a few hacks I had to go through to get an equivalent kickstart build for ESXi 4.1 compared to classic ESX. I am sure there are other gotchas, but these are the ones I had encountered. Overall, ESXi 4.1 has greatly improved in terms of automating an unattended installation and configuration from ESXi 4.0, but there is definitely more work that needs to be done by VMware before users can easily transition from classic ESX to ESXi.

Tip #15
Here is a post on How to automatically add ESX(i) host to vCenter in Kickstart

In additional to VMware's documentation which is still limiting, here are additional tools and links that will be useful when creating your ks.cfg for ESXi 4.1: