Automating ESXi 4.1 Kickstart Tips & Tricks

While testing the new kickstart functionality in ESXi 4.1, I ran into a few issues trying to convert a classic ESX 4.x deployment to ESXi 4.1. I thought I share some of the tips and tricks I have learned, so others will not encounter the same issues.

Before diving in and creating an ESXi 4.1 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide.

UPDATE: For ESXi 5, Check out ESXi5 Kickstart Tips & Tricks

If you are going to specify a ks.cfg (kickstart configuration file) in your pxelinux file, make sure that the kickstart entry is appended after the *vmkboot.gz* but before *vmkernel.gz* entry as highlighted in green in the screenshot. If you place it anywhere else in the boot line option, you will receive an error that is not easy to diagnose. Also you want to make sure you add triple dashes (---) after the kickstart line following the required syntax for the boot options as highlighted in orange in the screenshot.

While developing and testing your ks.cfg, you may want to use the new dryrun parameter which parses your kickstart configuration file looking for syntax and formatting errors. In dryrun mode, no installation will be performed but you will be provided with a log of whether your ks.cfg had any errors, warnings or was successful in being validated.

The following screenshot shows a warning where I purposely left out --hostname entry which is generally recommended within the "network" portion of the ks.cfg:

If there are other errors or warnings, they will be displayed within this screen and you can login to the host to view the log for more details (esxi_install.log):

To login to the host, you will press "enter" and you will be prompted for login (press Alt+F1 to go to login screen). The username by default will be "root" and the password is blank, just press enter for the password:

Once logged in, you will want to take a look at the esxi_install.log for more details on how your ks.cfg is being processed and if there are any errors or warnings discovered by the parser:

If you want to enable both local and remote (SSH access) Tech Support Mode on your ESXi host, you now have the ability to do this via host services. You can use the vim-cmd (vimsh) utility to enable these services and both local and remote TSM is disabled by default.

Note: If you want to enable either local and/or remote TSM, you need to make sure you enable and start the service for you to actually be able to SSH into your ESXi host.

With classic ESX, if you needed to transfer additional packages or files to your host, you could easily mount an NFS volume, with ESXi, an NFS client is not available. If need to transfer files for configuration purposes, you can utilize the wget utility.

The syntax for wget is the following:

wget http://webserver/file -O /tmp/file

I have been told by support that you could not configure syslog for your ESXi host without relying on external tools such as vCLI, PowerCLI or vSphere Client. I have found that you actually can configure syslog configurations, though you have to dig a little bit into vim-cmd (vimsh) as it is not available using any of the local esxcfg-* commands. There only three syslog options as provided via vSphere Client in the Advanced Host Configurations: Syslog.Remote.Hostname, Syslog.Remote.Port and Syslog.Local.DatastorePath

Here is the syntax for the syslog options:

vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string syslog.primp-industries.com
vim-cmd hostsvc/advopt/update Syslog.Remote.Port int 514
vim-cmd hostsvc/advopt/update Syslog.Local.DatastorePath string "[datastoreName] /logfiles/hostName.log"

Note: Currently you can only configure one syslog server for your ESXi host to forward logs to.

Another new new kickstart parameter introduced with ESXi 4.1 is --level that is used in conjunction with %firstboot stanza. This parameter specifies the specific order in which the kickstart firstboot configurations should run with respect to the other startup scripts when your ESXi host first boots. By default, if you leave this out, VMware will automatically create a script called firstboot_001 and number it 999 which will be the very last script to execute. It is a good idea to move any post configurations to the very end, since most of post configuration may rely on specific VMware CLIs and services which must be started up before executing. You of course can change level, but be careful about moving it too early in the boot process.

Here is an example of changing the level to 998:

Once the host has booted up, you can login to see the script that was created from your %firstboot stanza under /etc/vmware/init/init.d

Note: As you can see, the firstboot script has now changed to 998. You will also notice two other scripts set at level 999 that handles updating the password if you decide to set a root password from the default blank, which you should. These custom scripts are generated after the initial build and upon the next reboot, these will be automatically removed.

You may have noticed in Tip #6, we changed the --level to 998, by default all three of these init scripts are set to boot order 999. This was actually done on purpose, the reason being as described earlier, the root password is blank by default. One issue that I found while testing is the inability to enable "Management Traffic" for a VMkernel interface. You can easily enable vMotion and FT Traffic for a VMkernel interface using vim-cmd (vimsh), but you can not for Management Traffic. One way I solved this problem is creating a python script which connects to the local ESXi MOB and enables Management Traffic on a particular VMkernel interface. I have shared this specific script on the on the VMTN communities which can be found here. The script is actually based on modified version that was initially created by Justin Guidroz who blogged about it here.

Here is the snippet that would be included in the %firstboot in which does not require you to expose the root password as it is empty by default:

ESXi 4.1

import sys,re,os,urllib,urllib2
 
# connection info to MOB
url = "https://localhost/mob/?moid=ha-vnic-mgr&method=selectVnic"
username = "root"
password = ""
 
#auth
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
#execute method
params = {'nicType':'management','device':'vmk0'}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params)
page = urllib2.urlopen(req).read()
__ENABLE_MGMT_INT__
 
python /tmp/enableVmkInterface.py

ESXi 4.1 Update 1 ( Requires CSRF code update)

cat > /tmp/enableVmkInterface.py << __ENABLE_MGMT_INT__
import sys,re,os,urllib,urllib2
 
# connection info to MOB
url = "https://localhost/mob/?moid=ha-vnic-mgr&method=selectVnic"
username = "root"
password = ""
 
# Create global variables
global passman,authhandler,opener,req,page,page_content,nonce,headers,cookie,params,e_params
 
#auth
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
# Code to capture required page data and cookie required for post back to meet CSRF requirements  ###
req = urllib2.Request(url)
page = urllib2.urlopen(req)
page_content= page.read()
 
# regex to get the vmware-session-nonce value from the hidden form entry
reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')
nonce = reg.search(page_content).group(1)
 
# get the page headers to capture the cookie
headers = page.info()
cookie = headers.get("Set-Cookie")
 
#execute method
params = {'vmware-session-nonce':nonce,'nicType':'management','device':'vmk0'}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params, headers={"Cookie":cookie})
page = urllib2.urlopen(req).read()
__ENABLE_MGMT_INT__
 
python /tmp/enableVmkInterface.py

As you can see, we first create the python script and then we execute it. This allows us to call other utilities within the Busybox console without having to specify the interpreter to be python, we can just use busybox as the interpreter.

Tip #7a Here is an alternative solution to enable management traffic type on ESXi - Another way to enable management traffic on ESXi

If you tried to configure NTP by echoing your NTP servers into /etc/ntpd.conf and restarting ntpd, you will notice that the changes do not take effect. The only way I have been able to get this to work is by issuing another reboot which is specified at the very end of the %firstboot which will then be picked up upon boot up by the host.

If you would like customize the DCUI Welcome Screen, take a look at my blog post How to add a splash of color to ESXi DCUI Welcome Screen.

If you want to update the default datastore name from "datastore1" to something more useful such as [hostname]-local-storage-1, you can use vim-cmd (vimsh) to do so. Here is the syntax for the command if you want to use the short hostname and append "-local-storage-1" (this should be done in the %firstboot section of your ks.cfg): vim-cmd hostsvc/datastore/rename datastore1 "$(hostname -s)-local-storage-1"

SNMP is another one of those configurations that can not be configured and started up via normal services as you would have done in classic ESX. You can make the appropriate edits to the configuration file and you will need to reboot the host for the changes to take affect just like NTP configurations. You will need to edit /etc/vmware/snmpd.xml and add that to your firstboot section. Here is an example of snmpd.xml file:

<config>
  <snmpsettings> 
    <communities>public1;private1</communities> 
    <enable>true</enable> 
    <port>163</port> 
    <targets>192.168.1.5 public1;192.168.1.6@163 private1</targets
  </snmpsettings>
</config>

Tip #12

A VMTN user recently posted an issue when applying patches during firstboot, that the init scripts were not being removed and the scripts continue to execute upon every reboot. The problem was that the boot.cfg were not being properly updated under /vmfs/volumes/Hypervisor{1,2}. I did some testing and found that if you created a second customization script and perform the patching as the very last step and issued a reboot, that you would not run into this problem.  Here is a small snippet of what your ks.cfg would like look with 2 custom init scripts, one configured at 998 and the other configured at 9999:

%firstboot --unsupported --interpreter=busybox --level=998
# do your customization
# in this section
 
%firstboot --unsupported --interpreter=busybox --level=9999
# do your patching
# in this section
 
#issue one final reboot
reboot

Here is a post on Automating Active Directory Domain Join in ESXi Kickstart

Here is a post on Automating Active Directory User Management in ESXi Kickstart As you can see, there are quite a few hacks I had to go through to get an equivalent kickstart build for ESXi 4.1 compared to classic ESX. I am sure there are other gotchas, but these are the ones I had encountered. Overall, ESXi 4.1 has greatly improved in terms of automating an unattended installation and configuration from ESXi 4.0, but there is definitely more work that needs to be done by VMware before users can easily transition from classic ESX to ESXi.

Tip #15
Here is a post on How to automatically add ESX(i) host to vCenter in Kickstart

In additional to VMware's documentation which is still limiting, here are additional tools and links that will be useful when creating your ks.cfg for ESXi 4.1:

• http://communities.vmware.com/blogs/vmwareinsmb/2010/07/13/esxi-41-scripted-installation-via-pxe-and-kickstart
• http://www.kendrickcoleman.com/index.php?/Tech-Blog/esxi-41-kickstart-install-wip.html
• http://labs.vmware.com/flings/vmware-auto-deploy