ESXi 6.0 – Page 7

How to create custom ESXi boot menu to support multiple Kickstart files?

06.11.2015 by William Lam // 31 Comments

I recently received a question from one of my readers who was looking to migrate from ESXi 4.1 to newer version and one of the challenges they faced was around their ESXi scripted installs, better known as ESXi Kickstart. Previously, they had relied on using a custom syslinux boot menu to be able to select a specific Kickstart configuration file that resided locally on a bootable ESXi Image (USB, ISO or CDROM) as a PXE/DHCP environment was not allowed in their environment. There was a small change to how ESXi boot files were reference between ESXi 4.x and ESXi 5.x/6.x and a new boot.cfg configuration is now used which I had written about here with respect to scripted installs when ESXi 5.0 was first released.

Luckily, even with these changes one can still use a custom menu with ESXi 5.x/6.x and be able to select a specific Kickstart configurations based on user input. Here is a screenshot example of a custom ESXi Image that I built providing three different install options that could be selected which would map to three different Kickstart configurations which can be either local to the boot media or can also be retrieved remotely.

The first thing you should be aware of if you plan to boot the custom ESXi Image from local media such as USB, CDROM or ISO is that the path to the Kickstart file must be in all UPPER CASE which is mentioned in this VMware KB 1026373. The next caveat that I found in my testing is that if you plan to store the local Kickstart files inside of a directory within the ESXi Image, the name of the directory can not be too long. I would recommend using "ks" as "kickstart" apparently was too long.

After you have extracted the contents of an ESXi ISO which you have downloaded, you will want to create a root directory called "ks" which will contain the different Kickstart configuration files. Here is an example of what structure look like:

ks
├── ks1.cfg
├── ks2.cfg
└── ks3.cfg

Next, you will need to edit the isolinux.cfg file which comes by default within the ESXi ISO. This is where you will add the different Kickstart options that a user will be able to select from. In this first example, we will look at referencing the Kickstart files locally on the media which can be either USB or CDROM and you will need to ensure you specify the right boot option as shown here in the VMware documentation. The path to the Kickstart file needs to be appended to the line that contains boot.cfg reference and you must ensure you include "+++" at the end of that line.

Here is an example of referencing a Kickstart file that lives on a USB device under this path /ks/ks.cfg:

APPEND -c boot.cfg ks=usb:/KS/KS.CFG +++

Here is an example of my isolinux.cfg for the boot menu that I have shown above which provides three different options mapping to three different Kickstart configuration files:

DEFAULT menu.c32
MENU TITLE vGhetto Custom ESXi 6.0 Boot Menu
NOHALT 1
PROMPT 0
TIMEOUT 80
LABEL Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS1.CFG +++
  MENU LABEL ^1 Ghetto Install
LABEL A bit More Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS2.CFG +++
  MENU LABEL ^2 A bit More Ghetto Install
LABEL Super Ghetto ESXi Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS3.CFG +++
  MENU LABEL ^3 Super Ghetto ESXi Install
LABEL hddboot
  LOCALBOOT 0x80
  MENU LABEL ^Boot from local disk

As I mentioned earlier, the Kickstart configuration file can either be retrieved locally or it can also be retireved remotely using one of the following supported protocols: http, https, ftp & nfs as shown here in the VMware documentation.

Here is an example of isolinux.cfg for a boot menu which references both a local kickstart as well as one that remotely lives on a web server:

DEFAULT menu.c32
MENU TITLE vGhetto Custom ESXi 6.0 Boot Menu
NOHALT 1
PROMPT 0
TIMEOUT 80
LABEL Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=cdrom:/KS/KS1.CFG +++
  MENU LABEL ^1 Ghetto Install
LABEL A bit More Ghetto Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=http://172.30.0.108/ks/ks2.cfg +++
  MENU LABEL ^2 A bit More Ghetto Install
LABEL Super Ghetto ESXi Install
  KERNEL mboot.c32
  APPEND -c boot.cfg ks=http://172.30.0.108/ks/ks3.cfg +++
  MENU LABEL ^3 Super Ghetto ESXi Install
LABEL hddboot
  LOCALBOOT 0x80
  MENU LABEL ^Boot from local disk

For additional ESXi Kickstart resources and example, be sure to check out my pages here.

Log filtering capability in ESXi 6.0

05.20.2015 by William Lam // 5 Comments

When it comes to troubleshooting, something that you can never have too much of are logs! However, you can have excessive logs in the form of repeated log entries for a particular event which not only adds to the amount of logs you must sift through but it also adds unnecessary load and processing to the network. This is especially problematic if you these repeated entries also being forwarded from multiple sources to a centralized syslog server.

With earlier release of ESX (yes, classic ESX), it was possible to filter out specific log entries from the host side and prevent them from showing up in the local logs stored on the filesystem after N-occurrences and would also prevent them from being forwarded to a syslog server. However, when ESXi was first introduced, this particular capability wast not ported over which I can only assume was based on usage from our customer base. In speaking with GSS, this is usually something they require for troubleshooting purposes, although I have also seen a few customers ask about this capability on several occasions in the past.

Having said that, I was pleasantly surprised to learn from Alan Castonguay (former GSS Engineer, now working over in our MBU) that ESXi 6.0 actually now includes a new log filtering capability or rather I should say, it has re-introduced the log filtering capability 🙂

To enable log filtering, you will need to add the following parameter to /etc/vmsyslog.conf

enable_logfilters = true

To add a specific log filter, you will need to add an entry to /etc/vmware/logfilters using the following format: numLogs | ident | logRegexp

numLogs - Number of times the log entry can appear in the log before it is then filtered and ignored
ident - The ident source of the log, you can find all ident by looking in /etc/vmsyslog.conf.d/*.conf
logRegexp - regular expression confirming to the Python regular expression syntax

Below is an example where I only want the log entry "SOCKET connect failed, error 2: No such file or directory.*" to show up only two times in the logs before it is filter and ignored and the source of this log entry is from the hostd logs.

2| Hostd | SOCKET connect failed, error 2: No such file or directory.*

UPDATE (03/18/16): I just got a heads up from GSS that the "hostd" logs is actually case sensitive when creating the filter. You will need to use a capital "H" in case you are not seeing results from your regexp.

Once you have created all your log filters, you will need to reload the syslog service by running the following ESXCLI command:

esxcli system syslog reload

One thing to be aware of is that once a log entry has been filtered out and the local logs have been rotated out, that particular entry will no longer show up in future logs. It is definitely recommended that you use the log filtering feature sparingly and ensure that you are also forwarding your logs to a centralized syslog server like Log Insight for example so that you have all log entries at your disposal for both troubleshooting and auditing purposes. There is already a request to add this to our official VMware documentation so that it customers can easily find this in the syslog configuration section of the ESXi documentation, but for now I have documented it here so others can benefit from this capability if needed.

Easily automate ESXi 6.0 Active Directory join using domainjoin-cli

04.06.2015 by William Lam // 9 Comments

A nice little enhancement that I recently came across in ESXi 6.0 is the inclusion of the Likewise utility called domainjoin-cli which allows you to join a system to an Active Directory Domain. Previously, if you wanted to automate the process of joining an ESXi host to an Active Directory Domain, you had to either manually configure it using the vSphere Web/Client, using Host Profiles or creating an external script using the vSphere APIs.

All of these options were mostly executed during the post-provisioning process and if you wanted to include Active Directory configuration as part of the provisioning process, you may have had to resort to something like calling into the vSphere MOB within a Kickstart script as I had shown back in 2011 in this article here. The solution I came up with was not ideal but it worked for those that did not want to have additional steps after initial provisioning.

With the domainjoin-cli utility now included in the ESXi Shell of ESXi 6.0, you easily automate the joining an Active Directory Domain with just a couple of lines added to your Kickstart or provisioning scripts. Before you can use the command-line utility, you will need to ensure the Likewise Service Manager Daemon is running by running the following two commands which will start the service and also ensure the service automatically starts up:

/etc/init.d/lwsmd start
chkconfig lwsmd on

Next, to join to your Active Directory Domain, you will need to specify the following 3 parameters:

join - Specifying the operation is a join versus a leave
AD Domain Name - Active Directory Domain to join
AD Username - Active Directory username to join to the domain
AD Password - Active Directory password to join to the domain (optional as you will be prompted if it is not specified)

Here is an example of what the command looks like joining my Active Directory Domain in my lab:

/usr/lib/vmware/likewise/bin/domainjoin-cli join primp-industries.com administrator [PASSWORD]

You should see a success message if the ESXi host was successfully joined to the Active Directory Domain and you will want to reboot your ESXi host for the changes to take full effect. This is definitely a simpler method to include into an ESXi Kickstart script to automate the joining of an Active Directory Domain and hopefully you will find this handy when using ESXi 6.0.