WilliamLam.com

Search Results for: kickstart

How to inject custom drivers into an ESXi 4.1 image using vibddi?

by // 11 Comments

Over the holiday break, I spent some time cleaning up some of the development virtual machines in our ghettoDatacenter. I came across the VMware Auto Deploy appliance that I deployed awhile back ago. I did not think I had a use for it since we already have an automated deployment system using PXE and kickstart. Auto Deploy was launched relatively recently from the VMware Flings lab. It was originally slated for release as part of vSphere 4.1 but during the transition from the BETA to RC, it was dropped and never made it into the GA release of vSphere 4.1

I decided to give the documentation one last read before deleting and to my surprise, I stumbled across an interesting gem, vibddi. vibddi (pronounced vib d-d-i) stands for VIB (vSphere Instalaltion Bundle) Disk Dump Image, which is actually a Perl utility that was created to help customize ESXi images more easily.

If you ever had a need to customize an ESXi image and inject custom drivers or configurations, you know it can be long and complex process. There are many tutorials on the internet including a recent post by Eric Sloof on injecting drivers into an ESXi installer. vibddi is meant to expedite the process and make it much simpler to inject custom drivers into an ESXi image.

****Disclaimer Since this tool is not very well documented and it is most likely not officially supported by VMware, please use test and validate the images generated prior to using in an production environment Disclaimer****

To run vibddi, you need to use sudo. Here are the available options:

[vi-admin@autodeploy ~]$ sudo vibddi -h
Password:

vibddi: Query and update vibs on a VMvisor dd image or device

Usage:
vibddi -h --- Print this

vibddi -i -q --- Query vibs installed on the image

vibddi -i -c --- Check bootbank filesystems on the image

vibddi -i -v [ -g ] [ -n ] --- Update the image with a single vib

vibddi -i -m -b [ -p ] [ -g ] [ -n ] --- Update the image with an online bulletin

vibddi -i -o [ -g ] [ -n ] --- Update the image with an offline bundle

vibddi -i -e [ -a ] --- Export boot modules from the image

vibddi -i -t --- Add/Remove a VMkernel option

vibddi -i -x --- Transform image to ThinESX format

vibddi -i -l --- Install a license file (vmware.lic) on the image

vibddi -d -q --- Query vibs installed on the device

vibddi -d -c --- Check bootbank filesystems on the device

vibddi -d -v [ -n ] --- Update the device with a single vib

vibddi -d -m -b [ -p ] [ -n ] --- Update the device with an online bulletin

vibddi -d -o [ -n ] --- Update the device with an offline bundle

vibddi -d -e [ -a ] --- Export boot modules from the device

vibddi -d -t --- Add/Remove a VMkernel option

vibddi -d -x --- Transform image to ThinESX format

vibddi -f -k --- Add a customized kickstart file to the ThinESX/Recovery CD ISO

Where:
VMvisor-dd - The VMvisor dd image that is going to be customized

VMvisor-dev - The VMvisor device that is going to be updated

vib-path - The local file path to the vib

metadata-URL - The URL to the metadata.zip file (Ex. http://www.oem.com/depot/metadata.zip)

bulletin-ID - The bulletin ID to install

bundle-path - The local file path to the offline bundle

proxy (OPTIONAL) - Proxy used to download vib, for update operation only

-g (OPTIONAL) - Generate customized ThinESX/Recovery CD ISOs

-n (OPTIONAL) - Bypass signature check, for update operation only

export-path - Directory to export boot modules

alternate-conf (OPTIONAL) - Alternate export configuration file

kernel-opt - VMkernel option

license-path - vmware.lic file (Format: 00000-00000-00000-00000-00000)

iso-path - The local file path to the ThinESX/Recovery CD ISO

kickstart-path - The local file path to the kickstart file

Here are a few examples of using the vibddi tool:
Mount ESXi 4.1 ISO to extract the DD image:

[vi-admin@autodeploy scratch]$ sudo mount -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/iso/

Unzip the DD image and extract to current directory:

[vi-admin@autodeploy scratch]$ sudo bunzip2 -c /mnt/iso/imagedd.bz2 > imagedd

You now should have the DD image called imagedd located in your current working directory.You can name the file anything you want, but I'm using the suggested name as noted in the Auto Deploy documentation.

To list vibs installed on the image, you'll use the following command:

sudo vibddi -i [imagedd] -q

Here is an example of the vibs installed with default installation of ESXi 4.1:

To inject the image with an offline bundle, you'll use the following command:

sudo vibddi -i [imagedd] -o [offline_bundle] -n

Note: The -n flag should be used when performing updates as it bypasses the signature checks, else you will get an error.

Here is an example of injecting the Cisco Nexus 1000 Virtual Ethernet Module offline bundle as part of the default ESXi 4.1 installation:

We can confirm the Cisco VEM is part of the default image by running the query command again:

To inject the image with a single VIB, you'll use the following command:

sudo vibddi -i [imagedd] -v [vib] -n

Here is an example injecting the Cisco Nexus 1000 Virtual Ethernet Module VIB as part of the default ESXi 4.1 installation:

To inject VMkernel boot parameters, you'll use the following command:

sudo vibddi -i [imagedd] -t [vmkernel_option]

Note: Here is a list of a few VMkernel options documented by Dave Mishchenko. The -t argument only accepts one VMkernel option at a time. If you want to updated more than one option, you will need to run the command for each VMkernel option.

With a default installation of ESXi 4.1, there are no VMkernel options defined. To see whether or not these have been defined, you will need to login to Tech Support Mode and view boot.cfg:

~ # cat bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- aam.vgz
build=4.1.0-260247
updated=1
bootstate=0

Here is an example of injecting the following two VMkernel options: noACPI and nopowerManagement:

To inject a license file, you'll use the following command:

sudo vibddi -i [imagedd] -l [license_file]

Note: The license file must contain a single entry using the following format - 00000-00000-00000-00000-00000

Here is an example of injecting license file:

To inject a custom kickstart configuration, you'll use the following command:

sudo vibddi -f [esxi-iso-path] -k [kickstart_file]

Here is an example of injecting a custom kickstart file:

Note: This actually injects a custom ks.cfg into the ESXi .iso which can then be used to deploy an ESXi host including the custom configurations found in the kickstart file. A brand new .iso will be created in the current working directory which includes the timestamp of kickstart injection as part of its filename.

We now can loop mount the new .iso and verify the custom kickstart has been injected:

Note: I'm using the sample ks.cfg found on Kendrick Colemans's site.

You can also extract certain items from the DD image, you'll use the following command:

sudo vibddi -i [imagedd] -e [export-path]

Here is an example of extracting the entire DD image to a temporarily directory:

To check the bootbank filesystem, you'll use the following command:

sudo vibddi -i [imagedd] -c

Here is an example of verifying bootbank filesystem:

Once the imagedd has been updated with all the drivers, you will need to compress the image back to .bz2 using bzip2. From here, you will have two options: A) copy the modified imagedd.bz2 over to your PXE/TFTP server used for automated kickstart installation B) Create a new ESXi .iso, there are a bunch of tutorials online such as here and here.

If you need to troubleshoot or would like to view the process of vibddi, you can take a look at the logs stored in /var/log/vibddi.log. You can also see the injection process which includes both informational and debugging logs in /var/log/esxupdate.log.

As you can see, this tool is extremely useful for injecting and customizing ESXi images. Hopefully one day VMware will officially release this tool and make it available on both UNIX/Linux and Windows platform so that everyone can benefit. For now, if you want to use vibddi, you will need to download and use Auto Deploy appliance. Looks like I'll be keeping this appliance around 😉

How to add a splash of color to ESXi DCUI Welcome Screen

by // 25 Comments

Earlier this year I created a simple vSphere SDK for Perl script that allows you to update ESXi's DCUI (Direct Console User Interface) banner with multiple lines of text. I originally thought you could not customize the text color or the background color, though recently I found out that was not the case. While doing some testing on ESXi 4.1, I noticed two files (support,welcome) located under /etc/vmware and looking at the contents of support, it made realize we might be able to change the colors.

Here is the contents of /etc/vmware/support, notice the special formatting of the variables including color tags:

I decided to use one of my favorite UNIX utility, "strings" to take at the dcui binary that is located under /sbin in the Busybox Console (Tech Support Mode) and discovered you can control both the font color and background color. There are also special variables that can be used to display information about the ESXi host such as the product version or IP Address.

Here are the supported colors:

white
black
dark-grey
light-grey
yellow

Here are the special variables:

assettag
BIOSversion
BMCversion
CIM_Chassis
CPLDversion
esxproduct
esxversion
hostname
ip
license
memory
OMC_MCFirmwareIdentity
OMC_SMASHFirmwareIdentity
OtherIdentifyingInfo
PLSAversion
serial-number
servicetag
ssl-thumbprint
supportperiod
supportstart
VersionString
VMware_HHRCSoftwareIdentity

There are two ways of updating the DCUI welcome banner: using local or remote esxcfg-advcfg or manually editing /etc/vmware/welcome file.

Here is an example of using vCLI's esxcfg-advcfg:

Here is what that looks like on the DCUI:

As you can see, this is not easy if you want to update multiple lines. You would need to add a lot more spaces to force newlines, but this becomes tedious and pretty much unreadable. The second method is edit the welcome file that is located in the Busybox Console, which requires you to enable ESXi's Tech Support Mode. I wrote a quick Perl script called generateDCUIScreen.pl which accepts an input file and allows a user to customize the output and the script generates the "welcome" file which is uploaded to your ESXi host.

Here is an example of the input file:

The script will parse the input file which will contain definitions for:

  • bgcolor and color as described above
  • special variables as described above (must use braces for variables to be translated)
  • custom text
  • "=space=" string which is unique to my script which generates the newlines

The script requires that you have Perl, but you do not need to have vSphere SDK for Perl. For ease of use, I executed the script using vMA.

Here is an example execution using the input file from above:

You will need to scp the new"welcome" file to your ESXi host under /etc/vmware which is empty by default. For the changes to take effect, you will need to run the following command at the console:

kill $(ps | grep dcui | awk '{print $1}')

This will kill dcui utility and watchdog process will spawn a new instance causing the change to take effect Note: A reboot will also do the job, but be sure to run /sbin/auto-backup.sh before doing so, that way the change will be backed up.

Here is what DCUI screen looks like:

As you can see, you can control variety of pre-defined variables including hostname and IP Address and custom text for your organization. This is useful for those that do not want to expose all the information that available on the default DCUI screen, which may be a security concern for some organizations. A few things to note, I was not able to fill the entire screen like the default DCUI banner and the "welcome" file is character sensitive and you need to use tabs or white spaces to force the background to get filled. There is also a limit in the number of characters per row before it wraps to the next line.

I am sure there is someone out there that will create some interesting ASCII art, but here is my 5min free hand attempt at it 😉

This can easily be integrated into a scripted install using the new ESXi 4.1 kickstart feature.

What is VMware vsish?

by // 20 Comments

Recently while I was working on testing automated ESXi kickstart installations, I needed to extract some information as part of the build process, but the utilities that were used no longer existed in ESXi's Busybox console. Looking around, I found another way to extract the information I needed, which was using VMware's undocumented vsish utility, also known as the VMkernel Sys Info Shell. There is not much information around the web regarding this vsish utility (probably for good reason), but it has been described by few as a representation of classic Service Console /proc nodes and allows the ability to extract system reliability information similar to mcelog in Linux.

If I recall correctly, the vsish utility used to be bundled with classic ESX, but at some point it was removed. However, in ESXi, the utility is included and that is also true for latest release of ESXi 4.1. When you generate vm-support log, a dump of the vsi nodes are generally included which provides VMware support with the state of your system. To read the vsi node dump, you need a matching version of the vsish utility based on the version of ESX or ESXi you are running. Currently for classic ESX, to obtain the vsish utility, it must be provided by VMware support via debugging package that needs to be installed.

Note: A word of caution before using this utility, you should not make any changes that you are unfamiliar with. Always consult with VMware support before making changes as it can severely impact your host and virtual machines. Okay, now on to the fun stuff 🙂

On ESXi Busybox Console, you can launch the vsish utility by just typing "vsish":

You can perform various operations such as listing the various nodes, get and set parameters, etc. to see the available options, just type "help":

There is a huge amount of information that can be retrieved from vsish. One interesting leaf node within vsish is called "config", this actually maps to the Advanced Settings found on an ESX(i) host:

As you can see, the majority of the sub leaf nodes within "config" is exposed in the Advanced Settings, but there are some that are hidden. In fact, with ESX(i) 4.1, there is a total of 771 configurable options with 250 of those hidden that can only be seen using vsish (more on the configuration option later)!

There are two ways to interact with vsish, you can interactively login to VSI shell and perform ls, get or set operations or perform the same operation through non-interactive mode.

Here is an example of an interactive session listing config's under "COW" leaf node and getting and setting the value for "COWDiskSizeIncrement" which is one of the 250 hidden configuration options:

Here is an example of an non-interactive session performing the same operation as the one listed above:

With the release of vSphere 4.1, there have been a few new additions to the VSI nodes. Here is an excerpt slide from vSphere 4.1 to 4.0 differences presented by Iwan Rahabok - Senior Systems Consultant at VMware listing some of the new Storage I/O Control features:

Here is a screenshot on some of these values if you can not make it out in the slide:

For more details on differences between vSphere 4.0 and vSphere 4.1 - Check out the detailed 2 part deep-dive power point presentation here.

vsish provides an enormous amount of information and I have only begin to scratch the surface. Having said that I did manage to capture all the advanced host settings which includes both public and hidden options. Using a few for loops and some shell scripting, I have generated the following two lists:

Complete vSphere ESXi 4.1 vsish configurations including hidden options - 771 Total:

For the complete list, take a look at https://s3.amazonaws.com/virtuallyghetto-download/complete_vsish_config.html

Hidden vSphere ESXi 4.1 vsish configurations only - 250 Total:

For the hidden list only, take a look at https://s3.amazonaws.com/virtuallyghetto-download/hidden_vsish_config.html

There are definitely some interesting options that can be configured and I can see why VMware would want to hide these from the general public. What is nice about the compiled output, is that it clearly states the path to the configuration item, the current, default, min, max, is hidden or description of the parameter. Again, use at your own risk Hopefully this these two documents will be useful for curious users to explore vsish advanced configs.

Using the table above, you can actually query and modify these values using the standard esxcfg-advcfg utility that exists both on ESX(i). The following example will show you how to translate the vsish node path to proper format that is required for local esxcfg-advcfg utility.

In the example, we will be using vsish path "/config/COW/intOpts/COWDiskSizeIncrement"

Using a local copy of esxcfg-advcfg on ESX or ESXi, you will need to convert the above to the following:

[root@himalaya ]# esxcfg-advcfg -g /COW/COWDiskSizeIncrement
Value of COWDiskSizeIncrement is 32768

Note: You just need to extract the root node and the individual config leaf node in "/config/COW/intOpts/COWDiskSizeIncrement" which is highlighted in green and blue.

Some other interesting nodes that I found that might be useful are under /system:
/system/bootOption
/system/bootCmdLine
/system/systemUuid
/system/bootFsUUID

One other interesting tidbit of information that I found was under /system/version which actually shows the build date and time of vSphere ESXi 4.1:

Now that we have a better understanding of the vsish utility, how does this help with my original inquiry? I found that you can extract networking information from your vNICs by looking at /net/tcpip/* nodes:

As you can see from the output above, it's format is in hexidecimal but when converted, you will get the IP Address, netmask and the gateway for a given VMkernel interface. To accomplish this, I used a modified python script to convert these entries to their human readable addresses:

Here some additional links referencing vsish that may be of interest: