WilliamLam.com

How to inject custom drivers into an ESXi 4.1 image using vibddi?

by // 11 Comments

Over the holiday break, I spent some time cleaning up some of the development virtual machines in our ghettoDatacenter. I came across the VMware Auto Deploy appliance that I deployed awhile back ago. I did not think I had a use for it since we already have an automated deployment system using PXE and kickstart. Auto Deploy was launched relatively recently from the VMware Flings lab. It was originally slated for release as part of vSphere 4.1 but during the transition from the BETA to RC, it was dropped and never made it into the GA release of vSphere 4.1

I decided to give the documentation one last read before deleting and to my surprise, I stumbled across an interesting gem, vibddi. vibddi (pronounced vib d-d-i) stands for VIB (vSphere Instalaltion Bundle) Disk Dump Image, which is actually a Perl utility that was created to help customize ESXi images more easily.

If you ever had a need to customize an ESXi image and inject custom drivers or configurations, you know it can be long and complex process. There are many tutorials on the internet including a recent post by Eric Sloof on injecting drivers into an ESXi installer. vibddi is meant to expedite the process and make it much simpler to inject custom drivers into an ESXi image.

****Disclaimer Since this tool is not very well documented and it is most likely not officially supported by VMware, please use test and validate the images generated prior to using in an production environment Disclaimer****

To run vibddi, you need to use sudo. Here are the available options:

[vi-admin@autodeploy ~]$ sudo vibddi -h
Password:

vibddi: Query and update vibs on a VMvisor dd image or device

Usage:
vibddi -h --- Print this

vibddi -i -q --- Query vibs installed on the image

vibddi -i -c --- Check bootbank filesystems on the image

vibddi -i -v [ -g ] [ -n ] --- Update the image with a single vib

vibddi -i -m -b [ -p ] [ -g ] [ -n ] --- Update the image with an online bulletin

vibddi -i -o [ -g ] [ -n ] --- Update the image with an offline bundle

vibddi -i -e [ -a ] --- Export boot modules from the image

vibddi -i -t --- Add/Remove a VMkernel option

vibddi -i -x --- Transform image to ThinESX format

vibddi -i -l --- Install a license file (vmware.lic) on the image

vibddi -d -q --- Query vibs installed on the device

vibddi -d -c --- Check bootbank filesystems on the device

vibddi -d -v [ -n ] --- Update the device with a single vib

vibddi -d -m -b [ -p ] [ -n ] --- Update the device with an online bulletin

vibddi -d -o [ -n ] --- Update the device with an offline bundle

vibddi -d -e [ -a ] --- Export boot modules from the device

vibddi -d -t --- Add/Remove a VMkernel option

vibddi -d -x --- Transform image to ThinESX format

vibddi -f -k --- Add a customized kickstart file to the ThinESX/Recovery CD ISO

Where:
VMvisor-dd - The VMvisor dd image that is going to be customized

VMvisor-dev - The VMvisor device that is going to be updated

vib-path - The local file path to the vib

metadata-URL - The URL to the metadata.zip file (Ex. http://www.oem.com/depot/metadata.zip)

bulletin-ID - The bulletin ID to install

bundle-path - The local file path to the offline bundle

proxy (OPTIONAL) - Proxy used to download vib, for update operation only

-g (OPTIONAL) - Generate customized ThinESX/Recovery CD ISOs

-n (OPTIONAL) - Bypass signature check, for update operation only

export-path - Directory to export boot modules

alternate-conf (OPTIONAL) - Alternate export configuration file

kernel-opt - VMkernel option

license-path - vmware.lic file (Format: 00000-00000-00000-00000-00000)

iso-path - The local file path to the ThinESX/Recovery CD ISO

kickstart-path - The local file path to the kickstart file

Here are a few examples of using the vibddi tool:
Mount ESXi 4.1 ISO to extract the DD image:

[vi-admin@autodeploy scratch]$ sudo mount -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/iso/

Unzip the DD image and extract to current directory:

[vi-admin@autodeploy scratch]$ sudo bunzip2 -c /mnt/iso/imagedd.bz2 > imagedd

You now should have the DD image called imagedd located in your current working directory.You can name the file anything you want, but I'm using the suggested name as noted in the Auto Deploy documentation.

To list vibs installed on the image, you'll use the following command:

sudo vibddi -i [imagedd] -q

Here is an example of the vibs installed with default installation of ESXi 4.1:

To inject the image with an offline bundle, you'll use the following command:

sudo vibddi -i [imagedd] -o [offline_bundle] -n

Note: The -n flag should be used when performing updates as it bypasses the signature checks, else you will get an error.

Here is an example of injecting the Cisco Nexus 1000 Virtual Ethernet Module offline bundle as part of the default ESXi 4.1 installation:

We can confirm the Cisco VEM is part of the default image by running the query command again:

To inject the image with a single VIB, you'll use the following command:

sudo vibddi -i [imagedd] -v [vib] -n

Here is an example injecting the Cisco Nexus 1000 Virtual Ethernet Module VIB as part of the default ESXi 4.1 installation:

To inject VMkernel boot parameters, you'll use the following command:

sudo vibddi -i [imagedd] -t [vmkernel_option]

Note: Here is a list of a few VMkernel options documented by Dave Mishchenko. The -t argument only accepts one VMkernel option at a time. If you want to updated more than one option, you will need to run the command for each VMkernel option.

With a default installation of ESXi 4.1, there are no VMkernel options defined. To see whether or not these have been defined, you will need to login to Tech Support Mode and view boot.cfg:

~ # cat bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- aam.vgz
build=4.1.0-260247
updated=1
bootstate=0

Here is an example of injecting the following two VMkernel options: noACPI and nopowerManagement:

To inject a license file, you'll use the following command:

sudo vibddi -i [imagedd] -l [license_file]

Note: The license file must contain a single entry using the following format - 00000-00000-00000-00000-00000

Here is an example of injecting license file:

To inject a custom kickstart configuration, you'll use the following command:

sudo vibddi -f [esxi-iso-path] -k [kickstart_file]

Here is an example of injecting a custom kickstart file:

Note: This actually injects a custom ks.cfg into the ESXi .iso which can then be used to deploy an ESXi host including the custom configurations found in the kickstart file. A brand new .iso will be created in the current working directory which includes the timestamp of kickstart injection as part of its filename.

We now can loop mount the new .iso and verify the custom kickstart has been injected:

Note: I'm using the sample ks.cfg found on Kendrick Colemans's site.

You can also extract certain items from the DD image, you'll use the following command:

sudo vibddi -i [imagedd] -e [export-path]

Here is an example of extracting the entire DD image to a temporarily directory:

To check the bootbank filesystem, you'll use the following command:

sudo vibddi -i [imagedd] -c

Here is an example of verifying bootbank filesystem:

Once the imagedd has been updated with all the drivers, you will need to compress the image back to .bz2 using bzip2. From here, you will have two options: A) copy the modified imagedd.bz2 over to your PXE/TFTP server used for automated kickstart installation B) Create a new ESXi .iso, there are a bunch of tutorials online such as here and here.

If you need to troubleshoot or would like to view the process of vibddi, you can take a look at the logs stored in /var/log/vibddi.log. You can also see the injection process which includes both informational and debugging logs in /var/log/esxupdate.log.

As you can see, this tool is extremely useful for injecting and customizing ESXi images. Hopefully one day VMware will officially release this tool and make it available on both UNIX/Linux and Windows platform so that everyone can benefit. For now, if you want to use vibddi, you will need to download and use Auto Deploy appliance. Looks like I'll be keeping this appliance around 😉

How to obtain GID and LWID from esxtop?

by // 11 Comments

Last week I remember seeing a tweet from Duncan Epping regarding the use of VMware vsish to obtain the VMX Cartel ID, also known as the LWID (Leader World Id) for a virtual machine world. This VMX Cartel ID is then used to obtain the GID (Group Id) which was documented by Duncan on his esxtop page to limit the view in esxtop to a particular VM of interest:

VMWID=`vm-support -x | grep |awk '{gsub("wid=", "");print $1}'`
VMXCARTEL=`vsish -e cat /vm/$VMWID/vmxCartelID`
vsish -e cat /sched/memClients/$VMXCARTEL/SchedGroupID

As you can see from the above, Duncan utilizes the vm-support to obtain the WID (World ID) and then using vsish to query for the VMX Cartel ID (LWID). Finally, using LWID, he was able to obtain the GID (Group ID). This example only applies to ESXi, since classic ESX does not include vsish tool.

Here is an example of where to locate the values in esxtop under CPU section:

I remember during some of my skunk work adventures, there were other methods of obtaining these IDs. Due to my VCAP-DCD cramming last week, I did not get a change to further investigate. Now that I have some time, I thought share some of these other methods for obtaining the GID and LWID in both ESX and ESXi.

ESXi

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

~ # /sbin/vmdumper -l
wid=16881 pid=-1 cfgFile="/vmfs/volumes/4cdeeb09-1ad4c18a-5ff9-003048d9586a/scofield/scofield.vmx" uuid="42 34 e6 bc f1 83 c2 db-fb fb 08 73 b7 0c 26 40" displayName="scofield" vmxCartelID=16880

ps - The VMX Cartel ID is actually both the PID and CID within process status in ESXi

~ # ps -Cc | grep "scofield" | grep -v grep | awk '{print $2}'
16880

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

~ # /sbin/vscsiStats -l | grep "scofield" | awk '{print $4}' | sed 's/,//g'
16880

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

~ # /sbin/esxcli vms vm list | grep -A3 "scofield" | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
16880

Obtaining GID

Method1 - Using vscsiStats and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method2 - Using esxcli and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method3 - Using esxcli and vsish to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');/sbin/vsish -e cat /sched/memClients/${VMX_CARTEL}/SchedGroupID
4639

Note: You just need to substitute VM_NAME variable with the display name of of the virtual machine you are interested in. There are actually multiple commands being executed in this one line. If your VM has spaces, make sure you put quotes around it

ESX

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

[root@himalaya ~]# /usr/lib/vmware/bin/vmdumper -l
wid=29250 pid=-1 cfgFile="/vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/STA202G/STA202G.vmx" uuid="42 30 d1 75 c5 3e 81 2a-14 15 1f 86 bb 5b b9 e5" displayName="STA202G" vmxCartelID=29249

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

[root@himalaya ~]# /usr/lib/vmware/bin/vscsiStats -l | grep "STA202G" | awk '{print $4}' | sed 's/,//g'
29249

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

[root@himalaya ~]# /usr/sbin/esxcli vms vm list | grep -A3 STA202G | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
29249

Obtaining GID

Method1 - Using esxcli and esxcfg-info to extract VM's GID

[root@himalaya ~]# VM_NAME=STA202G ;VMX_CARTEL=$(/usr/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
197

Method2 - Using sched-stats to extract GID

[root@himalaya ~]# VM_NAME=STA202G;VMX_CARTEL=$(/usr/lib/vmware/bin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');/usr/bin/sched-stats -t groups | grep "vm.${VMX_CARTEL}" | awk '{print $1}'
197

Method3 - Using /proc information from sched

[root@himalaya ~]# VM_NAME=STA202G ;grep "${VM_NAME}" /proc/vmware/sched/drm-stats | awk '{print $1}'
197

As you can see, there are more than one way to obtain the same exact values and I am sure there are probably a few others. For command simplicity, I would probably recommend method #3 for ESXi and method #3 for ESX. As with anything, be careful when you using these methods as they are not really supported by VMware unless directed by their support organization.

How to configure and use vMA's vi-fastpass with fpauth and adauth on vSphere 4.1

by // 7 Comments

From time to time, I see users posting on the VMTN forums with some questions and confusion around the proper implementation and functionality of vMA's vi-fastpass. The confusion is further enhanced with the new Active Directory functionality and integration with vMA's new vi-fastpass type called adauth.

The vi-fastpass component found in vMA is a credentials caching mechanism to allow you to connect to your ESX(i) or vCenter servers. Prior to vMA 4.1, vMA 4.0 only supported one type of vi-fastpass which is just called fpauth (fastpass authentication). This fpauth basically allows you to manage an ESX(i) or vCenter server under vMA by creating a vi-adminXX and vi-userXX account. The password for these two accounts are obfuscated using a simple XOR cipher. A user can now initialize one of these managed targets and execute either vCLI or vSphere SDK for Perl scripts without having to specify credentials each and every time, this works because the vi-adminXX credentials are being used to connect to your target. This can make running a simple command across n-number of hosts simple without having to provide the credentials for every host.

[Read more...]