WilliamLam.com

Poor man's traceroute for ESXi

by // 1 Comment

There was a question today on the VMTN community forum about traceroute and ESXi. The user was troubleshooting a networking issue and logging into Tech Support Mode (Busybox Console) of ESXi, had noticed this common networking utility was not available. Since the user was working in a production environment, a re-compilation of a custom Busybox applet was out of the question and most likely unsupported by VMware. The user had to explore other avenues of troubleshooting, or did they have to?

It may not be well known, but within the Busybox Console, a Python interpreter is installed and accessible with a limited set of modules. Armed with this fact, you can actually create your own "poor man's traceroute" which gave you functionality similar to that of the actual traceroute/tracert implementation. This "poor man's" version was dubbed by a blog article I had found online (sorry, no ghetto in this name) by Leonid Grinberg - Learning by doing: Writing your own traceroute in 8 easy steps. The author goes into extreme detail providing step by step instructions on his python implementation of traceroute. Leonid provides a fully functional script at the end of his blog post, but he does caveat the following statement at the end:

This is actually not quite how the real traceroute works. Rather than checking the IP addresses of the hosts and stopping when the destination address matches, it stops when it receives a ICMP “port unreachable” message, which means that the host has been reached. For our purposes, though, this simple address heuristic is good enough.

This was definitely another alternative if one needed functionality similar to that of traceroute. I used the author's polished version of traceroute.py and scp'ed the script onto an ESXi host. It ran just as well as the real traceroute on the classic ESX Service Console.

Though this may not have solved the original network issue, it is important to know what utilities you have at your disposal, even if it is not the actual tool that you are familiar with. There is always more than one way to do something and it is always good to know a few.

VMware does limit the number of "common" UNIX/Linux utilities that are bundled within the Busybox applet and based on demand and feature requests, we may see future utilities added to Tech Support Mode. One example is the release of vSphere 4.1, we saw the addition of nc (netcat) and tcpdump which are invaluable troubleshooting utilities for a vSphere admin. If there are other utilities that you feel should be included in the default Busybox applet, be sure to submit your feedback to either your VMware rep or file a feature request.

How to inject custom drivers into an ESXi 4.1 image using vibddi?

by // 11 Comments

Over the holiday break, I spent some time cleaning up some of the development virtual machines in our ghettoDatacenter. I came across the VMware Auto Deploy appliance that I deployed awhile back ago. I did not think I had a use for it since we already have an automated deployment system using PXE and kickstart. Auto Deploy was launched relatively recently from the VMware Flings lab. It was originally slated for release as part of vSphere 4.1 but during the transition from the BETA to RC, it was dropped and never made it into the GA release of vSphere 4.1

I decided to give the documentation one last read before deleting and to my surprise, I stumbled across an interesting gem, vibddi. vibddi (pronounced vib d-d-i) stands for VIB (vSphere Instalaltion Bundle) Disk Dump Image, which is actually a Perl utility that was created to help customize ESXi images more easily.

If you ever had a need to customize an ESXi image and inject custom drivers or configurations, you know it can be long and complex process. There are many tutorials on the internet including a recent post by Eric Sloof on injecting drivers into an ESXi installer. vibddi is meant to expedite the process and make it much simpler to inject custom drivers into an ESXi image.

****Disclaimer Since this tool is not very well documented and it is most likely not officially supported by VMware, please use test and validate the images generated prior to using in an production environment Disclaimer****

To run vibddi, you need to use sudo. Here are the available options:

[vi-admin@autodeploy ~]$ sudo vibddi -h
Password:

vibddi: Query and update vibs on a VMvisor dd image or device

Usage:
vibddi -h --- Print this

vibddi -i -q --- Query vibs installed on the image

vibddi -i -c --- Check bootbank filesystems on the image

vibddi -i -v [ -g ] [ -n ] --- Update the image with a single vib

vibddi -i -m -b [ -p ] [ -g ] [ -n ] --- Update the image with an online bulletin

vibddi -i -o [ -g ] [ -n ] --- Update the image with an offline bundle

vibddi -i -e [ -a ] --- Export boot modules from the image

vibddi -i -t --- Add/Remove a VMkernel option

vibddi -i -x --- Transform image to ThinESX format

vibddi -i -l --- Install a license file (vmware.lic) on the image

vibddi -d -q --- Query vibs installed on the device

vibddi -d -c --- Check bootbank filesystems on the device

vibddi -d -v [ -n ] --- Update the device with a single vib

vibddi -d -m -b [ -p ] [ -n ] --- Update the device with an online bulletin

vibddi -d -o [ -n ] --- Update the device with an offline bundle

vibddi -d -e [ -a ] --- Export boot modules from the device

vibddi -d -t --- Add/Remove a VMkernel option

vibddi -d -x --- Transform image to ThinESX format

vibddi -f -k --- Add a customized kickstart file to the ThinESX/Recovery CD ISO

Where:
VMvisor-dd - The VMvisor dd image that is going to be customized

VMvisor-dev - The VMvisor device that is going to be updated

vib-path - The local file path to the vib

metadata-URL - The URL to the metadata.zip file (Ex. http://www.oem.com/depot/metadata.zip)

bulletin-ID - The bulletin ID to install

bundle-path - The local file path to the offline bundle

proxy (OPTIONAL) - Proxy used to download vib, for update operation only

-g (OPTIONAL) - Generate customized ThinESX/Recovery CD ISOs

-n (OPTIONAL) - Bypass signature check, for update operation only

export-path - Directory to export boot modules

alternate-conf (OPTIONAL) - Alternate export configuration file

kernel-opt - VMkernel option

license-path - vmware.lic file (Format: 00000-00000-00000-00000-00000)

iso-path - The local file path to the ThinESX/Recovery CD ISO

kickstart-path - The local file path to the kickstart file

Here are a few examples of using the vibddi tool:
Mount ESXi 4.1 ISO to extract the DD image:

[vi-admin@autodeploy scratch]$ sudo mount -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/iso/

Unzip the DD image and extract to current directory:

[vi-admin@autodeploy scratch]$ sudo bunzip2 -c /mnt/iso/imagedd.bz2 > imagedd

You now should have the DD image called imagedd located in your current working directory.You can name the file anything you want, but I'm using the suggested name as noted in the Auto Deploy documentation.

To list vibs installed on the image, you'll use the following command:

sudo vibddi -i [imagedd] -q

Here is an example of the vibs installed with default installation of ESXi 4.1:

To inject the image with an offline bundle, you'll use the following command:

sudo vibddi -i [imagedd] -o [offline_bundle] -n

Note: The -n flag should be used when performing updates as it bypasses the signature checks, else you will get an error.

Here is an example of injecting the Cisco Nexus 1000 Virtual Ethernet Module offline bundle as part of the default ESXi 4.1 installation:

We can confirm the Cisco VEM is part of the default image by running the query command again:

To inject the image with a single VIB, you'll use the following command:

sudo vibddi -i [imagedd] -v [vib] -n

Here is an example injecting the Cisco Nexus 1000 Virtual Ethernet Module VIB as part of the default ESXi 4.1 installation:

To inject VMkernel boot parameters, you'll use the following command:

sudo vibddi -i [imagedd] -t [vmkernel_option]

Note: Here is a list of a few VMkernel options documented by Dave Mishchenko. The -t argument only accepts one VMkernel option at a time. If you want to updated more than one option, you will need to run the command for each VMkernel option.

With a default installation of ESXi 4.1, there are no VMkernel options defined. To see whether or not these have been defined, you will need to login to Tech Support Mode and view boot.cfg:

~ # cat bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- aam.vgz
build=4.1.0-260247
updated=1
bootstate=0

Here is an example of injecting the following two VMkernel options: noACPI and nopowerManagement:

To inject a license file, you'll use the following command:

sudo vibddi -i [imagedd] -l [license_file]

Note: The license file must contain a single entry using the following format - 00000-00000-00000-00000-00000

Here is an example of injecting license file:

To inject a custom kickstart configuration, you'll use the following command:

sudo vibddi -f [esxi-iso-path] -k [kickstart_file]

Here is an example of injecting a custom kickstart file:

Note: This actually injects a custom ks.cfg into the ESXi .iso which can then be used to deploy an ESXi host including the custom configurations found in the kickstart file. A brand new .iso will be created in the current working directory which includes the timestamp of kickstart injection as part of its filename.

We now can loop mount the new .iso and verify the custom kickstart has been injected:

Note: I'm using the sample ks.cfg found on Kendrick Colemans's site.

You can also extract certain items from the DD image, you'll use the following command:

sudo vibddi -i [imagedd] -e [export-path]

Here is an example of extracting the entire DD image to a temporarily directory:

To check the bootbank filesystem, you'll use the following command:

sudo vibddi -i [imagedd] -c

Here is an example of verifying bootbank filesystem:

Once the imagedd has been updated with all the drivers, you will need to compress the image back to .bz2 using bzip2. From here, you will have two options: A) copy the modified imagedd.bz2 over to your PXE/TFTP server used for automated kickstart installation B) Create a new ESXi .iso, there are a bunch of tutorials online such as here and here.

If you need to troubleshoot or would like to view the process of vibddi, you can take a look at the logs stored in /var/log/vibddi.log. You can also see the injection process which includes both informational and debugging logs in /var/log/esxupdate.log.

As you can see, this tool is extremely useful for injecting and customizing ESXi images. Hopefully one day VMware will officially release this tool and make it available on both UNIX/Linux and Windows platform so that everyone can benefit. For now, if you want to use vibddi, you will need to download and use Auto Deploy appliance. Looks like I'll be keeping this appliance around 😉

Does SIOC actually require Enterprise Plus & vCenter Server?

by // 1 Comment

After reading a recent blog post by Duncan Epping, SIOC, tying up some loose ends, I decided to explore whether or not VMware's Storage I/O Control feature actually requires an Enterprise Plus license and vCenter Server. To be completely honest, Duncan's article got me thinking but it was also my recent experience with VMware's vsish and the blog post I wrote What is VMware vsish? that made me think this might be a possibility. vsish is only available on ESXi 4.1 within the Tech Support Mode, but if you have access to debugging rpm from VMware, you can also obtain vsish for classic ESX.

Within vsish, there is a storage section and within that section there is a devices sub-section which provides information regarding your storage devices that includes paths, partitions, IO statistics, queue depth and new SIOC state information.

Here is an example of the various devices that I can view on an ESXi 4.1 host:

~ # vsish
/> ls /storage/scsifw/devices/
t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/
mpx.vmhba1:C0:T0:L0/
mpx.vmhba32:C0:T0:L0/

Here is an example of various properties accessible to a given storage device:

/> ls /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/
worlds/
handles/
filters/
paths/
partitions/
uids/
iormInfo
iormState
maxQueueDepth
injectError
statson
stats
inquiryVPD/
inquirySTD
info

In particular, we are interested in iormState and you can see the value by just using the cat command:

/> cat /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState
1596

This value may not mean a whole lot and I have seen this as the default value when SIOC is disabled as well as 2000 from my limited set of tests. Now, since we can access these particular SIOC parameter, I wanted to see how this value was affected when SIOC is enabled and disabled. To test this, I used the following VMware KB1022091 to enabling additional SIOC logging which goes directly to /var/log/messages with the logger tag "storageRM", this allows you to easily filter out SIOC logs via simple grep.

For testing purposes, you can just enable the logging to level 2 which is more than sufficient to get the necessary output. You will perform the following command to change the default SIOC logging level from 0 to 2 using Tech Support Mode:

~ # esxcfg-advcfg -s 2 /Misc/SIOControlLogLevel
Value of SIOControlLoglevel is 2

Now you will want to open a separate SSH session to your ESXi host and tail /var/log/messages to monitor the SIOC logs:

~ # tail -f /var/log/messages | grep storageRM
Oct 10 18:39:05 storageRM: Number of devices on host = 3
Oct 10 18:39:05 storageRM: Checked device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 iormEnabled= 0 LatThreshold =30
Oct 10 18:39:05 storageRM: Checked device mpx.vmhba1:C0:T0:L0 iormEnabled= 0 LatThreshold =232
Oct 10 18:39:05 storageRM: Checked device mpx.vmhba32:C0:T0:L0 iormEnabled= 0 LatThreshold =30
Oct 10 18:39:05 storageRM: rateControl: Current log level: 2, new: 2
Oct 10 18:39:05 storageRM: rateControl: Alas - No device with IORM enabled!

You should see something similar to the above. In my lab, it is seeing an iSCSI volume, local storage and CD-ROM and you will notice there is an iormEnabled flag and all three has SIOC disabled and the default latency threshold specified by LatThreshold, which is 30ms by default.

Now we know what these values are when SIOC is disabled, let's see what happens when we enable SIOC from vCenter on this ESXi 4.1 host. I am using evaluation license for the host which supports the Storage I/O Control from a licensing perspective. After enabling SIOC and using the default 30ms on my iSCSI volume, I took a look at the SIOC logs and saw that there were some changes in the logs:

~ # tail -f /var/log/messages | grep storageRM
Oct 10 18:48:56 storageRM: Number of devices on host = 3
Oct 10 18:48:56 storageRM: Checked device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 iormEnabled= 1 LatThreshold =30
Oct 10 18:48:56 storageRM: Found device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 with datastore openfiler-iSCSI-1
Oct 10 18:48:56 storageRM: Adding device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 with datastore openfiler-iSCSI-1

As you can see now, SIOC is enabled and the iormEnabled flag has changed from 0 to 1. This should not be a surprise, now let's take a look at vsish storage property and see if that has changed:

~ # vsish -e get /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState
1597

If you recall from the previous command above, the default value was 1596 and after enabling SIOC, the value has incremented by one. I found this to be an interesting observation and I tried a few other configurations including enabling SIOC on local storage and found that this value was always incremented by 1 if SIOC was enabled and decrement or kept the same if SIOC is disabled.

As you may or may not know, SIOC does not use vCenter, it is only required when enabling the feature and from this simple test, this looks to be the case. It is also important to note, as pointed out by Duncan in his blog post that the latency statistics is stored in .iormstats.sf file which is stored within each of the VMFS datastores that has SIOC enabled. Putting all this together, I hypothesize that Storage I/O Control could actually be enabled without an Enterprise Plus license and without vCenter Server.

The test utilized the following configuration:

  • 2 x virtual ESXi 4.1 hosts licensed as free ESXi (vSphere 4.1 Hypervisor)
  • 1 x 50GB iSCSI volume exported from Openfiler VM
  • 2 x CentOS VM installed with iozone to generate some IO

Here is a screenshot of the two ESXi 4.1 free licensed hosts displaying the licensed version and each running CentOS VM residing on this shared VMFS iSCSI volume:

I configured vm1 which resides on esxi4-1 and set the disk shares to Low with default value of 500 and configured vm2 which resides on esxi4-4 and set the disk shares to High with default value of 2000:

I then ran iozone a filesystem benchmark tool on both CentOSes VM which is generating some amount of IO on the single VMFS volume shared by both ESXi 4.1 hosts:

I then view the SIOC logs on both ESXi 4.1 in /var/log/vmkernel and tail the IO statistics for the VMFS iSCSI datastore using vsish:

Note: The gray bar tells you which host you are viewing the data for which is being displayed by using GNU Screen. The first two screen displays the current status of SIOC which is currently disabled and the second two screen displays the current device queue depth which in my lab environment defaulted to 128, by default it should be 32 as I remember.

Now I enable SIOC on both ESXi 4.1 hosts using vsish and perform the command:

~ # vsish -e set /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState 1597

Note: Remember to run a "get" operation to check the default value and you will just need to increment by one to enable SIOC. From my testing, the default value will either be 1596 or 2000 and you will change it to 1597 or 2001 respectively.

You can now validate that SIOC is enabled by going back to your SSH session and verify the logs:

As you can see, the iormEnabled flag has now changed from 0 to 1, which means SIOC is now enabled.

If you have running virtual machines on the VMFS volume and SIOC is enabled, you should now see a  iormstats.sf latency file stored in the VMFS volume:

After awhile, you can view IO statistics via vsish to see what the device queue is currently configured to and slowly see the throttle based on latency. For this particular snapshot, you can see the vm1 was configured with "high" disk share and vm2 was configured with "low" disk share and there is a large queue depth for the very bottom ESXi host versus the other host which only has a smaller queue depth.

Note: During my test I did notice that the queue depth was dramatically decreased from 128 and even from 32 to single digits. I am pretty sure it was due to the limited amount of resources in my lab that some of these numbers were a little odd.

To summarize, it seems that you can actually enable Storage I/O Control without the use of vCenter Server and an Enterprise Plus license, however, this requires the use of vsish which is only found on ESXi 4.1 and not on classic ESX 4.1. I also found that if you enabled SIOC via this method and joined your host to vCenter Server, it is not aware of these changes and marks SIOC as disabled even though the host actually has SIOC enabled. If you want vCenter to see the update, you will need to enabled SIOC via vCenter.

I would also like to thank Raphael Schitz for helping me validate some of my initial findings.

Update: I also found hidden Storage I/O Control API method called ConfigureDatastoreIORMOnHost which allows you to enable SIOC directly on the host, which validates the claim from above that this can be done directly on an ESX or ESXi host.