WilliamLam.com

What is VMware vsish?

by // 20 Comments

Recently while I was working on testing automated ESXi kickstart installations, I needed to extract some information as part of the build process, but the utilities that were used no longer existed in ESXi's Busybox console. Looking around, I found another way to extract the information I needed, which was using VMware's undocumented vsish utility, also known as the VMkernel Sys Info Shell. There is not much information around the web regarding this vsish utility (probably for good reason), but it has been described by few as a representation of classic Service Console /proc nodes and allows the ability to extract system reliability information similar to mcelog in Linux.

If I recall correctly, the vsish utility used to be bundled with classic ESX, but at some point it was removed. However, in ESXi, the utility is included and that is also true for latest release of ESXi 4.1. When you generate vm-support log, a dump of the vsi nodes are generally included which provides VMware support with the state of your system. To read the vsi node dump, you need a matching version of the vsish utility based on the version of ESX or ESXi you are running. Currently for classic ESX, to obtain the vsish utility, it must be provided by VMware support via debugging package that needs to be installed.

Note: A word of caution before using this utility, you should not make any changes that you are unfamiliar with. Always consult with VMware support before making changes as it can severely impact your host and virtual machines. Okay, now on to the fun stuff 🙂

On ESXi Busybox Console, you can launch the vsish utility by just typing "vsish":

You can perform various operations such as listing the various nodes, get and set parameters, etc. to see the available options, just type "help":

There is a huge amount of information that can be retrieved from vsish. One interesting leaf node within vsish is called "config", this actually maps to the Advanced Settings found on an ESX(i) host:

As you can see, the majority of the sub leaf nodes within "config" is exposed in the Advanced Settings, but there are some that are hidden. In fact, with ESX(i) 4.1, there is a total of 771 configurable options with 250 of those hidden that can only be seen using vsish (more on the configuration option later)!

There are two ways to interact with vsish, you can interactively login to VSI shell and perform ls, get or set operations or perform the same operation through non-interactive mode.

Here is an example of an interactive session listing config's under "COW" leaf node and getting and setting the value for "COWDiskSizeIncrement" which is one of the 250 hidden configuration options:

Here is an example of an non-interactive session performing the same operation as the one listed above:

With the release of vSphere 4.1, there have been a few new additions to the VSI nodes. Here is an excerpt slide from vSphere 4.1 to 4.0 differences presented by Iwan Rahabok - Senior Systems Consultant at VMware listing some of the new Storage I/O Control features:

Here is a screenshot on some of these values if you can not make it out in the slide:

For more details on differences between vSphere 4.0 and vSphere 4.1 - Check out the detailed 2 part deep-dive power point presentation here.

vsish provides an enormous amount of information and I have only begin to scratch the surface. Having said that I did manage to capture all the advanced host settings which includes both public and hidden options. Using a few for loops and some shell scripting, I have generated the following two lists:

Complete vSphere ESXi 4.1 vsish configurations including hidden options - 771 Total:

For the complete list, take a look at https://s3.amazonaws.com/virtuallyghetto-download/complete_vsish_config.html

Hidden vSphere ESXi 4.1 vsish configurations only - 250 Total:

For the hidden list only, take a look at https://s3.amazonaws.com/virtuallyghetto-download/hidden_vsish_config.html

There are definitely some interesting options that can be configured and I can see why VMware would want to hide these from the general public. What is nice about the compiled output, is that it clearly states the path to the configuration item, the current, default, min, max, is hidden or description of the parameter. Again, use at your own risk Hopefully this these two documents will be useful for curious users to explore vsish advanced configs.

Using the table above, you can actually query and modify these values using the standard esxcfg-advcfg utility that exists both on ESX(i). The following example will show you how to translate the vsish node path to proper format that is required for local esxcfg-advcfg utility.

In the example, we will be using vsish path "/config/COW/intOpts/COWDiskSizeIncrement"

Using a local copy of esxcfg-advcfg on ESX or ESXi, you will need to convert the above to the following:

[root@himalaya ]# esxcfg-advcfg -g /COW/COWDiskSizeIncrement
Value of COWDiskSizeIncrement is 32768

Note: You just need to extract the root node and the individual config leaf node in "/config/COW/intOpts/COWDiskSizeIncrement" which is highlighted in green and blue.

Some other interesting nodes that I found that might be useful are under /system:
/system/bootOption
/system/bootCmdLine
/system/systemUuid
/system/bootFsUUID

One other interesting tidbit of information that I found was under /system/version which actually shows the build date and time of vSphere ESXi 4.1:

Now that we have a better understanding of the vsish utility, how does this help with my original inquiry? I found that you can extract networking information from your vNICs by looking at /net/tcpip/* nodes:

As you can see from the output above, it's format is in hexidecimal but when converted, you will get the IP Address, netmask and the gateway for a given VMkernel interface. To accomplish this, I used a modified python script to convert these entries to their human readable addresses:

Here some additional links referencing vsish that may be of interest:

How to kill a stuck VM on ESX(i) 4.1 using esxcli

by // 1 Comment

With so many new features in vSphere 4.1, you might have not noticed the new additions to the esxcli utility. One new feature is the ability to remotely kill a stuck or hung virtual machine running on ESX or ESXi. Prior to vSphere 4.1, the process of killing a stuck VM meant you had to use a variety of methods and tools that included vmware-cmd, vm-support, vmkload_app and ps/kill and required the use of either Service Console on ESX or Tech Support Mode for ESXi. Depending on how "stuck" your VM was, you would start from the most gentle method to killing your VM with prejudice (kill -9 [pid]).

A new namespace called "vms" has now been added and there are two available operations: list running VMs on a host or killing a specific VM. To kill a VM, you will need to specify the kill type and VM's world id.

Here is an example of listing all running VMs on an ESXi host:

When a VM gets into a stuck state in which you have no other option than to forcefully stop the VM, you will have three types of options to kill a VM: soft, hard and force. Per the help command, you should always perform the kill operations in the following order:

soft - Will give the VMX process a chance to shutdown cleanly (like kill or kill -SIGTERM)
hard - Will shutdown the process immediately (like kill -9 or kill -SIGKILL)
force - Should be used as a last resort attempt to kill the VM

The final resort, is to reboot the host if none of the above work.

Here is an example of various kill types:

Here is an example of killing a VM using the "soft" method, you should see the value of "true" if the operation was successful.

This was one feature users have been asking for with regards to managing VMs on ESXi, where the Service Console was no longer available. You do not need to login to Tech Support Mode to forcefully kill a VM and from what I have been told, if the above does not work, it will not work in Tech Support Mode either. I am hoping a public API for esxcli operations will be exposed which can be leverage by the various vSphere SDK's (e.g. vSphere SDK for Perl, PowerCLI, VI Java, etc.)

How to backup VMs in ESX onto Google Storage

by // 4 Comments

Over the weekend I received an email invite for Google Storage for Developers, which I had applied for a month or two back. I did not think I would get into the limited beta and at the time I was just curious about the offering from Google. Back in January I wrote an article How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud which uses Amazon's S3 storage offering and their s3cmd to upload a VM into the cloud. I spent a little time over the weekend playing with Google's storage offering and was able to install Google's python gsutil utility on to an ESX 4.1 host and backup a VM to Google Storage.

Before you begin, you will to have access to Google Storage for Developers and you will need to download the gsutil and transfer the tarball to your ESX host. The test was performed on an ESX 4.1 host, but I do not see why it will not work on ESX 4.0.

1. Extract the contents of gsutil.tar.gz

tar -zxvf gsutil.tar.gz

2. Edit .bashrc

vi .bashrc

Add the following two lines:

By default, gsutil requires you to be running python 2.5.1 or greater. ESX 4.1 only comes with python 2.4.3. What I found is that for the basic operations, you actually do not need 2.5.1 and you can edit the gsutil utility to not check for the version.

3. Edit gsutil located in /root/gsutil/gsutil and change the following on line 1137
From:
To: 4. Login to Google Storage for Developers, you will need to setup key management under "Google Storage Manager" tab. This will be necessary for the next step.

5. For the first time, you will need to run gsutil to setup boto configuration file which will contain your Google access key ID and secret.

[root@esx4-1 ~]# gsutil ls -l
You have no boto config file. This script will create one at /root/.boto containing your credentials, based on your responses to the following questions. What is your Google access key ID? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX What is your Google secret access key? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Configuration file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. Please try running gsutil again now.

Once you have successfully provided the correct keys, you are now ready to use Google Storage from your ESX host.

To start using Google Storage, you will need to first create a bucket:

[root@esx4-1 ~]# gsutil mb gs://ghettovcb

Note: The bucket name must be all lower case, else you will get an error

We can now list the bucket that was just created:

[root@esx4-1 ~]# gsutil ls -l gs://ghettovcb/: ACL: [Owner:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, [UserById: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY]: u'FULL_CONTROL']

We can also see the new bucket using Google Storage Manager browser:

Let's say we have a Virtual Machine on the ESX host called "dummyVM" and we wanted to upload to Google Storage. We will want to compress the directory. [root@esx4-1 esx4-1-local-storage-1]# ls dummyVM/ dummyVM-flat.vmdk dummyVM.vmdk dummyVM.vmsd dummyVM.vmx dummyVM.vmxf [root@esx4-1 esx4-1-local-storage-1]# tar -zcf dummyVM.tar.gz dummyVM/
Upload to Google Storage:

[root@esx4-1 esx4-1-local-storage-1]# gsutil cp dummyVM.tar.gz gs://ghettovcb

We can verify by listing the contents of the bucket:

[root@esx4-1 esx4-1-local-storage-1]# gsutil ls gs://ghettovcb gs://ghettovcb/dummyVM.tar.gz

We can also verify by viewing the bucket on Google Storage Manager:

As you can see, it is pretty easy to manage your Google Storage using gsutil utility. For more information on other commands, take a look at Getting Started Guide for Google Storage. I was not able to get gsutil working on ESXi 4.1, because of missing python modules that were needed. I spent some time trying to copy existing modules from class ESX 4.1, but it did not resolve all dependencies.