WilliamLam.com

CIM monitoring caveat with ESXi

by // 9 Comments

I recently started to play around with the CIM API to monitor the hardware on an ESXi host. Instead of relying on hardware agents/scripts that would normaly run in the traditional Service Console of ESX, CIM is an API that allows you to monitor the health of your ESX or ESXi host. You can see the health status of your host by logging directly into the ESX(i) host using the vSphere Client and clicking on Configuration->Health Status tab.

I decided to start off with a small python script that would run on vMA and using the wbem python module to make a simple connection to query for the ESXi version. Here are the steps to get the following script working:

1. Download pywbem onto vMA

2. Extract the contents of pywbem

tar -zxvf pywbem-0.7.0.tar.gz

3. Install pywbem

sudo python setup.py install

4. The script expects the hostname/IP of your ESX(i) host as it's first argument and the username as the second argument and then you will be prompted for the password

./cim.py himalaya.primp-industries.com root

If everything went according plan, you should see the version of your ESX(i) host printed on the screen.

Next I wanted to create a dedicated service account so that I do not have to use the root account, I thought a read-only role would suffice.

To my surprise, when I ran the script again with this user, I received an unauthorized access error.

At first I thought the user account required an "Administrator" role to perform the operation but after further investigation, I found that the user account must be part of the "root" user group. Even for a read operation, it still needed to be in that that group.

After I made the change using the vSphere Client and re-ran the script, it executed as I expected.

After speaking to someone at VMware regarding this issue, it was confirmed by engineering that this is in fact a software bug and it should not require the user account to be part of the "root" user group to query from the CIM API.

I was still interested in using CIM, but I wanted to lock down the account as much as possible and came up with the following snippet of code which can be included in your ESX(i) kickstart configuration.

The script creates a regular user who does NOT have login access to the ESX(i) host. It then puts the user into the "root" user group and then creates a new role called CIM with a single privilege Host.Cim.CimInteraction and then associates this user with this role.This ensures that the account can only perform read-only operations against the CIM API and does not allow for host logins. Until the bug is resolved, this should be an acceptable work around.

So what type of monitoring can you do with CIM? Well pretty much anything and everything. There is a popular Nagios script that monitors the hardware health of an ESXi host using the CIM API called check_esxi_webem.py that one can implement to alert on your hardware components.

The script currently expects three arguments: hostname/IP of ESX(i) host, username and password on the command line (this can be changed with minor modifications). If you run it using those defaults, you will either get an OK or WARN/ERROR which will include additional information about the component that is alarming.

If you would like to get more details on the components being checked, you can pass in a fourth parameter called "verbose" and the script will provide more information on what is being checked.

If you are not big on python, there is also a Perl SDK for CIM/WSMAN as part of the vCLI installation and if you are using vMA, you can find some great examples under /usr/share/doc/vmware-vcli/samples/WSMan

The checksensorhealth.pl is definitely one to take a look at, here is an example output:

If you are interested to learn more about CIM, take a look at the these resources:
CIM SDK
7 Part series on ESXi Chronicles blogs about CIM and hardware monitoring

How to mount a cdrom using vsish on ESXi

by // 17 Comments

While browsing the VMTN community forums today, I noticed a very interesting post about mounting the CD-ROM device from within ESXi. The solution involved the use of vsish as the traditional mount utility did not function as expected in the Busybox Console of ESXi. I thought this was a very clever solution and might be worth sharing for those that may have similar needs.

Note: This trick looks like it only works with ESXi 4.1, as previous releases of ESXi may not include the iso9660 VMkernel module.

First you will need to load a VMkernel module: iso9660 which will allow you to mount and access the CD-ROM device.

You will need to run the following command:

vmkload_mod iso9660

You should see a successful message after the module has loaded:

Note: To see a list of VMkernel modules that can be loaded/unloaded, take a look at /usr/lib/vmware/vmkmod

Next you will need to identify the path to your CD-ROM device, you can do so by using the esxcfg-mpath utility.

You can run the following shorthand command to locate the path:

esxcfg-mpath -b | grep "CD-ROM"

You should see something in the form of mpx.*:

Now we will use vsish to perform the mount operation. Before we get started, if you did not perform the VMkernel module load of iso9660, you would not see the following path in /vmkModules/iso9660. Once the module has been loaded, you can perform a "ls" (listing) of the operations supported by this module which is mount and umount. You will need to run a "set" operation on the "mount" command and specify the device in which you would like to mount.

You will need to run the following command:

vsish -e set /vmkModules/iso9660/mount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

If you do not see any messages after executing the command, it was successful, else you may see an error/warning for incorrect syntax.

The CD-ROM device will automatically be mounted under /vmfs/volumes/mpx.* and there will also be a symlink with the CD-ROM label. In this example, I mounted an ESXi 4.1 Update 1 ISO file.

We can change into the directory to confirm we actually have the CD-ROM device mounted and list the contents.

Once you are done with your task, to umount is the same syntax except you will be using the "umount" operation. You can unmount by using the following command:

vsish -e set /vmkModules/iso9660/umount $(esxcfg-mpath -b | grep "CD-ROM" | awk '{print $1}')

Another alternative is to mount the image on another system and copy the contents to your ESXi host, but if you only had your ESXi host and needed to pull something from a CD-ROM, this is how you would do it.

Thanks again to agodwin for sharing this tidbit.

How to automatically add ESX(i) host to vCenter in Kickstart

by // 65 Comments

While recently updating my Automating Active Directory Domain Join in ESX(i) Kickstart article, it reminded me an old blog post by Justin Guidroz who initially identified a way to add an ESXi host to vCenter using python and the vSphere MOB. The approach was very neat but was not 100% automated as it required some user interaction with the vSphere MOB to identify certain API properties before one could potentially script it within a kickstart installation.

I decided to revisit this problem as it was something I had investigated awhile back. There are numerous ways on getting something like this to work in your environment, but it all boils down to your constraints, naming convention and provisioning process. If you have a well defined environment and utilizing a good naming structure and can easily identify which vCenter a given ESX(i) host should be managed from, then this can easily be integrated into your existing kickstart with minor tweaks. This script was tested on vCenter 4.1 Update 1 and ESXi 4.1 and 4.1 Update1.

UPDATE (03/29/2011): Updated the IP Address extraction to use gethostbyname and added proper logout logic after joining vCenter.

UPDATE (02/01/2013): I have provided a download link to the joinvCenter.py script below as there have been some funky formatting issues when displaying the script. For ESXi 5.x hosts, you will need to ensure httpClient is enabled (disabled by default) on the ESXi firewall else it will not be able to connect to your vCenter Server. Please refer below for the instructions.

There are a few steps that are necessary before we get started and a recommended one for those that have security concerns around this solution.

Step 1 - You will need to extract some information from the vCenter server in which you would like your ESX(i) hosts to join. You will need to generate an inventory path to the vCenter cluster which will take the form of: [datacenter-name]/host/[cluster-name], this will automatically locate the managed object ID of your vCenter cluster which is required as part of the host add process. This was a manual process in Justin's original solution.

In this example, I have a datacenter called "Primp-Skunkworks" and a cluster under that datacenter called "Primp-Skunkworks-Cluster", the inventory path will look like the following:  

"Primp-Skunkworks/host/Primp-Skunkworks-Cluster"

You will need this value to populate a variable in the script which will be described a little bit later

Step 2 - As you may have guessed, to add an ESX(i) host to vCenter, you will need to connect to vCenter server and use an account that has the permission to add a host. It is recommended that you do not use or expose any administrative accounts for this as the credentials are stored within the script unencrypted. A work around is to create a service account whether that is a local account or an Active Directory account with only the permission to add an ESX(i) host to a vCenter cluster. You will create a new role, in this example I call it "JoinvCenter" and you just need to provide the Host->Inventory->Add host to cluster privilege.

Once you have created the role, you will need to assign this role to the service account user either globally in vCenter if you want to add to multiple cluster or a given datacenter/cluster.

Now that we have the pre-requisites satisfied, we will need to populate a few variables within the script which will be used in your %post section of ESX(i) kickstart configuration file.

This variable defines the name of your vCenter server, please provide the FQDN:

This variable define the vCenter cluster path which was generated earlier:

These variables define the server account credentials used to add an ESX(i) host to vCenter. You will need to run the following command to encode the selected password. You will need access to a system with python interpreter to run the following command:

python -c "import base64; 
print base64.b64encode('MySuperDuperSecretPasswordYo')"

Note: This does not encrypt your password but obfuscate it slightly so that you are not storing the password in plain text. If a user has access to the encoded hash, it is trivial to decode it.

These variables define the ESX(i) root credentials which is required as part of the vCenter add process. If you do not want to store these in plain text, you will also need to encode them using the command in previous section:

We are now all done and ready to move forward with the actual script which will be included in your kickstart configuration. As a sanity check, you can run this script manually on an existing ESX(i) host to ensure that the process works before testing in kickstart. For ESXi 5.x hosts, ensure httpClient firewall ruleset is enabled by following ESXCLI command:

esxcli network firewall ruleset set -e true -r httpClient

 You should also ensure this is the very last script to execute as I ran into a race condition while the root password was being updated automatically from the default 999.* scripts. To ensure this is the very last script, set the --level to something like 9999 in your %firstboot stanza

Download: joinvCenter.py

To aide in troubleshooting, the script also outputs the details to syslog and on ESX(i), it will be stored in /var/log/messages and you can just search for the string "GHETTO-JOIN-VC". If everything is successful, after %firstboot section has completed, you should be able to see an ESX(i) host join vCenter and the following in the logs.

Tips: You should only see "Success" messages, if you see any "Failed" messages, something went wrong. If you are still running into issues, make sure your ESX(i) host has it's hostname configured with FQDN and you should see an error on your vCenter server if it fails to whether it's due to hostname and/or credentials. You can also redirect the output of the script to local VMFS volume for post-troubleshooting.

Depending on your provision process and how you determine which ESX(i) host should join which vCenter/cluster, you can easily add logic in the main kickstart configuration file to automatically determine or extract from a configuration file and dynamically update joinvCenter.py script prior to execution.

I would like to thank Justin Guidroz and VMTN user klich for their contributions on the python snippets that were used in the script. 

FYI - I am sure the python code could be cleaner but I will leave that as an exercise for those more adept to python. My python-fu is not very strong 😉