WilliamLam.com

Other handy vSphere VOBs for creating vCenter Alarms

04.24.2014 by William Lam // 8 Comments

Lately I have been writing on a variety of topics regarding the use of VOBs (VMkernel Observations) for creating useful vCenter Alarms such as:

I figure it would also be useful to collect a list of all the vSphere VOBs, at least from what I can gather by looking at /usr/lib/vmware/hostd/extensions/hostdiag/locale/en/event.vmsg on the latest version of ESXi. The list below is quite extensive, there are a total of 308 vSphere VOBs not including the VSAN VOBs in my previous articles. For those those of you who use vSphere Replication, you may also find a couple of handy ones in the list.

[Read more...]

Handy VSAN VOBs for creating vCenter Alarms

04.22.2014 by William Lam // 3 Comments

There have been quite a few questions lately around vCenter Server Alarms for VSAN, one in particular that I have noticed is around individual disk failure for VSAN. Outside of the generic default datastore alarms, there seems to be only two VSAN specific alarms:

I figure there must be other useful alarms that we could create, especially after showing how you can create a vCenter Server Alarm to monitor the VSAN component count threshold based on a particular VSAN VOB. I took a look around and found the following VSAN specific VOBs which could be useful for creating additional vCenter Alarms.

VOB ID	VOB Description
esx.audit.vsan.clustering.enabled	VSAN clustering services have been enabled.
esx.clear.vob.vsan.pdl.online	VSAN device has come online.
esx.clear.vsan.clustering.enabled	VSAN clustering services have now been enabled.
esx.clear.vsan.vsan.network.available	VSAN now has at least one active network configuration.
esx.clear.vsan.vsan.vmknic.ready	A previously reported vmknic now has a valid IP.
esx.problem.vob.vsan.lsom.componentthreshold	VSAN Node: Near node component count limit.
esx.problem.vob.vsan.lsom.diskerror	VSAN device is under permanent error.
esx.problem.vob.vsan.lsom.diskgrouplimit	Failed to create a new disk group.
esx.problem.vob.vsan.lsom.disklimit	Failed to add disk to disk group.
esx.problem.vob.vsan.pdl.offline	VSAN device has gone offline.
esx.problem.vsan.clustering.disabled	VSAN clustering services have been disabled.
esx.problem.vsan.lsom.congestionthreshold	VSAN device Memory/SSD congestion has changed.
esx.problem.vsan.net.not.ready	A vmknic added to VSAN network configuration doesn't have valid IP. Network is not ready.
esx.problem.vsan.net.redundancy.lost	VSAN doesn't haven any redundancy in its network configuration.
esx.problem.vsan.net.redundancy.reduced	VSAN is operating on reduced network redundancy.
esx.problem.vsan.no.network.connectivity	VSAN doesn't have any networking configuration for use.
esx.problem.vsan.vmknic.not.ready	A vmknic added to VSAN network configuration doesn't have valid IP. It will not be in use.

Looking at the list above, the following two VOBs seems like they would be useful for alerting on a disk failure is:

esx.problem.vob.vsan.lsom.diskerror
esx.problem.vob.vsan.pdl.offline

Disclaimer: There are no guarantees that a disk error or failure will automatically trigger these VOBs due to the unknown nature of how a disk may be fail, especially if it is intermittently.

Even though we can not simulate a disk error on a physical disk, we can still do some magic using a Nested VSAN environment. The worse case scenario that you could run into is that one of the disk just goes completely offline. We can simulate a similar behavior in a Nested ESXi environment by removing one of the virtual disks from the Virtual Machine (not deleting it).

To demonstrate the following scenario, here are the steps to create a vCenter Alarm for the following two VOBs:

Step 1 - Create a new vCenter Alarm and give it a name. Select “Hosts” for Monitor and “Specific event occurring …” for Monitor:

Step 2 - Add the following two VOBs above into the Event trigger:

Step 3 - Remove one of the Virtual Disks (SSD/MD) from the Virtual Machine running the Nested ESXi VM.

Step 4 - There are two ways in which you can trigger the alarm. You can either create a new Virtual Machine which will try to write to the Nested ESXi VM in which you remove the Virtual Disk or you can rescan the storage adapter for the Nested ESXi VM. In my environment, I happen to have a VM running on an NFS datastore and I performed a Storage vMotion of the VM onto my VSAN Datastore using the default FTT=1 policy on a three node VSAN Cluster. This immediately triggered the alarm as seen in the screenshots below:

Automating Log Insight 2.0 configurations

04.21.2014 by William Lam // Leave a Comment

Last week I had a chance to deploy the latest release of vCenter Log Insight 2.0 (currently in public beta) in my lab to give it a spin. I must say, I am very impressed with the slick new UI and some of the new capabilities like the scale-out and high availability feature.

The actual deployment of the Virtual Appliance is pretty straight forward and the only thing I would mention when selecting the OVF Deployment Size is that the default "Small" option is not the smallest configuration possible. There is actually an "Extra Small" option that can be selected in the drop-down menu which is targeted for POCs and lab evaluations. This will help with minimizing the resource constraints for lab environments.

Something that I am always interested in when evaluating a new solution is to see how easy an automated and unattended configuration is. With the help of some of the Log Insight folks, I was able to create the following shell script which will perform a basic configuration of Log Insight which includes the backend database, admin password and NTP servers:

#!/bin/bash
# William Lam
# www.virtuallyghetto.com

LOG_INSIGHT_ADMIN_PASSWORD=vmware123
LOG_INSIGHT_DB_PASSWORD=vmware123
NTP_SERVERS="0.pool.ntp.org, 1.pool.ntp.org"

### DO NOT EDIT BEYOND HERE ###

LOG_INSIGHT_CONFIG_DIR=/storage/core/loginsight/config
NODE_TOKEN_FILE=node-token
LOG_INSIGHT_CONFIG_FILE=loginsight-config.xml#1
NODE_UUID=$(uuidgen)

echo "Creating ${LOG_INSIGHT_CONFIG_DIR} .."
[ ! -e ${LOG_INSIGHT_CONFIG_DIR} ] && mkdir -p ${LOG_INSIGHT_CONFIG_DIR}

echo "Generating Log Insight Node UUID ..."
echo ${NODE_UUID} > ${LOG_INSIGHT_CONFIG_DIR}/${NODE_TOKEN_FILE}

echo "Generating Log Insight Configuration file ..."
cat > ${LOG_INSIGHT_CONFIG_DIR}/${LOG_INSIGHT_CONFIG_FILE} << __LOG_INSIGHT__
<config>
<version>
<strata-version value="2.0.1-1734312.UNSTABLE" release-name="Nightly"/>
</version>
<distributed overwrite-children="true">
<daemon port="16520" token="${NODE_UUID}">
<service-group name="standalone"/>
</daemon>
</distributed>
<database>
<password value="${LOG_INSIGHT_DB_PASSWORD}"/>
<port value="12543"/>
</database>
<ntp>
<ntp-servers value="${NTP_SERVERS}"/>
</ntp>
</config>
__LOG_INSIGHT__

echo "Restarting Log Insight ..."
service loginsight restart

echo "Setting Admin password ..."
ADMINPASSWORD=${LOG_INSIGHT_ADMIN_PASSWORD} /opt/vmware/bin/li-reset-admin-passwd.sh

You will need to edit the following variables within the script:

LOG_INSIGHT_ADMIN_PASSWORD
LOG_INSIGHT_DB_PASSWORD
NTP_SERVERS

Here is an example of running the script against a newly deployed Log Insight system:

The above is just an example of what could be automated for Log Insight. If you take a look at the Configuration section of Log Insight, there are many more options.

If you decide you want to automate additional configurations. The way you would accomplish this is to first configure everything from the Log Insight configuration UI. Once you are happy with the configuration, SSH into your Log Insight system. In /storage/core/loginsight/config you will find a couple of configuration files loginsight-config.xml#X with a numeric number at the end. If you take a look at the file with the highest number, it will contain the latest changes to Log Insight and the configurations you made using the UI. You can then take that file and update the script to automate the other configuration options.