WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How to create vCenter Alarm to alert on ESXi 5.5u1 NFS APD issue?

04.19.2014 by William Lam // 14 Comments

As some of you may have heard, there is currently a known issue with NFS based datastores (includes VSA NFS datastores) after upgrading to vSphere 5.5 Update 1. The issue causes NFS datastores to disconnect and go into an APD (All Paths Down) state. VMware is currently aware of the problem and you can follow KB 2076392 for the latest updates.

While going through my Twitter stream this morning, I noticed an interesting question from fellow Blogger and friend Jase McCarty who asked the following:

vsphere55u1-nfs-apd-alarm-2
I was quite surprised to hear that there were no vCenter Alarms being triggered for this issue. I decided to take a look at the KB to better understand the symptoms and see if there was anything I could do to help. From what I can tell, the only way to identify this particular problem is by looking at the logs which the KB has an example of what you would see.

Once I took a look at the logs, I knew there was at least two methods in which one could get alerts. One option would be to leverage vCenter Log Insight and create a query based on the particular string but no every customer is using Log Insight and it does require a bit of setup. The second more obvious option for me would be to key off of the VMkernel VOBs that are being generated which I have written about in the past for detecting duplicate IP Addresses for ESXi and VSAN component threshold count.

Here are the steps to create vCenter Alarm:

Step 1 - Create a new vCenter Alarm and give it a name. Select "Hosts" for Monitor and "Specific event occurring ..." for Monitor for

vsphere55u1-nfs-apd-alarm-0
Step 2 - For the Trigger, you will add the following VOB entries (just copy/paste them in)

  • esx.problem.storage.apd.start
  • esx.problem.vmfs.nfs.server.disconnect
  • esx.problem.storage.apd.timeout

Note: The alarm will activate if ANY of the VOBs are seen since it is an OR statement. It would have been nice to be able to group these together to generate the alarm

vsphere55u1-nfs-apd-alarm-1
Once the alarm has been created, you will at least have a way to get notified if you are potentially affected by this problem. I would still highly recommend you subscribe to KB 2076392 for all the latest updates.

Categories // ESXi, vSphere 5.5 Tags // apd, ESXi 5.5, nfs, vob, vSphere 5.5

How to automatically monitor VSAN Component threshold using a vCenter Alarm?

04.14.2014 by William Lam // 6 Comments

There was an interesting VMware KB article that was shared by Ron Oglesby last week which had caught my eye.

vsan-component-threshold
I had noticed earlier in the week that Ron was interested in finding the current VSAN Component count which is exposed in a variety of interfaces: RVC (vSphere Ruby Console) available on both Windows and Linux as well as through the vSphere API. I even created some recent scripts here and here using the vSphere API to remotely query the number of VSAN components for each ESXi host. I much prefer this option from a management standpoint and not have to log into each individual ESXi host.

After looking at VMware KB 2071379, I can see why Ron had asked his question as I also felt the KB was incomplete. However, to the unsuspecting eye it may not be obvious but the KB actually does contain the answer but it does not really go into any details that can be consumed by a customer. In the article, it mentions that VSAN has the ability to trigger an alarm when the threshold of the number of VSAN components on a particular host has reached 80%. What the article lacks are the details of how and where this alarm is triggered. First off, the alarm mentioned here is for vCenter Server. Secondly, this is made possible through the use of the VOB (VMkernel Observation) ID mentioned in the article. You can actually create vCenter alarms based on these ESXi host generated VOBs which I have written about in the past such as this one on detecting duplicate IP Address for your ESXi hosts. The process in creating this vCenter Alarm is pretty straight forward and I agree that this alarm should have been created by default (something I will raise internally with the engineering team).

Here are the steps to create a vCenter Server Alarm to notify at the 80% VSAN Component threshold:

Step 1 - Create a new vCenter Server Alarm and give it a name and select "Monitor specific event ..." for a Host and make sure it is enabled.

Screen Shot 2014-04-11 at 4.41.27 AM
Step 2 - Add in esx.problem.vob.vsan.lsom.componentthreshold for the Event

Screen Shot 2014-04-11 at 4.42.01 AM
Step 3 - You can leave the actions to be empty which will just generate a regular vSphere Alarm or you can specify an action.

Once we have our vCenter Alarm created, we will probably want to test and verify the alarm is working using either a Nested ESXi VSAN environment or an actual VSAN environment. The next question is how do we go about creating 2400 VSAN components? Well, instead of manually creating 2400 VMs which will probably take awhile, we can easily do so by leveraging a neat little utility found in the ESXi Shell called /usr/lib/vmware/osfs/bin/objtool

Disclaimer: The tools and scripts used in this article is mainly for education and information purposes The command below will create an object called object-1 with size 1KB leveraging the VSAN Policy of hostFailuresToTolerate=0 & forceProvisioning=1:

/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-1 -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"

For this particular test, we just want to quickly create 2400 VSAN components. To do so, you will need about 32GB of memory to reach the maximum amount of supported VSAN Components. This should not be a problem for a "real" VSAN environment but for my Nested ESXi environment I had to increase my resources for this test. Since VSAN is a Distributed Object Store, the objects being created will be randomly placed within a VSAN Cluster. To quickly get to 2400 components, I also put 2 out 3 ESXi hosts into Maintenance Mode to ensure all objects are created o the first ESXi host.

vsan-component-count-alarm-3
Finally, to assist with the creation of these VSAN Objects automatically, I created a quick script which you can run in the ESXi Shell

#!/bin/sh

START=1
END=2400

for i in $(seq ${START} ${END});
do
	echo "Creating VSAN Component ${i} ..."
	RES=$(/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-${i} -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))")
	UUID=$(echo ${RES} | awk -F 'UUID:' '{print $2}')
	echo ${UUID} >> /tmp/uuid
done

The creation of each object will have an associated UUID which will be saved into a temporary file /tmp/uuid and you can use the following script to delete the object once you have confirmed the vCenter Alarm works.

#!/bin/sh

for i in $(cat /tmp/uuid);
do
	echo "Removing VSAN Object $i ..."
	/usr/lib/vmware/osfs/bin/objtool delete -u $i
done

Once the 2400 VSAN Component count has been reach, you should now see the alarm we created earlier triggering for reaching the 80% threshold.

vsan-component-count-alarm-0

Categories // VSAN, vSphere 5.5 Tags // alarm, components, objtool, vob, VSAN, vSphere 5.5

Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm

01.28.2013 by William Lam // 6 Comments

The motivation for this article was a tweet I noticed from Duncan Epping this morning. Per Duncan's tweet, it looks like he may have accidentally assigned an IP Address to one of his virtual machines which was already being used by an existing ESXi host causing a duplicate IP Address error. We probably have all experienced this once in our lives and it can be quite difficult and frustrating to troubleshoot. Similar to a Windows OS, ESXi can also detect a duplicate IP Addresses but instead of a notification window, it is just logged in the VMkernel logs which looks like the following:

2013-01-21T15:52:35.989Z cpu1:2049)Tcpip_Vmk: 112: arp: 00:50:56:bd:3b:2b is using my IP address 172.30.0.213 on vmk1! 

The biggest challenge of course is to identify which ESXi host actually has a conflict and then taking a look at the logs to find the offending MAC Address and shutting them down yourself or with the help of a Network Administrator. Wouldn't it be great if we had an alarm to automatically notify us when a duplicate IP Address is detected? Well I am glad you asked and the answer is YES! 🙂

In addition to logging to the VMkernel logs, ESXi also logs this "observation" in /var/log/vobd.log which stands for the VMkernel Observation. These "observations" can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. The really neat thing about these VOBs is that you can create vCenter Alarms when a specific VOB has been detected. I have shown an example of this before in my Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm article.

We can do exactly the same for detecting a duplicate IP Address for an ESXi host. The first thing we need to do is identify the VOB ID by looking in /var/log/vobd.log:

2013-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for 172.30.0.83 on interface vmk0, current owner being 00:50:56:bd:3b:2b

We can see the VOB ID for this is esx.problem.net.vmknic.ip.duplicate and this will be used in our vCenter Alarm trigger.

Step 1 - Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor For will be for a specific event:

Step 2 - Copy the VOB ID that we have identified from above and specify that as our alarm Trigger:

Step 3 - If you wish to receive an email notification or send an SNMP trap go ahead and configure additional actions, else just click next which will just display a vCenter Server alert in the UI.

Now that our alarm has been created, we will want to give this a test drive .... who can we ask? Well it just happens that I have a new user in my environment and I provisioned him a new VM which is already connected to the network. Let's hope he does not try to change the IP Address (because this never happens, right?)

After the user statically assigns the IP Address of an existing ESXi host in the VM, we should see our new alarm trigger in vCenter.

As you can see, we have quickly identified the ESXi host that is impacted and we can then login to DCUI via the console to take a look at the logs to find the offending MAC Address. Hopefully duplicate IP Addresses is not a common problem in your environment but it does happen from time to time and having an alarm to help you quickly narrow down the culprit can be quite useful.

Categories // Uncategorized Tags // alarm, duplicate IP, ESXi, ip address, vob, vSphere

  • « Previous Page
  • 1
  • 2
  • 3
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • VCF 9.0 Hardware Considerations 05/30/2025
  • VMware Flings is now available in Free Downloads of Broadcom Support Portal (BSP) 05/19/2025
  • VMUG Connect 2025 - Minimal VMware Cloud Foundation (VCF) 5.x in a Box  05/15/2025
  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...