There was an interesting VMware KB article that was shared by Ron Oglesby last week which had caught my eye.
I had noticed earlier in the week that Ron was interested in finding the current VSAN Component count which is exposed in a variety of interfaces: RVC (vSphere Ruby Console) available on both Windows and Linux as well as through the vSphere API. I even created some recent scripts here and here using the vSphere API to remotely query the number of VSAN components for each ESXi host. I much prefer this option from a management standpoint and not have to log into each individual ESXi host.
After looking at VMware KB 2071379, I can see why Ron had asked his question as I also felt the KB was incomplete. However, to the unsuspecting eye it may not be obvious but the KB actually does contain the answer but it does not really go into any details that can be consumed by a customer. In the article, it mentions that VSAN has the ability to trigger an alarm when the threshold of the number of VSAN components on a particular host has reached 80%. What the article lacks are the details of how and where this alarm is triggered. First off, the alarm mentioned here is for vCenter Server. Secondly, this is made possible through the use of the VOB (VMkernel Observation) ID mentioned in the article. You can actually create vCenter alarms based on these ESXi host generated VOBs which I have written about in the past such as this one on detecting duplicate IP Address for your ESXi hosts. The process in creating this vCenter Alarm is pretty straight forward and I agree that this alarm should have been created by default (something I will raise internally with the engineering team).
Here are the steps to create a vCenter Server Alarm to notify at the 80% VSAN Component threshold:
Step 1 - Create a new vCenter Server Alarm and give it a name and select "Monitor specific event ..." for a Host and make sure it is enabled.
Step 2 - Add in esx.problem.vob.vsan.lsom.componentthreshold for the Event
Step 3 - You can leave the actions to be empty which will just generate a regular vSphere Alarm or you can specify an action.
Once we have our vCenter Alarm created, we will probably want to test and verify the alarm is working using either a Nested ESXi VSAN environment or an actual VSAN environment. The next question is how do we go about creating 2400 VSAN components? Well, instead of manually creating 2400 VMs which will probably take awhile, we can easily do so by leveraging a neat little utility found in the ESXi Shell called /usr/lib/vmware/osfs/bin/objtool
Disclaimer: The tools and scripts used in this article is mainly for education and information purposes The command below will create an object called object-1 with size 1KB leveraging the VSAN Policy of hostFailuresToTolerate=0 & forceProvisioning=1:
/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-1 -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"
For this particular test, we just want to quickly create 2400 VSAN components. To do so, you will need about 32GB of memory to reach the maximum amount of supported VSAN Components. This should not be a problem for a "real" VSAN environment but for my Nested ESXi environment I had to increase my resources for this test. Since VSAN is a Distributed Object Store, the objects being created will be randomly placed within a VSAN Cluster. To quickly get to 2400 components, I also put 2 out 3 ESXi hosts into Maintenance Mode to ensure all objects are created o the first ESXi host.
Finally, to assist with the creation of these VSAN Objects automatically, I created a quick script which you can run in the ESXi Shell
#!/bin/sh
START=1
END=2400
for i in $(seq ${START} ${END});
do
echo "Creating VSAN Component ${i} ..."
RES=$(/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-${i} -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))")
UUID=$(echo ${RES} | awk -F 'UUID:' '{print $2}')
echo ${UUID} >> /tmp/uuid
done
The creation of each object will have an associated UUID which will be saved into a temporary file /tmp/uuid and you can use the following script to delete the object once you have confirmed the vCenter Alarm works.
#!/bin/sh for i in $(cat /tmp/uuid); do echo "Removing VSAN Object $i ..." /usr/lib/vmware/osfs/bin/objtool delete -u $i done
Once the 2400 VSAN Component count has been reach, you should now see the alarm we created earlier triggering for reaching the 80% threshold.
