components

Reporting vSAN Object distribution across vSAN Disk Groups using PowerCLI

09.26.2017 by William Lam // Leave a Comment

Several weeks back, I was cleaning up my scratch space, where I store all my random code snippets for various questions which I receive on a regular basis and I came across a nifty little script that I had put together for a particular customer request. I had completely forgotten about it and I thought it could come in handy for some folks who might be curious in how their current vSAN Objects are currently being distributed across all vSAN Disk Groups within a vSAN Cluster.

RVC already provides a nice command called vsan.check_limits which gives you a break down of the number of components across all disks within a vSAN Cluster as shown in the screenshot below.

However, in the case of this particular customer, they wanted the break down on a per Disk Group level rather than individual disks.

Luckily, all of this information is already exposed using the vSAN Management APIs, you simply just need to aggregate it one level up. With that, I created a PowerCLI script called VSANObjectDistribution.ps1 which allows you to provide the name of a vSAN Cluster and it will automatically provide you with both the number of components distributed across the different vSAN Disk Groups as well as the amount of storage consumed by these components.

Here is a screenshot for a 3-Node vSAN Cluster where each ESXi host contains two vSAN Disk Groups:

Since there is no actual number for a vSAN Disk Group, by default, I output the Canonical Disk Name of the "Cache" device for the given vSAN Disk Group so you can map it back.

If you prefer to see the vSAN UUID for the "Cache" device instead, you can simply set the -ShowvSANID parameter to true as shown in the screenshot below.

To correlate back the specific vSAN Disk Group, you simply select a particular vSAN Disk Group for the ESXi host you are interested in. At the bottom, add "vSAN UUID" column highlighted in orange and you can then compare either that ID or Canonical Disk Name highlighted in blue.

Why is my VSAN Component maximum showing less than 3000?

01.28.2015 by William Lam // Leave a Comment

This is a question that I have seen come up on several occasions in both the VMTN Community forums as well as in our internal Socialcast group. I have not seen anyone blog about this topic yet and figure I would share the answer since this was a question I had asked myself when I had initially setup VSAN. If you are not familiar with VSAN Components, I highly recommend you check out Cormac Hogan's blog article VSAN Part 4: Understanding Objects and Components.

In vSphere 5.5 Update 1, the maximum number of supported components for VSAN is 3000 which is a per ESXi host maximum. What some folks are noticing when they run the RVC vsan.check_limits command on their VSAN Cluster, they are finding out that the maximum is coming up much lower as seen in the example below.

/localhost/VSAN-Datacenter/computers> vsan.check_limits VSAN-Cluster/
2015-01-28 15:34:25 +0000: Gathering stats from all hosts ...
2015-01-28 15:34:27 +0000: Gathering disks info ...
+--------------------------------+-------------------+-------------------------------------------+
| Host                           | RDT               | Disks                                     |
+--------------------------------+-------------------+-------------------------------------------+
| vesxi55-3.primp-industries.com | Assocs: 30/20000  | Components: 8/750                         |
|                                | Sockets: 17/10000 | naa.6000c2932c3f51f04e4cd395f4a11752: 8%  |
|                                | Clients: 3        | naa.6000c294f6496a99ad756857b9b06f01: 0%  |
|                                | Owners: 5         |                                           |
| vesxi55-2.primp-industries.com | Assocs: 10/20000  | Components: 8/750                         |
|                                | Sockets: 13/10000 | naa.6000c294bde5987d60398e0305978b00: 9%  |
|                                | Clients: 0        | naa.6000c292a964255b82410099360a9b27: 0%  |
|                                | Owners: 0         |                                           |
| vesxi55-1.primp-industries.com | Assocs: 24/20000  | Components: 8/750                         |
|                                | Sockets: 15/10000 | naa.6000c298b69006b820e367b5fde97cbf: 11% |
|                                | Clients: 3        | naa.6000c29db3f272cfb7fb4d08bffad3ab: 0%  |
|                                | Owners: 3         |                                           |
+--------------------------------+-------------------+-------------------------------------------+

The reason for this is actually due to the amount of physical memory available to each ESXi host. If you are running VSAN in a Nested ESXi environment like I am in the example above, I only have 8GB of memory configured for each ESXi host. The number of supported VSAN Components will definitely differ from an actual physical host with more memory and the nice thing about vsan.check_limits command is that it is dynamic in nature based on the actual available resources. Funny enough, the majority of the questions actually came from folks who ran VSAN in a Nested Environment, so this would explain why this question keeps popping up.

If I run the same RVC command on an environment where VSAN was running on real hardware with a decent amount of memory which most modern systems these days have, then I can see the VSAN Component maximum is properly displaying the 3000 limit as expected in the example below.

/localhost/datacenter01/computers> vsan.check_limits vsan-cluster01/
2015-01-28 15:28:47 +0000: Querying limit stats from all hosts ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx021.vmwcs.com (may take a moment) ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx022.vmwcs.com (may take a moment) ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx024.vmwcs.com (may take a moment) ...
2015-01-28 15:28:51 +0000: Done fetching VSAN disk infos
+---------------------------+--------------------+---------------------------------------------------------------------------------+
| Host                      | RDT                | Disks                                                                           |
+---------------------------+--------------------+---------------------------------------------------------------------------------+
| esx021.vmwcs.com          | Assocs: 223/45000  | Components: 97/3000                                                             |
|                           | Sockets: 132/10000 | t10.ATA_____WDC_WD1002FAEX2D00Z3A0________________________WD2DWCATRC061926: 18% |
|                           | Clients: 14        | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017C69____: 0%   |
|                           | Owners: 29         |                                                                                 |
| esx022.vmwcs.com          | Assocs: 252/45000  | Components: 96/3000                                                             |
|                           | Sockets: 143/10000 | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017CA2____: 0%   |
|                           | Clients: 14        | t10.ATA_____WDC_WD1002FAEX2D00Z3A0________________________WD2DWCATRC050466: 19% |
|                           | Owners: 38         |                                                                                 |
| esx024.vmwcs.com          | Assocs: 197/45000  | Components: 96/3000                                                             |
|                           | Sockets: 122/10000 | t10.ATA_____ST2000DL0032D9VT166__________________________________5YD73PRP: 8%   |
|                           | Clients: 17        | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017C5B____: 0%   |
|                           | Owners: 22         |                                                                                 |
+---------------------------+--------------------+---------------------------------------------------------------------------------+

The lesson here is that even though I am a huge supporter of using Nested ESXi to learn about new products, features and how they work from a functional perspective, there is no amount of Nested ESXi testing that can ever replace actual testing of real hardware.

How to automatically monitor VSAN Component threshold using a vCenter Alarm?

04.14.2014 by William Lam // 6 Comments

There was an interesting VMware KB article that was shared by Ron Oglesby last week which had caught my eye.

I had noticed earlier in the week that Ron was interested in finding the current VSAN Component count which is exposed in a variety of interfaces: RVC (vSphere Ruby Console) available on both Windows and Linux as well as through the vSphere API. I even created some recent scripts here and here using the vSphere API to remotely query the number of VSAN components for each ESXi host. I much prefer this option from a management standpoint and not have to log into each individual ESXi host.

After looking at VMware KB 2071379, I can see why Ron had asked his question as I also felt the KB was incomplete. However, to the unsuspecting eye it may not be obvious but the KB actually does contain the answer but it does not really go into any details that can be consumed by a customer. In the article, it mentions that VSAN has the ability to trigger an alarm when the threshold of the number of VSAN components on a particular host has reached 80%. What the article lacks are the details of how and where this alarm is triggered. First off, the alarm mentioned here is for vCenter Server. Secondly, this is made possible through the use of the VOB (VMkernel Observation) ID mentioned in the article. You can actually create vCenter alarms based on these ESXi host generated VOBs which I have written about in the past such as this one on detecting duplicate IP Address for your ESXi hosts. The process in creating this vCenter Alarm is pretty straight forward and I agree that this alarm should have been created by default (something I will raise internally with the engineering team).

Here are the steps to create a vCenter Server Alarm to notify at the 80% VSAN Component threshold:

Step 1 - Create a new vCenter Server Alarm and give it a name and select "Monitor specific event ..." for a Host and make sure it is enabled.

Step 2 - Add in esx.problem.vob.vsan.lsom.componentthreshold for the Event

Step 3 - You can leave the actions to be empty which will just generate a regular vSphere Alarm or you can specify an action.

Once we have our vCenter Alarm created, we will probably want to test and verify the alarm is working using either a Nested ESXi VSAN environment or an actual VSAN environment. The next question is how do we go about creating 2400 VSAN components? Well, instead of manually creating 2400 VMs which will probably take awhile, we can easily do so by leveraging a neat little utility found in the ESXi Shell called /usr/lib/vmware/osfs/bin/objtool

Disclaimer: The tools and scripts used in this article is mainly for education and information purposes The command below will create an object called object-1 with size 1KB leveraging the VSAN Policy of hostFailuresToTolerate=0 & forceProvisioning=1:

/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-1 -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))"

For this particular test, we just want to quickly create 2400 VSAN components. To do so, you will need about 32GB of memory to reach the maximum amount of supported VSAN Components. This should not be a problem for a "real" VSAN environment but for my Nested ESXi environment I had to increase my resources for this test. Since VSAN is a Distributed Object Store, the objects being created will be randomly placed within a VSAN Cluster. To quickly get to 2400 components, I also put 2 out 3 ESXi hosts into Maintenance Mode to ensure all objects are created o the first ESXi host.

Finally, to assist with the creation of these VSAN Objects automatically, I created a quick script which you can run in the ESXi Shell

#!/bin/sh

START=1
END=2400

for i in $(seq ${START} ${END});
do
	echo "Creating VSAN Component ${i} ..."
	RES=$(/usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 3 -n object-${i} -p "((\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i1))")
	UUID=$(echo ${RES} | awk -F 'UUID:' '{print $2}')
	echo ${UUID} >> /tmp/uuid
done

The creation of each object will have an associated UUID which will be saved into a temporary file /tmp/uuid and you can use the following script to delete the object once you have confirmed the vCenter Alarm works.

#!/bin/sh

for i in $(cat /tmp/uuid);
do
	echo "Removing VSAN Object $i ..."
	/usr/lib/vmware/osfs/bin/objtool delete -u $i
done

Once the 2400 VSAN Component count has been reach, you should now see the alarm we created earlier triggering for reaching the 80% threshold.