William Lam

VSAN Flash/MD capacity reporting

04.29.2014 by William Lam // Leave a Comment

One of the capabilities that is available with VSAN when creating a VM Storage Policy is the ability to specify the amount of to Flash to reserve for a Virtual Machine object as a read cache. For Virtual Machines that require high levels of performance, you can assign this policy to the Virtual machine and VSAN will ensure a percentage of the Flash capacity is provided to your workload.

A couple weeks back I was asked whether it was possible to report on the total amount of Flash capacity available to a VSAN Cluster including what has been reserved and in use. I thought that this was a great idea as users would probably want to be able see their utilization over time and ensure they do not over provision their Flash capacity.

For those of you who have used RVC, this information is somewhat available today using the vsan.disks_stats command. The only problem is that this information is only provided at a per device level for each ESXi host and not in an aggregate view for the entire VSAN Cluster.

Leveraging the work I had done earlier with exploring the VSAN API and looking at the VSAN component count, I was able to extract the necessary information that I was looking for to provide an aggregate view. To demonstrate this functionality, I have created two sample scripts: vSphere SDK for Perl script called vsanFlashAndMDCapacity.pl and PowerCLI script called vsanFlashAndMDCapacity.ps1

Disclaimer: These scripts are provided for informational and educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

Both scripts work exactly the same way, you just need to connect it to a vCenter Server that has at least one VSAN Cluster. The script will automatically search for all VSAN enabled vSphere Cluster and provide the following information:

Total SSD Capacity
Total SSD Reserved Capacity
Total SSD Used Capacity
Total MD Capacity
Total MD Reserved Capacity
Total MD Used Capacity

Here is an example screenshot for the vSphere SDK for Perl script:

Here is an example screenshot for the PowerCLI script:

One question I had myself while looking at the results was regarding the "Used" property and what it meant. I think this is best explained with an example after learning about the details from engineering.

Lets say there are 2 VSAN objects:

Object1: Configured size: 100GB, space reservation 10%, actual data written 5GB.
Object2: Configured size: 100GB, space reservation 10%, actual data written 15GB.

This would mean:

Object1:
Configured/Provisioned: 100GB
Reserved: 10GB
Physical Used: 5GB
Used: 10GB

Object2:
Configured/Provisioned: 100GB
Reserved: 10GB
Physical Used: 15GB
Used: 15GB

The "Used" property is then calculated as the MAX(Physical Used, Reserved). I have also shared this information with engineering, perhaps they may consider adding this information to RVC 🙂 If you think this is something you would like to see in RVC, please leave a comment.

Other handy vSphere VOBs for creating vCenter Alarms

04.24.2014 by William Lam // 8 Comments

Lately I have been writing on a variety of topics regarding the use of VOBs (VMkernel Observations) for creating useful vCenter Alarms such as:

I figure it would also be useful to collect a list of all the vSphere VOBs, at least from what I can gather by looking at /usr/lib/vmware/hostd/extensions/hostdiag/locale/en/event.vmsg on the latest version of ESXi. The list below is quite extensive, there are a total of 308 vSphere VOBs not including the VSAN VOBs in my previous articles. For those those of you who use vSphere Replication, you may also find a couple of handy ones in the list.

[Read more...]

Handy VSAN VOBs for creating vCenter Alarms

04.22.2014 by William Lam // 3 Comments

There have been quite a few questions lately around vCenter Server Alarms for VSAN, one in particular that I have noticed is around individual disk failure for VSAN. Outside of the generic default datastore alarms, there seems to be only two VSAN specific alarms:

I figure there must be other useful alarms that we could create, especially after showing how you can create a vCenter Server Alarm to monitor the VSAN component count threshold based on a particular VSAN VOB. I took a look around and found the following VSAN specific VOBs which could be useful for creating additional vCenter Alarms.

VOB ID	VOB Description
esx.audit.vsan.clustering.enabled	VSAN clustering services have been enabled.
esx.clear.vob.vsan.pdl.online	VSAN device has come online.
esx.clear.vsan.clustering.enabled	VSAN clustering services have now been enabled.
esx.clear.vsan.vsan.network.available	VSAN now has at least one active network configuration.
esx.clear.vsan.vsan.vmknic.ready	A previously reported vmknic now has a valid IP.
esx.problem.vob.vsan.lsom.componentthreshold	VSAN Node: Near node component count limit.
esx.problem.vob.vsan.lsom.diskerror	VSAN device is under permanent error.
esx.problem.vob.vsan.lsom.diskgrouplimit	Failed to create a new disk group.
esx.problem.vob.vsan.lsom.disklimit	Failed to add disk to disk group.
esx.problem.vob.vsan.pdl.offline	VSAN device has gone offline.
esx.problem.vsan.clustering.disabled	VSAN clustering services have been disabled.
esx.problem.vsan.lsom.congestionthreshold	VSAN device Memory/SSD congestion has changed.
esx.problem.vsan.net.not.ready	A vmknic added to VSAN network configuration doesn't have valid IP. Network is not ready.
esx.problem.vsan.net.redundancy.lost	VSAN doesn't haven any redundancy in its network configuration.
esx.problem.vsan.net.redundancy.reduced	VSAN is operating on reduced network redundancy.
esx.problem.vsan.no.network.connectivity	VSAN doesn't have any networking configuration for use.
esx.problem.vsan.vmknic.not.ready	A vmknic added to VSAN network configuration doesn't have valid IP. It will not be in use.

Looking at the list above, the following two VOBs seems like they would be useful for alerting on a disk failure is:

esx.problem.vob.vsan.lsom.diskerror
esx.problem.vob.vsan.pdl.offline

Disclaimer: There are no guarantees that a disk error or failure will automatically trigger these VOBs due to the unknown nature of how a disk may be fail, especially if it is intermittently.

Even though we can not simulate a disk error on a physical disk, we can still do some magic using a Nested VSAN environment. The worse case scenario that you could run into is that one of the disk just goes completely offline. We can simulate a similar behavior in a Nested ESXi environment by removing one of the virtual disks from the Virtual Machine (not deleting it).

To demonstrate the following scenario, here are the steps to create a vCenter Alarm for the following two VOBs:

Step 1 - Create a new vCenter Alarm and give it a name. Select “Hosts” for Monitor and “Specific event occurring …” for Monitor:

Step 2 - Add the following two VOBs above into the Event trigger:

Step 3 - Remove one of the Virtual Disks (SSD/MD) from the Virtual Machine running the Nested ESXi VM.

Step 4 - There are two ways in which you can trigger the alarm. You can either create a new Virtual Machine which will try to write to the Nested ESXi VM in which you remove the Virtual Disk or you can rescan the storage adapter for the Nested ESXi VM. In my environment, I happen to have a VM running on an NFS datastore and I performed a Storage vMotion of the VM onto my VSAN Datastore using the default FTT=1 policy on a three node VSAN Cluster. This immediately triggered the alarm as seen in the screenshots below: