VSAN – Page 34

Required ESXi advanced setting to support 16+ node VSAN Cluster

03.13.2014 by William Lam // Leave a Comment

This evening I came across a very interesting VSAN knowledge base article KB 2073930 which states: to support a VSAN Cluster with greater than 16+ nodes, an ESXi Advanced Setting must be configured. The wording in the current KB is a bit confusing as it mentions the advanced setting must be applied to all hosts and then it mentions it should only be applied to the "additional" hosts. I have reached out to engineering and confirmed that the setting should be applied to ALL ESXi hosts and the KB will be corrected shortly. After applying this setting, you will also need to reboot your ESXi host for the changes to take affect.

The advanced setting we are interested in is called CMMDS.goto11 and you can check its current value by running the following ESXCLI command:

esxcli system settings advanced list -o /CMMDS/goto11

The following ESXCLI command will configure the required setting (default it is 0):

esxcli system settings advanced set -o /CMMDS/goto11 -i 1

The reason this setting is disabled by default is that it does consume a tiny amount of memory, we are talking about sub ~150MB per ESXi host. Though this amount is extremely small, the VSAN Engineers wanted to be very cognizant of smaller VSAN deployments and not have to require additional resources if it is not necessary. In my opinion if you plan to eventually scale beyond 16+ nodes in a VSAN Cluster and you can afford the additional memory overhead, then I would recommend enabling this setting as part of your default configuration so you do not need to do it later. If not, you can always do it later, but you know how that goes 🙂

To assist with this change, I have created two scripts: vSphere SDK for Perl and PowerCLI which will automate the configuration of this advanced setting for ALL ESXi hosts within a VSAN Cluster.

Disclaimer: These scripts are provided for informational and educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

You can download the sample vSphere SDK for Perl script called: enable16PlusVSANNodeSupport.pl

Here is an example of how you would run it:

./enable16PlusVSANNodeSupport.pl --server vcenter55-1.primp-industries.com --username root --cluster VSAN-Cluster

UPDATE (07/27/17) - Here is a PowerCLI script called VSANLargeClusterAdvSettings.ps1 which contains two functions (Get-VsanLargeClusterAdvancedSetting and Set-VsanLargeClusterAdvancedSetting) for retrieving and updating the necessary ESXi Advanced Settings to enable large vSAN Clusters for ESXi hosts running on 5.5, 6.0 & 6.5. There are two different KBs depending on the version of ESXi, please see https://kb.vmware.com/kb/2073930 and https://kb.vmware.com/kb/2110081 for more details.

Finally, some fun VSAN trivial from the engineering team. You may have noticed the advanced settings is called "goto11" which I was initially scratching my head over on why the engineers would name it that. I initially though it was either binary 11 OR it was a literal GoTo statement. It turns out, the answer can be found in this YouTube video here. Thanks to Dinesh Nambisan for sharing this awesome little tidbit. I wonder what other surprises lies in the code 🙂

Exploring VSAN APIs Part 4 – VSAN Disk Mappings

03.10.2014 by William Lam // Leave a Comment

A useful feature that is provided in the vSphere Web Client for VSAN is the ability to visualize the VSAN disk mappings for each ESXi host contributing to the VSAN cluster. This information will come in handy for troubleshooting as well as checking or servicing a disk failure.

You can easily retrieve the VSAN Disk Groups for each ESXi host and their respective SSD and HDD within that disk group by looking at the following property vsanSystem->config->storageInfo->diskMapping. There is a variety of information for each device such as the name, capacity, state, health, etc. of each disk. To demonstrate the above operation, I have created a vSphere SDK for Perl sample script called vsanHostTrafficTypeMgmt.pl

Disclaimer: These scripts are provided for informational and educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

To show the disk mappings, you will need to specify a vSphere Cluster running VSAN using the following command:

./vsanHostDiskMapping.pl --server vcenter55-1.primp-industries.co --username root --cluster VSAN-Cluster

In the above output, you will see the VSAN Disk Groups for each ESXi host along with their respective device name and capacity. Here is another screenshot for a VSAN environment that contains multiple VSAN Disk Groups:

This is just a very tiny sub-set of the available properties for a disk, for more information take a look at the scsiLun definition within the vSphere API Reference.

Exploring VSAN APIs Part 3 – Enable VSAN Traffic Type

03.06.2014 by William Lam // Leave a Comment

In addition to having a minimum of one SSD and one HDD to contribute to the VSAN Cluster, an ESXi host must also have a VMkernel interface that is enabled for VSAN traffic. In the vSphere Web Client, this is done by going into each ESXi host and under the Networking section and editing the VMkernel adapter to allow for a particular traffic type.

From a vSphere API perspective, this operation is actually exposed at the vSphere Cluster level which took me by surprised since enabling the other types of VMkernel traffic type is done else where. You will need to use the ReconfigureComputeResource_Task() method and specify a list of ESXi hosts using spec->vsanHostConfigSpec->HostSystem and list of VMkernel adapters that you wish to enable VSAN traffic type on using spec->vsanHostConfigSpec->networkInfo->port->device. The neat thing about doing it at the cluster level is you only need to make a single API call versus having to go to each and every ESXi host. To demonstrate the above operation, I have created a vSphere SDK for Perl sample script called vsanHostTrafficTypeMgmt.pl

Disclaimer: These scripts are provided for informational and educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

To check the current VSAN configuration for a given vSphere Cluster, you can run the “query” operation:

./vsanHostTrafficTypeMgmt.pl --server vcenter55-1.primp-industries.com --username root --cluster VSAN-Cluster --operation query

As you can see from the output above, there are no VMkernel adapters listed that have VSAN traffic type enabled.

To simplify the sample script, as part of the "enable" operation, I have added a --vihost option which requires you to specify a particular ESXi host in which you wish to enable VSAN traffic type. You will also need to provide the --vmk option to specify the list of VMkernel interfaces you wish to enable VSAN traffic type.

./vsanHostTrafficTypeMgmt.pl --server vcenter55-1.primp-industries.com --username root --cluster VSAN-Cluster --operation enable --vihost vesxi55-1.primp-industries.com --vmk vmk0

The "disable" operation is just the opposite of the enable, providing the same input of ESXi host and VMkernel interface.

We can now re-run our "query" operation and we should now see that all of our ESXi hosts have vmk0 enabled with VSAN traffic type: