This evening I came across a very interesting VSAN knowledge base article KB 2073930 which states: to support a VSAN Cluster with greater than 16+ nodes, an ESXi Advanced Setting must be configured. The wording in the current KB is a bit confusing as it mentions the advanced setting must be applied to all hosts and then it mentions it should only be applied to the "additional" hosts. I have reached out to engineering and confirmed that the setting should be applied to ALL ESXi hosts and the KB will be corrected shortly. After applying this setting, you will also need to reboot your ESXi host for the changes to take affect.
The advanced setting we are interested in is called CMMDS.goto11 and you can check its current value by running the following ESXCLI command:
esxcli system settings advanced list -o /CMMDS/goto11
The following ESXCLI command will configure the required setting (default it is 0):
esxcli system settings advanced set -o /CMMDS/goto11 -i 1
The reason this setting is disabled by default is that it does consume a tiny amount of memory, we are talking about sub ~150MB per ESXi host. Though this amount is extremely small, the VSAN Engineers wanted to be very cognizant of smaller VSAN deployments and not have to require additional resources if it is not necessary. In my opinion if you plan to eventually scale beyond 16+ nodes in a VSAN Cluster and you can afford the additional memory overhead, then I would recommend enabling this setting as part of your default configuration so you do not need to do it later. If not, you can always do it later, but you know how that goes 🙂
To assist with this change, I have created two scripts: vSphere SDK for Perl and PowerCLI which will automate the configuration of this advanced setting for ALL ESXi hosts within a VSAN Cluster.
Disclaimer: These scripts are provided for informational and educational purposes only. It should be thoroughly tested before attempting to use in a production environment.
You can download the sample vSphere SDK for Perl script called: enable16PlusVSANNodeSupport.pl
Here is an example of how you would run it:
./enable16PlusVSANNodeSupport.pl --server vcenter55-1.primp-industries.com --username root --cluster VSAN-Cluster
UPDATE (07/27/17) - Here is a PowerCLI script called VSANLargeClusterAdvSettings.ps1 which contains two functions (Get-VsanLargeClusterAdvancedSetting and Set-VsanLargeClusterAdvancedSetting) for retrieving and updating the necessary ESXi Advanced Settings to enable large vSAN Clusters for ESXi hosts running on 5.5, 6.0 & 6.5. There are two different KBs depending on the version of ESXi, please see https://kb.vmware.com/kb/2073930 and https://kb.vmware.com/kb/2110081 for more details.
Finally, some fun VSAN trivial from the engineering team. You may have noticed the advanced settings is called "goto11" which I was initially scratching my head over on why the engineers would name it that. I initially though it was either binary 11 OR it was a literal GoTo statement. It turns out, the answer can be found in this YouTube video here. Thanks to Dinesh Nambisan for sharing this awesome little tidbit. I wonder what other surprises lies in the code 🙂
Thanks for the comment!