Hidden HA and VPXA Configurations

Applying the strings method as described in my last article, I decided to also take a look at /opt/vmware/vpxa/vpx/vpxa binary to see if there were anymore hidden goodies. To my surprise, I was able to locate additional HA and VPXA advanced configuration options. While going through and testing some of the HA advanced options, I found that only 19 out of 47 have not been documented and much of the documentation I found online was from Mr. Duncan Epping's blog. This really shows how open Duncan has been around the advanced options with VMware HA, if only VMware as a whole could be so open with the other advanced options that are used throughout VMware but left undocumented.

Here is the list of 47 HA Advanced Configurations, the ones with links have been publicly documented by either Duncan or VMware. The rest have not been documented and I have done some testing which will be discussed further.

das.allowNetwork
das.allowVMotionNetworks
das.appMonFailureInterval - #
das.bypassNetCompatCheck
das.checkVmStateDelay - # (default 120)
das.consoleNode - [fqdn]
das.consolePerm - [PERM_USER,PERM_OPER,PERM_ALL]
das.consoleUser - [username]
das.defaultFailoverHost
das.disableUWSwapRequirement
das.failureDetectionInterval
das.failureDetectionTime
das.failureInterval
das.ignoreRedundantNetWarning
das.includeFTcomplianceChecks
das.iostatsInterval
das.isolationAddress
das.isolationShutdownTimeout
das.maxFailures
das.maxFailureWindow
das.maxftvmrestartcount - # (default 5)
das.maxFtVmsPerHost
das.maxvmrestartcount
das.minUptime
das.perHostConcurrentFailoversLimit
das.perHostVMLimit - #
das.powerOffOnIsolation
das.preferredPrimaries
das.primaryCount - [2,3,4,5] (default 5)
das.restartPriority
das.sensorPollingFreq
das.sim.enabled
das.sim.numVms
das.sim.powerOpsPerMinute
das.sim.resourceOpsPerMinute
das.sim.sendInterval
das.slotCpuInMHz
das.slotMemInMB
das.slotNumVcpus
das.trace - [on,off] (default on)
das.traceLevel - [0,1,2,3,4,functiontracing] (default 3)
das.traceOutput - [stdout, eventLog, file] (default file)
das.useDefaultIsolationAddress
das.vmCpuMinMHz
das.vmFailoverEnabled
das.vmMemoryMinMB

You can actually use AAM's (VMware HA) cli utility to view some of these hidden configurations, by using /opt/vmware/aam/bin/Cli

Here is a screenshot of the default VMware HA configuration using the getrule VMWareClusterManager command to list cluster rules, make a note of the parameters with red arrows:

Note: I found it interesting that the "VMWareClusterManager" had an upper case "W" in VMware. It looks like this name was not vetted by VMware marketing 🙂

Next, we add some of the advanced HA configurations using the vSphere Client under Advanced Options (HA) section:

Note: das.primaryCount only supports values 2 through 5, if you try to specify any other value, you will get an error. VMware has selected 5 primary nodes for a specific reason, you should not try to change it from the default.

We reconfigure the cluster with these new HA options and re-run getrule VMWareClusterManager and now we see the values get updated along with some new ones being added:

Note: You can also see some of these changes in /var/log/vmware/aam/aam_config_util_addnode.log

You can also use AAM cli to set some of these configurations:

Usage: setRuleVar|srv
AAM> setRuleVar VMWareClusterManager ft_Trace on

The advanced VPXA configurations applies to the vCenter Agent configuration file located in /etc/opt/vmware/vpxa/vpxa.cfg on an ESX(i) host. There was a total of 47 with only 8 being used in the default vpxa.cfg file.

Here is the list of 47 VPXA Configurations, the ones listed in blue are used by default, the ones with links are documented by VMware and the rest are undocumented:

vpxa.alarm.checkFrequency
vpxa.alarm.powerOnSilence
vpxa.bundleVersion
vpxa.cfg
vpxa.checkAAMPeriod
vpxa.checkNodeResourcePeriod
vpxa.connectTo
vpxa.disableOldQuickStats
vpxa.editionCheckTime
vpxa.ft.cleanupTimeout
vpxa.getchanges.delay
vpxa.getchanges.envbrowserRefreshRate
vpxa.getchanges.timeout
vpxa.healthSystem.throttleInterval
vpxa.heartbeat.interval
vpxa.HostdCnxBypassProxy
vpxa.hostdMonitorPeriod
vpxa.hostIp
vpxa.hostKey
vpxa.hostPort
vpxa.ignoreAamErrorCount
vpxa.ignoreAamErrorTime
vpxa.ignoreAamShutdownCount
vpxa.licenseExpiryNotificationThreshold
vpxa.maxEventReportingDelaySeconds
vpxa.maxFds
vpxa.memoryCheckerTimeInSecs
vpxa.mmapThresholdInKB
vpxa.nodeResourceThreshold
vpxa.partialVmUpdates
vpxa.preserveServerIp
vpxa.respool.changeThreshold
vpxa.respool.changeThresholdInMB
vpxa.respool.enableOverheadLimitTightLoop
vpxa.serverIp
vpxa.serverPort
vpxa.soapAdapterOverNamedPipe
vpxa.soapAdapterPort
vpxa.sslVersion
vpxa.StopMemInMB
vpxa.timeDiffThreshold
vpxa.useSsl
vpxa.vmapTimeoutCountThreshold
vpxa.vmapTimeoutTimeThreshold
vpxa.vmImages
vpxa.vmotion.vmIdAcquireTimeout
vpxa.WarnMemInMB

Note: The VPXA parameters is pretty difficult to reverse engineer and figure out what values each parameter can accept. I did notice the "periods" within each key actually represents the XML node separation within the vpxa.cfg.

Taking vpxa.vmotion.vmIdAcquireTimeout would translate to the following:

<vpxa>
    <vmotion>
         <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
</vpxa>

Again, please use extreme caution if you decide to experiment with these parameters and happy hacking 🙂