vpxa

Applying the strings method as described in my last article, I decided to also take a look at /opt/vmware/vpxa/vpx/vpxa binary to see if there were anymore hidden goodies. To my surprise, I was able to locate additional HA and VPXA advanced configuration options. While going through and testing some of the HA advanced options, I found that only 19 out of 47 have not been documented and much of the documentation I found online was from Mr. Duncan Epping's blog. This really shows how open Duncan has been around the advanced options with VMware HA, if only VMware as a whole could be so open with the other advanced options that are used throughout VMware but left undocumented.

I do have to stress, these are configurations that are not documented and probably not supported unless directed by VMware. You should be very careful if you decide to play with some of these options and ensure you do not test on a production environment, don't say I did not warn you 🙂

Here is the list of 47 HA Advanced Configurations, the ones with links have been publicly documented by either Duncan or VMware. The rest have not been documented and I have done some testing which will be discussed further.

das.allowNetwork
das.allowVMotionNetworks
das.appMonFailureInterval - #
das.bypassNetCompatCheck
das.checkVmStateDelay - # (default 120)
das.consoleNode - [fqdn]
das.consolePerm - [PERM_USER,PERM_OPER,PERM_ALL]
das.consoleUser - [username]
das.defaultFailoverHost
das.disableUWSwapRequirement
das.failureDetectionInterval
das.failureDetectionTime
das.failureInterval
das.ignoreRedundantNetWarning
das.includeFTcomplianceChecks
das.iostatsInterval
das.isolationAddress
das.isolationShutdownTimeout
das.maxFailures
das.maxFailureWindow
das.maxftvmrestartcount - # (default 5)
das.maxFtVmsPerHost
das.maxvmrestartcount
das.minUptime
das.perHostConcurrentFailoversLimit
das.perHostVMLimit - #
das.powerOffOnIsolation
das.preferredPrimaries
das.primaryCount - [2,3,4,5] (default 5)
das.restartPriority
das.sensorPollingFreq
das.sim.enabled
das.sim.numVms
das.sim.powerOpsPerMinute
das.sim.resourceOpsPerMinute
das.sim.sendInterval
das.slotCpuInMHz
das.slotMemInMB
das.slotNumVcpus
das.trace - [on,off] (default on)
das.traceLevel - [0,1,2,3,4,functiontracing] (default 3)
das.traceOutput - [stdout, eventLog, file] (default file)
das.useDefaultIsolationAddress
das.vmCpuMinMHz
das.vmFailoverEnabled
das.vmMemoryMinMB

You can actually use AAM's (VMware HA) cli utility to view some of these hidden configurations, by using /opt/vmware/aam/bin/Cli

Here is a screenshot of the default VMware HA configuration using the getrule VMWareClusterManager command to list cluster rules, make a note of the parameters with red arrows:

Note: I found it interesting that the "VMWareClusterManager" had an upper case "W" in VMware. It looks like this name was not vetted by VMware marketing 🙂

Next, we add some of the advanced HA configurations using the vSphere Client under Advanced Options (HA) section:

Note: das.primaryCount only supports values 2 through 5, if you try to specify any other value, you will get an error. VMware has selected 5 primary nodes for a specific reason, you should not try to change it from the default.

We reconfigure the cluster with these new HA options and re-run getrule VMWareClusterManager and now we see the values get updated along with some new ones being added:

Note: You can also see some of these changes in /var/log/vmware/aam/aam_config_util_addnode.log

You can also use AAM cli to set some of these configurations:

Usage: setRuleVar|srv
AAM> setRuleVar VMWareClusterManager ft_Trace on

The advanced VPXA configurations applies to the vCenter Agent configuration file located in /etc/opt/vmware/vpxa/vpxa.cfg on an ESX(i) host. There was a total of 47 with only 8 being used in the default vpxa.cfg file.

Here is the list of 47 VPXA Configurations, the ones listed in blue are used by default, the ones with links are documented by VMware and the rest are undocumented:

vpxa.alarm.checkFrequency
vpxa.alarm.powerOnSilence
vpxa.bundleVersion
vpxa.cfg
vpxa.checkAAMPeriod
vpxa.checkNodeResourcePeriod
vpxa.connectTo
vpxa.disableOldQuickStats
vpxa.editionCheckTime
vpxa.ft.cleanupTimeout
vpxa.getchanges.delay
vpxa.getchanges.envbrowserRefreshRate
vpxa.getchanges.timeout
vpxa.healthSystem.throttleInterval
vpxa.heartbeat.interval
vpxa.HostdCnxBypassProxy
vpxa.hostdMonitorPeriod
vpxa.hostIp
vpxa.hostKey
vpxa.hostPort
vpxa.ignoreAamErrorCount
vpxa.ignoreAamErrorTime
vpxa.ignoreAamShutdownCount
vpxa.licenseExpiryNotificationThreshold
vpxa.maxEventReportingDelaySeconds
vpxa.maxFds
vpxa.memoryCheckerTimeInSecs
vpxa.mmapThresholdInKB
vpxa.nodeResourceThreshold
vpxa.partialVmUpdates
vpxa.preserveServerIp
vpxa.respool.changeThreshold
vpxa.respool.changeThresholdInMB
vpxa.respool.enableOverheadLimitTightLoop
vpxa.serverIp
vpxa.serverPort
vpxa.soapAdapterOverNamedPipe
vpxa.soapAdapterPort
vpxa.sslVersion
vpxa.StopMemInMB
vpxa.timeDiffThreshold
vpxa.useSsl
vpxa.vmapTimeoutCountThreshold
vpxa.vmapTimeoutTimeThreshold
vpxa.vmImages
vpxa.vmotion.vmIdAcquireTimeout
vpxa.WarnMemInMB

Note: The VPXA parameters is pretty difficult to reverse engineer and figure out what values each parameter can accept. I did notice the "periods" within each key actually represents the XML node separation within the vpxa.cfg.

Taking vpxa.vmotion.vmIdAcquireTimeout would translate to the following:

<vpxa>
    <vmotion>
         <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
</vpxa>

Again, please use extreme caution if you decide to experiment with these parameters and happy hacking 🙂

There are two easy methods of capturing logs for both an ESX or ESXi server:

Use VMware vMA and vilogger to setup a syslog server. Simon Long wrote a very detailed article on how to configured and set it all up, check it out here.
Configure ESX or ESXi server to remotely log to an existing syslog server.

In ESXi, there are potentially three sets of logs to be captured: messages, hostd, and vpxa (if your host is being managed by vCenter). If you are using option #1 and you have enabled logging using vilogger, you will see the following:

If you are using option #2, there is a caveat to be aware of if you are using vCenter to manage your ESXi host, the vpxa logs are not actually being captured.

Here is an ESXi 4.0 Update 1 host that is configured to remotely log to a syslog server, tailing the logs you will notice only the messages and hostd logs:

I was recently made aware of this problem from VMTN community user aenagy who ran into this in his environment. After filing a support request with VMware, he found out there is an entry that is not added to vpxa.cfg configuration file when the host is joined to a vCenter. Per VMware, it was not considered a bug, but I would disagree.

To enable vpxa log capture, you will need to login to the unsupported Busybox console, also known as Tech Support Mode, for instructions please take a look at this VMware KB. You need to edit the following /etc/opt/vmware/vpxa/vpxa.cfg this is the configuration file for the vCenter agent running on the ESXi host:

<outputToSyslog>true<outputToSyslog>
<syslog>
    <ident>vpxa</ident>
    <facility>local4</facility>
</syslog>

Append the above entries between the ... tags. Once you have updated the vpxa.cfg file, you will want to run the follow command on the Busybox console to ensure the changes are saved and backed up to the local bootbank for ESXi. There is an automated cron job that runs every hour which calls /sbin/auto-backup.sh

You can just run that command manually which will backup the changes and you will see the diff at the bottom with the changes that were made:

~ # /sbin/auto-backup.sh
config implicitly loaded
local.tgz
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_rsa_host_key
etc/opt/vmware/vpxa/vpxa.cfg
etc/opt/vmware/vpxa/dasConfig.xml
etc/opt/vmware/aam/configBackup.tar.gz
etc/opt/init.d/vmware-aam
etc/sysconfig/network
etc/vmware/hostd/authorization.xml
etc/vmware/hostd/hostsvc.xml
etc/vmware/hostd/pools.xml
etc/vmware/hostd/vmAutoStart.xml
etc/vmware/hostd/proxy.xml
etc/vmware/ssl/rui.crt
etc/vmware/ssl/rui.key
etc/vmware/vmkiscsid/initiatorname.iscsi
etc/vmware/vmkiscsid/iscsid.conf
etc/vmware/config
etc/vmware/dvsdata.db
etc/vmware/esx.conf
etc/vmware/license.cfg
etc/vmware/locker.conf
etc/vmware/snmp.xml
etc/vmware/vmware.lic
etc/dhclient-vmk0.leases
etc/group
etc/hosts
etc/inetd.conf
etc/chkconfig.db
etc/passwd
etc/random-seed
etc/resolv.conf
etc/shadow
etc/syslog.conf
etc/sfcb/repository/root/interop/cim_indicationfilter.idx
etc/sfcb/repository/root/interop/cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/cim_indicationsubscription.idx
--- /etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:48:34 2010
+++ /tmp/auto-backup.26548.dir/etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:47:00 2010
@@ -8,11 +8,6 @@
10
1048576
false
- false
-
- vpxa
- local4
-
error
config implicitly loaded
Saving current state in /bootbank
Clock updated.
Time: 18:48:38 Date: 06/23/2010 UTC

Once the backup has completed, you will need to reboot the host before the configuration can take effect. Once your host is back online, if you take a look at the logs on your syslog server, you will notice now that vpxa logs are also being captured:

Hopefully VMware resolves this, because this is definitely a bug and can be a painful one to remediate, especially if you are managing a few hundred ESXi hosts.

Thanks to aenagy for the information.

UPDATE:

After the blog post, I received a few questions:

1) The first question was if the modification would persist from an upgrade and I can confirm it does. My test was originally from an ESXi 4.0u1 and after the change, I upgraded the host using VUM to 4.0u2 and the changes were preserved. The reason for this is because the /sbin/auto-backup.sh was executed which backs up the ESXi configuration and is reloaded upon ever reboot including upgrades.

2) The other question was around automating this change and the answer is yes! Assuming you have SSH access enabled on your ESXi host, you can use the following "ash" script (ESXi does not have a bash) to push the changes out to your host.

You can download the script here called esxiVPXASyslogFix.sh and make sure you have the right permissions on the script to execute.