WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple

Hidden HA and VPXA Configurations

11.03.2010 by William Lam // Leave a Comment

Applying the strings method as described in my last article, I decided to also take a look at /opt/vmware/vpxa/vpx/vpxa binary to see if there were anymore hidden goodies. To my surprise, I was able to locate additional HA and VPXA advanced configuration options. While going through and testing some of the HA advanced options, I found that only 19 out of 47 have not been documented and much of the documentation I found online was from Mr. Duncan Epping's blog. This really shows how open Duncan has been around the advanced options with VMware HA, if only VMware as a whole could be so open with the other advanced options that are used throughout VMware but left undocumented.

I do have to stress, these are configurations that are not documented and probably not supported unless directed by VMware. You should be very careful if you decide to play with some of these options and ensure you do not test on a production environment, don't say I did not warn you ๐Ÿ™‚

Here is the list of 47 HA Advanced Configurations, the ones with links have been publicly documented by either Duncan or VMware. The rest have not been documented and I have done some testing which will be discussed further.

das.allowNetwork
das.allowVMotionNetworks
das.appMonFailureInterval - #
das.bypassNetCompatCheck
das.checkVmStateDelay - # (default 120)
das.consoleNode - [fqdn]
das.consolePerm - [PERM_USER,PERM_OPER,PERM_ALL]
das.consoleUser - [username]
das.defaultFailoverHost
das.disableUWSwapRequirement
das.failureDetectionInterval
das.failureDetectionTime
das.failureInterval
das.ignoreRedundantNetWarning
das.includeFTcomplianceChecks
das.iostatsInterval
das.isolationAddress
das.isolationShutdownTimeout
das.maxFailures
das.maxFailureWindow
das.maxftvmrestartcount - # (default 5)
das.maxFtVmsPerHost
das.maxvmrestartcount
das.minUptime
das.perHostConcurrentFailoversLimit
das.perHostVMLimit - #
das.powerOffOnIsolation
das.preferredPrimaries
das.primaryCount - [2,3,4,5] (default 5)
das.restartPriority
das.sensorPollingFreq
das.sim.enabled
das.sim.numVms
das.sim.powerOpsPerMinute
das.sim.resourceOpsPerMinute
das.sim.sendInterval
das.slotCpuInMHz
das.slotMemInMB
das.slotNumVcpus
das.trace - [on,off] (default on)
das.traceLevel - [0,1,2,3,4,functiontracing] (default 3)
das.traceOutput - [stdout, eventLog, file] (default file)
das.useDefaultIsolationAddress
das.vmCpuMinMHz
das.vmFailoverEnabled
das.vmMemoryMinMB

You can actually use AAM's (VMware HA) cli utility to view some of these hidden configurations, by using /opt/vmware/aam/bin/Cli

Here is a screenshot of the default VMware HA configuration using the getrule VMWareClusterManager command to list cluster rules, make a note of the parameters with red arrows:

Note: I found it interesting that the "VMWareClusterManager" had an upper case "W" in VMware. It looks like this name was not vetted by VMware marketing ๐Ÿ™‚

Next, we add some of the advanced HA configurations using the vSphere Client under Advanced Options (HA) section:

Note: das.primaryCount only supports values 2 through 5, if you try to specify any other value, you will get an error. VMware has selected 5 primary nodes for a specific reason, you should not try to change it from the default.

We reconfigure the cluster with these new HA options and re-run getrule VMWareClusterManager and now we see the values get updated along with some new ones being added:

Note: You can also see some of these changes in /var/log/vmware/aam/aam_config_util_addnode.log

You can also use AAM cli to set some of these configurations:

Usage: setRuleVar|srv
AAM> setRuleVar VMWareClusterManager ft_Trace on

The advanced VPXA configurations applies to the vCenter Agent configuration file located in /etc/opt/vmware/vpxa/vpxa.cfg on an ESX(i) host. There was a total of 47 with only 8 being used in the default vpxa.cfg file.

Here is the list of 47 VPXA Configurations, the ones listed in blue are used by default, the ones with links are documented by VMware and the rest are undocumented:

vpxa.alarm.checkFrequency
vpxa.alarm.powerOnSilence
vpxa.bundleVersion
vpxa.cfg
vpxa.checkAAMPeriod
vpxa.checkNodeResourcePeriod
vpxa.connectTo
vpxa.disableOldQuickStats
vpxa.editionCheckTime
vpxa.ft.cleanupTimeout
vpxa.getchanges.delay
vpxa.getchanges.envbrowserRefreshRate
vpxa.getchanges.timeout
vpxa.healthSystem.throttleInterval
vpxa.heartbeat.interval
vpxa.HostdCnxBypassProxy
vpxa.hostdMonitorPeriod
vpxa.hostIp
vpxa.hostKey
vpxa.hostPort
vpxa.ignoreAamErrorCount
vpxa.ignoreAamErrorTime
vpxa.ignoreAamShutdownCount
vpxa.licenseExpiryNotificationThreshold
vpxa.maxEventReportingDelaySeconds
vpxa.maxFds
vpxa.memoryCheckerTimeInSecs
vpxa.mmapThresholdInKB
vpxa.nodeResourceThreshold
vpxa.partialVmUpdates
vpxa.preserveServerIp
vpxa.respool.changeThreshold
vpxa.respool.changeThresholdInMB
vpxa.respool.enableOverheadLimitTightLoop
vpxa.serverIp
vpxa.serverPort
vpxa.soapAdapterOverNamedPipe
vpxa.soapAdapterPort
vpxa.sslVersion
vpxa.StopMemInMB
vpxa.timeDiffThreshold
vpxa.useSsl
vpxa.vmapTimeoutCountThreshold
vpxa.vmapTimeoutTimeThreshold
vpxa.vmImages
vpxa.vmotion.vmIdAcquireTimeout
vpxa.WarnMemInMB

Note: The VPXA parameters is pretty difficult to reverse engineer and figure out what values each parameter can accept. I did notice the "periods" within each key actually represents the XML node separation within the vpxa.cfg.

Taking vpxa.vmotion.vmIdAcquireTimeout would translate to the following:

<vpxa>
    <vmotion>
         <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
</vpxa>

Again, please use extreme caution if you decide to experiment with these parameters and happy hacking ๐Ÿ™‚

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Categories // Uncategorized Tags // das, ha, vpxa

ESXi syslog caveat

06.24.2010 by William Lam // 5 Comments

There are two easy methods of capturing logs for both an ESX or ESXi server:

  1. Use VMware vMA and vilogger to setup a syslog server. Simon Long wrote a very detailed article on how to configured and set it all up, check it outย here.
  2. Configure ESX or ESXi server to remotely log to an existing syslog server.

In ESXi, there are potentially three sets of logs to be captured: messages, hostd, and vpxa (if your host is being managed by vCenter). If you are using option #1 and you have enabled logging using vilogger, you will see the following:

If you are using option #2, there is a caveat to be aware of if you are using vCenter to manage your ESXi host, the vpxa logs are not actually being captured.

Here is an ESXi 4.0 Update 1 host that is configured to remotely log to a syslog server, tailing the logs you will notice only theย messages and hostd logs:

I was recently made aware of this problem from VMTN community user aenagy who ran into this in his environment. After filing a support request with VMware, he found out there is an entry that is not added to vpxa.cfg configuration file when the host is joined to a vCenter. Per VMware, it was not considered a bug, but I would disagree.

To enable vpxa log capture, you will need to login to the unsupported Busybox console, also known as Tech Support Mode, for instructions please take a look at this VMware KB. You need to edit the following /etc/opt/vmware/vpxa/vpxa.cfg this is the configuration file for the vCenter agent running on the ESXi host:

<outputToSyslog>true<outputToSyslog>
<syslog>
    <ident>vpxa</ident>
    <facility>local4</facility>
</syslog>

Append the above entries between the ... tags. Once you have updated the vpxa.cfg file, you will want to run the follow command on the Busybox console to ensure the changes are saved and backed up to the local bootbank for ESXi. There is an automated cron job that runs every hour which calls /sbin/auto-backup.sh

You can just run that command manually which will backup the changes and you will see the diff at the bottom with the changes that were made:

~ # /sbin/auto-backup.sh
config implicitly loaded
local.tgz
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_rsa_host_key
etc/opt/vmware/vpxa/vpxa.cfg
etc/opt/vmware/vpxa/dasConfig.xml
etc/opt/vmware/aam/configBackup.tar.gz
etc/opt/init.d/vmware-aam
etc/sysconfig/network
etc/vmware/hostd/authorization.xml
etc/vmware/hostd/hostsvc.xml
etc/vmware/hostd/pools.xml
etc/vmware/hostd/vmAutoStart.xml
etc/vmware/hostd/proxy.xml
etc/vmware/ssl/rui.crt
etc/vmware/ssl/rui.key
etc/vmware/vmkiscsid/initiatorname.iscsi
etc/vmware/vmkiscsid/iscsid.conf
etc/vmware/config
etc/vmware/dvsdata.db
etc/vmware/esx.conf
etc/vmware/license.cfg
etc/vmware/locker.conf
etc/vmware/snmp.xml
etc/vmware/vmware.lic
etc/dhclient-vmk0.leases
etc/group
etc/hosts
etc/inetd.conf
etc/chkconfig.db
etc/passwd
etc/random-seed
etc/resolv.conf
etc/shadow
etc/syslog.conf
etc/sfcb/repository/root/interop/cim_indicationfilter.idx
etc/sfcb/repository/root/interop/cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/cim_indicationsubscription.idx
--- /etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:48:34 2010
+++ /tmp/auto-backup.26548.dir/etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:47:00 2010
@@ -8,11 +8,6 @@
10
1048576
false
- false
-
- vpxa
- local4
-
error
config implicitly loaded
Saving current state in /bootbank
Clock updated.
Time: 18:48:38 Date: 06/23/2010 UTC

Once the backup has completed, you will need to reboot the host before the configuration can take effect. Once your host is back online, if you take a look at the logs on your syslog server, you will notice now that vpxa logs are also being captured:

Hopefully VMware resolves this, because this is definitely a bug and can be a painful one to remediate, especially if you are managing a few hundred ESXi hosts.

Thanks to aenagy for the information.

UPDATE:
After the blog post, I received a few questions:

1) The first question was if the modification would persist from an upgrade and I can confirm it does. My test was originally from an ESXi 4.0u1 and after the change, I upgraded the host using VUM to 4.0u2 and the changes were preserved. The reason for this is because the /sbin/auto-backup.sh was executed which backs up the ESXi configuration and is reloaded upon ever reboot including upgrades.

2) The other question was around automating this change and the answer is yes! Assuming you have SSH access enabled on your ESXi host, you can use the following "ash" script (ESXi does not have a bash) to push the changes out to your host.

You can download the script here called esxiVPXASyslogFix.sh and make sure you have the right permissions on the script to execute.

The script will first check to see if the entry exists, if it does not, it will then update the vxpa.cfg and automatically /sbin/auto-backup.sh

Once the backup has completed, you will see a message that asks you to reboot the host for the changes to take effect.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Categories // Uncategorized Tags // esxi4, syslog, vpxa

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Support

Recent

  • ESXi running in unexpected places ... 05/20/2022
  • Quick Tip - Adding a vTPM (Virtual Trusted Platform Module) to a Nested ESXi VM 05/13/2022
  • vSphere Event-Driven Automation using VMware Event Router on VMware Cloud on AWS with Knative or AWS EventBridge 05/10/2022
  • Integrating VMware Event Broker Appliance (VEBA) with Zapier 04/28/2022
  • Using Terraform to activate Tanzu Kubernetes Grid Service on VMware Cloud on AWS 04/27/2022

Advertisment

Copyright WilliamLam.com © 2022