ESXi syslog caveat

There are two easy methods of capturing logs for both an ESX or ESXi server:

1. Use VMware vMA and vilogger to setup a syslog server. Simon Long wrote a very detailed article on how to configured and set it all up, check it out here.
2. Configure ESX or ESXi server to remotely log to an existing syslog server.

In ESXi, there are potentially three sets of logs to be captured: messages, hostd, and vpxa (if your host is being managed by vCenter). If you are using option #1 and you have enabled logging using vilogger, you will see the following:

If you are using option #2, there is a caveat to be aware of if you are using vCenter to manage your ESXi host, the vpxa logs are not actually being captured.

Here is an ESXi 4.0 Update 1 host that is configured to remotely log to a syslog server, tailing the logs you will notice only the messages and hostd logs:

I was recently made aware of this problem from VMTN community user aenagy who ran into this in his environment. After filing a support request with VMware, he found out there is an entry that is not added to vpxa.cfg configuration file when the host is joined to a vCenter. Per VMware, it was not considered a bug, but I would disagree.

To enable vpxa log capture, you will need to login to the unsupported Busybox console, also known as Tech Support Mode, for instructions please take a look at this VMware KB. You need to edit the following /etc/opt/vmware/vpxa/vpxa.cfg this is the configuration file for the vCenter agent running on the ESXi host:

<outputToSyslog>true<outputToSyslog>
<syslog>
    <ident>vpxa</ident>
    <facility>local4</facility>
</syslog>

Append the above entries between the ... tags. Once you have updated the vpxa.cfg file, you will want to run the follow command on the Busybox console to ensure the changes are saved and backed up to the local bootbank for ESXi. There is an automated cron job that runs every hour which calls /sbin/auto-backup.sh

You can just run that command manually which will backup the changes and you will see the diff at the bottom with the changes that were made:

~ # /sbin/auto-backup.sh
config implicitly loaded
local.tgz
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_rsa_host_key
etc/opt/vmware/vpxa/vpxa.cfg
etc/opt/vmware/vpxa/dasConfig.xml
etc/opt/vmware/aam/configBackup.tar.gz
etc/opt/init.d/vmware-aam
etc/sysconfig/network
etc/vmware/hostd/authorization.xml
etc/vmware/hostd/hostsvc.xml
etc/vmware/hostd/pools.xml
etc/vmware/hostd/vmAutoStart.xml
etc/vmware/hostd/proxy.xml
etc/vmware/ssl/rui.crt
etc/vmware/ssl/rui.key
etc/vmware/vmkiscsid/initiatorname.iscsi
etc/vmware/vmkiscsid/iscsid.conf
etc/vmware/config
etc/vmware/dvsdata.db
etc/vmware/esx.conf
etc/vmware/license.cfg
etc/vmware/locker.conf
etc/vmware/snmp.xml
etc/vmware/vmware.lic
etc/dhclient-vmk0.leases
etc/group
etc/hosts
etc/inetd.conf
etc/chkconfig.db
etc/passwd
etc/random-seed
etc/resolv.conf
etc/shadow
etc/syslog.conf
etc/sfcb/repository/root/interop/cim_indicationfilter.idx
etc/sfcb/repository/root/interop/cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/cim_indicationsubscription.idx
--- /etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:48:34 2010
+++ /tmp/auto-backup.26548.dir/etc/opt/vmware/vpxa/vpxa.cfg Wed Jun 23 18:47:00 2010
@@ -8,11 +8,6 @@
10
1048576
false
- false
-
- vpxa
- local4
-
error
config implicitly loaded
Saving current state in /bootbank
Clock updated.
Time: 18:48:38 Date: 06/23/2010 UTC

Once the backup has completed, you will need to reboot the host before the configuration can take effect. Once your host is back online, if you take a look at the logs on your syslog server, you will notice now that vpxa logs are also being captured:

Hopefully VMware resolves this, because this is definitely a bug and can be a painful one to remediate, especially if you are managing a few hundred ESXi hosts.

Thanks to aenagy for the information.

After the blog post, I received a few questions:

1) The first question was if the modification would persist from an upgrade and I can confirm it does. My test was originally from an ESXi 4.0u1 and after the change, I upgraded the host using VUM to 4.0u2 and the changes were preserved. The reason for this is because the /sbin/auto-backup.sh was executed which backs up the ESXi configuration and is reloaded upon ever reboot including upgrades.

2) The other question was around automating this change and the answer is yes! Assuming you have SSH access enabled on your ESXi host, you can use the following "ash" script (ESXi does not have a bash) to push the changes out to your host.

You can download the script here called esxiVPXASyslogFix.sh and make sure you have the right permissions on the script to execute.

The script will first check to see if the entry exists, if it does not, it will then update the vxpa.cfg and automatically /sbin/auto-backup.sh

Once the backup has completed, you will see a message that asks you to reboot the host for the changes to take effect.