syslog – Page 5

Automate forwarding of vRealize Operations Manager logs to Syslog Server

01.06.2015 by William Lam // 3 Comments

Forwarding logs to a syslog server from vRealize Operations Manager has been greatly simplified with the latest 6.0 release. Instead of having to manually tweak syslog-ng.conf with past releases, there is now a new UI option under Administration->Support->Logs which allows you to easily configure syslog configurations. There are currently 4 loggers (Analytics, Collector, Web & SuiteAPI) that can be configured to be forwarded to a syslog server.

To configure using the vROps UI, you select the root logger folder and then click on the gears icon at the top. From here, you can configure the syslog server as well as the syslog port.

Though this is a nice UI enhancement, it would have been really nice to have been able to configure this using an API. In doing some digging, I found that you could still automate the syslog configurations by simply adding a couple of lines into the respective log4j.properties files:

vROps Logger	Configuration File
Analytics	/usr/lib/vmware-vcops/user/conf/analytics/log4j.properties
Collector	/usr/lib/vmware-vcops/user/conf/collector/log4j.properties
Web	/usr/lib/vmware-vcops/user/conf/web/log4j.properties
SuiteAPI	/usr/lib/vmware-vcops/tomcat-enterprise/webapps/suite-api/WEB-INF/log4j.properties

Here is a simple shell script called configurevROpsSyslog.sh that you can run against your vROps instance to forward your logs. There are two variables SYSLOG_SERVER and SYSLOG_SERVER_PORT that will need to be updated before running the script. Here is an example of running the script that can come in handy to forwarding your logs to your favorite syslog server such as vRealize Operations Insight.

UPDATE (01/17/15) - Thanks to Steve Flanders for letting me know there is one additional location for configuring syslog which is the Audit logs located in under Administration->Audit. It looks like the configuration also goes into the Collector log4j.properties and I have updated my script to ensure this is also configured as part of the script.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 2

07.10.2013 by William Lam // 7 Comments

In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.

To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:

[vmsyslog-logger]
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp

Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:

Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.

Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:

esxcli system syslog reload

You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:

If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.

One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1

07.08.2013 by William Lam // 17 Comments

Using the new vCenter Log Insight product, you can easily forward application logs from various products within the vCloud Suite for easy analysis and troubleshooting. However, one very important set of logs that we have not been able to collect in the past is the virtual machine logs (vmware.log) which are stored in the working directory of a virtual machine. These logs can be extremely useful from a VMware GSS perspective such as when a virtual machine panics, or if you need to rebuild the .VMX configuration file using these logs or for even general VM auditing purposes.

A recent conversation that I had with Daniel de Sao Jose, who works in our VMware GSS organization reminded of a neat little vSphere 5.1 feature that Daniel had shared with me awhile back. The feature allows you to configure a virtual machine to forward its vmware.log to ESXi's syslog file as well as storing them in the virtual machine's working directory. At the time, there were still a few open questions that required some additional testing and I made a note of this on my ever growing to-do list. I finally around to this and finish up the testing.

UPDATE 1 (04/25/18) - In ESXi 6.7, the ability to forward a VM's vmware.log to an external syslog server also been restored and along with the change, enabling this configuration has been simplified. Instead of having multiple entries to enable the feature and specifying a unique string, you now only have to add a single entry which is vmx.log.syslogID to your VM. The value should be a unique string identifier that the VMX associates with the VM in the syslog. For example, if I use the value of "foo", then the VMX ID will be replaced with "foo" when searching through your syslog entries.

UPDATE 2 (05/04/18) - In ESXi 6.5, 6.5 Update 1 & 6.5 Update 2, the ability to forward a VM's vmware.log to an external syslog server has also been restored and along with the change, enabling this configuration has also been simplified. Simliar to ESXi 6.7, you now only have to add a single entry which is vmx.log.syslogID to your VM. The only difference is that the unique string provided WILL NOT replace the VMX ID in the syslog entry. If you desire the original behavior, you will need to use vSphere 6.7.

To enable this feature, you will need to add the following advanced virtual machine setting:

~~vmx.log.destination = "syslog-and-disk"~~

This of course can be enabled using either the vSphere Web Client or vSphere C# Client as well as automated, take a look at this article for more details.

Here is a screenshot showing showing the contents of the vmware.log in the ESXi host's syslog which is located in /var/log/syslog:

Note: The vmware.log is only generated when a virtual machine is powered on.

You also have the option of disabling the local vmware.log from being created in the virtual machine's working directory and only forwarded to ESXi host's syslog. To do so, you would change the advanced virtual machine setting to the following:

~~vmx.log.destination = "syslog"~~

By default, the log entries will be identified by the keyword vmx and the specific virtual machine's process ID such as vmx[5313]. However, this is not very user friendly and would still require you to query the VM PID to get the virtual machine name. This can be a challenge if you are viewing the logs from a centralized syslog server such as vCenter Log Insight where you potentially could have logs being forwarded from hundreds if not thousands of ESXi hosts.

To help with this, you can specify the string in which the virtual machine will identify itself when forwarding its logs using the following advanced virtual machine setting:

vmx.log.syslogID = SOME STRING

It made the most sense to me to set this to the name of the virtual machine, so you can easily identify the source of the logs. Here is a screenshot showing the name of the virtual machine instead of the generic "vmx" string.

If you have configured your ESXi host to forward its logs to vCenter Log Insight, you can see how easy it is to view individual virtual machine logs with a click of a button isolating on the syslog source.

One caveat that I would like to mention with this solution is that you are now storing all virtual machine logs in the ESXi hosts syslog file which is also logging other things about the ESXi host. This would cause the local logs to rotate much more frequently on the ESXi host due to the verbosity when powering on and off a virtual machine. This may not be an issue if you are forwarding to a remote syslog server, but ideally it would be nice to have separate log file primarily for the virtual machine logs. In Part 2 of this article, we will take a look at how we can accomplish this by extending ESXi's logger component.