In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.
To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:
[vmsyslog-logger]
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp
Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:
Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.
Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:
esxcli system syslog reload
You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:
If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.
One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.
