In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.
To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp
Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:
Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.
Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:
esxcli system syslog reload
You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:
If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.
One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.
Keith Symmonds says
Very interesting couple of posts. It's got me thinking. Why stop at VM products. Surely, if in a syslog format, couldn't anything we sent and indexed by Log Insight? My thoughs are Apache logs from a Web App VM? What are your thoughts?
William Lam says
Agreed. As long as you can send out syslog, vCenter Log Insight can consume it. What would be even cooler is to have other 3rd party start to create content packs or even community members start doing so based on their expertise on certain products.
party venues in los angeles says
I do like the manner in which you have presented this issue plus it does provide us some fodder for thought. Nonetheless, from everything that I have witnessed, I just wish good luck to you!
Shady Ali says
Thanks for this great post..
I wanna a small tip.. I'm using VMware Syslog Collector in my environment.. how to forward the new log of VM to it..??!!!
Shady Ali says
to make it more clear, on my Syslog Collector machine, the log file is only loggin entries from vpxa.log, hostd.log and some other logs. It doesn't log entries from the log file I created for the VM, however the log file already exists in /var/log.
is there any config file that I should add to fix that..??
Shady Ali says
I figured it out 🙂
All I had to do is to restart management agents using (services.sh restart)..
After all services were up, the new vmx logger appeared in Host Adv. Setting (Syslog->Loggers)
and syslog.log on VMware Syslog Collector began to catch logs from that vmx.log 🙂
Thanks so much again for your wonderful article :))