When it comes to troubleshooting, something that you can never have too much of are logs! However, you can have excessive logs in the form of repeated log entries for a particular event which not only adds to the amount of logs you must sift through but it also adds unnecessary load and processing to the network. This is especially problematic if you these repeated entries also being forwarded from multiple sources to a centralized syslog server.
With earlier release of ESX (yes, classic ESX), it was possible to filter out specific log entries from the host side and prevent them from showing up in the local logs stored on the filesystem after N-occurrences and would also prevent them from being forwarded to a syslog server. However, when ESXi was first introduced, this particular capability wast not ported over which I can only assume was based on usage from our customer base. In speaking with GSS, this is usually something they require for troubleshooting purposes, although I have also seen a few customers ask about this capability on several occasions in the past.
Having said that, I was pleasantly surprised to learn from Alan Castonguay (former GSS Engineer, now working over in our MBU) that ESXi 6.0 actually now includes a new log filtering capability or rather I should say, it has re-introduced the log filtering capability 🙂
To enable log filtering, you will need to add the following parameter to /etc/vmsyslog.conf
enable_logfilters = true
To add a specific log filter, you will need to add an entry to /etc/vmware/logfilters using the following format: numLogs | ident | logRegexp
- numLogs - Number of times the log entry can appear in the log before it is then filtered and ignored
- ident - The ident source of the log, you can find all ident by looking in /etc/vmsyslog.conf.d/*.conf
- logRegexp - regular expression confirming to the Python regular expression syntax
Below is an example where I only want the log entry "SOCKET connect failed, error 2: No such file or directory.*" to show up only two times in the logs before it is filter and ignored and the source of this log entry is from the hostd logs.
2| Hostd | SOCKET connect failed, error 2: No such file or directory.*
UPDATE (03/18/16): I just got a heads up from GSS that the "hostd" logs is actually case sensitive when creating the filter. You will need to use a capital "H" in case you are not seeing results from your regexp.
Once you have created all your log filters, you will need to reload the syslog service by running the following ESXCLI command:
esxcli system syslog reload
One thing to be aware of is that once a log entry has been filtered out and the local logs have been rotated out, that particular entry will no longer show up in future logs. It is definitely recommended that you use the log filtering feature sparingly and ensure that you are also forwarding your logs to a centralized syslog server like Log Insight for example so that you have all log entries at your disposal for both troubleshooting and auditing purposes. There is already a request to add this to our official VMware documentation so that it customers can easily find this in the syslog configuration section of the ESXi documentation, but for now I have documented it here so others can benefit from this capability if needed.