When it comes to troubleshooting, something that you can never have too much of are logs! However, you can have excessive logs in the form of repeated log entries for a particular event which not only adds to the amount of logs you must sift through but it also adds unnecessary load and processing to the network. This is especially problematic if you these repeated entries also being forwarded from multiple sources to a centralized syslog server.
With earlier release of ESX (yes, classic ESX), it was possible to filter out specific log entries from the host side and prevent them from showing up in the local logs stored on the filesystem after N-occurrences and would also prevent them from being forwarded to a syslog server. However, when ESXi was first introduced, this particular capability wast not ported over which I can only assume was based on usage from our customer base. In speaking with GSS, this is usually something they require for troubleshooting purposes, although I have also seen a few customers ask about this capability on several occasions in the past.
Having said that, I was pleasantly surprised to learn from Alan Castonguay (former GSS Engineer, now working over in our MBU) that ESXi 6.0 actually now includes a new log filtering capability or rather I should say, it has re-introduced the log filtering capability 🙂
To enable log filtering, you will need to add the following parameter to /etc/vmsyslog.conf
enable_logfilters = true
To add a specific log filter, you will need to add an entry to /etc/vmware/logfilters using the following format: numLogs | ident | logRegexp
- numLogs - Number of times the log entry can appear in the log before it is then filtered and ignored
- ident - The ident source of the log, you can find all ident by looking in /etc/vmsyslog.conf.d/*.conf
- logRegexp - regular expression confirming to the Python regular expression syntax
Below is an example where I only want the log entry "SOCKET connect failed, error 2: No such file or directory.*" to show up only two times in the logs before it is filter and ignored and the source of this log entry is from the hostd logs.
2| Hostd | SOCKET connect failed, error 2: No such file or directory.*
UPDATE (03/18/16): I just got a heads up from GSS that the "hostd" logs is actually case sensitive when creating the filter. You will need to use a capital "H" in case you are not seeing results from your regexp.
Once you have created all your log filters, you will need to reload the syslog service by running the following ESXCLI command:
esxcli system syslog reload
One thing to be aware of is that once a log entry has been filtered out and the local logs have been rotated out, that particular entry will no longer show up in future logs. It is definitely recommended that you use the log filtering feature sparingly and ensure that you are also forwarding your logs to a centralized syslog server like Log Insight for example so that you have all log entries at your disposal for both troubleshooting and auditing purposes. There is already a request to add this to our official VMware documentation so that it customers can easily find this in the syslog configuration section of the ESXi documentation, but for now I have documented it here so others can benefit from this capability if needed.
Dan Kahl says
Thanks William - great post! Do you know if I can tag the messages for ESXi hosts as well? I'm trying to see if I can tag the logs before sending them off to my syslog server.
William Lam says
I don't believe so.
This is cool. Thanks William for sharing. Once a filter is applied for a specific phrase, for how long the repetitive logs would be ignored? Until the log rotation or is there any specific time frame?
Will it work also on esxi 6.5 and 7.0
Graeme Hay says
Magic advice on the capitalization of the first letter in the ident!! Been banging my head against the desk until I read your post. Thanks! You'd think VMW would have that in their docs!