vSphere 6.0 – Page 32

Log filtering capability in ESXi 6.0

05.20.2015 by William Lam // 5 Comments

When it comes to troubleshooting, something that you can never have too much of are logs! However, you can have excessive logs in the form of repeated log entries for a particular event which not only adds to the amount of logs you must sift through but it also adds unnecessary load and processing to the network. This is especially problematic if you these repeated entries also being forwarded from multiple sources to a centralized syslog server.

With earlier release of ESX (yes, classic ESX), it was possible to filter out specific log entries from the host side and prevent them from showing up in the local logs stored on the filesystem after N-occurrences and would also prevent them from being forwarded to a syslog server. However, when ESXi was first introduced, this particular capability wast not ported over which I can only assume was based on usage from our customer base. In speaking with GSS, this is usually something they require for troubleshooting purposes, although I have also seen a few customers ask about this capability on several occasions in the past.

Having said that, I was pleasantly surprised to learn from Alan Castonguay (former GSS Engineer, now working over in our MBU) that ESXi 6.0 actually now includes a new log filtering capability or rather I should say, it has re-introduced the log filtering capability 🙂

To enable log filtering, you will need to add the following parameter to /etc/vmsyslog.conf

enable_logfilters = true

To add a specific log filter, you will need to add an entry to /etc/vmware/logfilters using the following format: numLogs | ident | logRegexp

numLogs - Number of times the log entry can appear in the log before it is then filtered and ignored
ident - The ident source of the log, you can find all ident by looking in /etc/vmsyslog.conf.d/*.conf
logRegexp - regular expression confirming to the Python regular expression syntax

Below is an example where I only want the log entry "SOCKET connect failed, error 2: No such file or directory.*" to show up only two times in the logs before it is filter and ignored and the source of this log entry is from the hostd logs.

2| Hostd | SOCKET connect failed, error 2: No such file or directory.*

UPDATE (03/18/16): I just got a heads up from GSS that the "hostd" logs is actually case sensitive when creating the filter. You will need to use a capital "H" in case you are not seeing results from your regexp.

Once you have created all your log filters, you will need to reload the syslog service by running the following ESXCLI command:

esxcli system syslog reload

One thing to be aware of is that once a log entry has been filtered out and the local logs have been rotated out, that particular entry will no longer show up in future logs. It is definitely recommended that you use the log filtering feature sparingly and ensure that you are also forwarding your logs to a centralized syslog server like Log Insight for example so that you have all log entries at your disposal for both troubleshooting and auditing purposes. There is already a request to add this to our official VMware documentation so that it customers can easily find this in the syslog configuration section of the ESXi documentation, but for now I have documented it here so others can benefit from this capability if needed.

Configuring per-VMDK IOPS reservations in vSphere 6.0

05.20.2015 by William Lam // 1 Comment

One of the new features in vSphere 6.0 that was quickly mentioned at the end of Duncan Epping's What's New Storage DRS blog post is the ability to configure an IOPS reservation on a per-VMDK basis which is now integrated with both Storage IO Control (SIOC) and Storage DRS. As Duncan mentioned at the end of his article, this feature is only consumable through the vSphere API and given that, it may not be a feature that is widely known or used. This topic had recently surfaced in an internal thread on how to set the IOPS reservations and below are the details if you wish to leverage this new vSphere 6.0 storage platform capability.

To be able to use this new feature, there are two requirements:

You need to set the IOPS reservation value on an individual VMDK which is under the StorageIOAllocationInfo property of a VMDK called, not surprisingly, reservation.
All ESXi hosts mounting the Datastore must be running vSphere 6.0

To be clear, this reservation property has been around since vSphere 5.5, but had only had support for local datastores. In vSphere 6.0, shared datastores are now supported along with both SIOC and Storage DRS being aware.

To exercise the use of this vSphere API, I have created a simple PowerCLI script called configurePerVMDKIOPS.ps1 (works with both vSphere 5.x & 6.0) which you will need to edit to include your vCenter Server, the name of the VM you wish to set the IOPS reservation along with the VMDK label and IOPS value.

Here is an example output for configuring a VM named Photon with IOPS reservations of 2000 on Hard Disk 1:

I have been told that in the future, the plan is to make this configurable available in the vSphere Web Client. Though, honestly why would anyone want to perform this change across multiple VMs by hand, when you can quickly and efficiently automate this across your environment with a simple script? 😉

How to download offline VSAN HCL file for VSAN Health Check Plugin?

05.16.2015 by William Lam // 4 Comments

One of the coolest features in the new VSAN Health Check Plugin is the automatic verification of your underlying hardware (hosts, disks, storage controller & drivers) by automatically checking it against VMware's VSAN HCL (Hardware Compatibility List).

The VSAN HCL database can either be downloaded automatically from VMware.com or manually uploaded if you do not have direct or proxy internet access. There was a question this morning on Twitter asking where the offline VSAN HCL file be downloaded from? I was actually curious as well and looking at Cormac Hogan's excellent VSAN Health Check documentation, I found the answer at the very end of the document 🙂

http://partnerweb.vmware.com/service/vsan/all.json

To download the offline VSAN HCL file which is actually is just a JSON file, you just need to load the above URL into a web browser and then save the file.

After you have downloaded the VSAN HCL file, you can either upload using the vSphere Web Client under the "Health" section of the VSAN Health Plugin or you using the following RVC command and specifying the path to the file:

vsan.health.hcl_update_db /localhost/ -l /root/all.json

As a bonus, I also had some fun parsing the VSAN HCL JSON file. Below is a graph that I was able to generate after extracting some useful information using the following script found here.