WilliamLam.com

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 2

by // 7 Comments

In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.

To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:

[vmsyslog-logger]
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp

Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:

Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.

Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:

esxcli system syslog reload

You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:

If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.

One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1

by // 17 Comments

Using the new vCenter Log Insight product, you can easily forward application logs from various products within the vCloud Suite for easy analysis and troubleshooting. However, one very important set of logs that we have not been able to collect in the past is the virtual machine logs (vmware.log) which are stored in the working directory of a virtual machine. These logs can be extremely useful from a VMware GSS perspective such as when a virtual machine panics, or if you need to rebuild the .VMX configuration file using these logs or for even general VM auditing purposes.

A recent conversation that I had with Daniel de Sao Jose, who works in our VMware GSS organization reminded of a neat little vSphere 5.1 feature that Daniel had shared with me awhile back. The feature allows you to configure a virtual machine to forward its vmware.log to ESXi's syslog file as well as storing them in the virtual machine's working directory. At the time, there were still a few open questions that required some additional testing and I made a note of this on my ever growing to-do list. I finally around to this and finish up the testing.

UPDATE 1 (04/25/18) - In ESXi 6.7, the ability to forward a VM's vmware.log to an external syslog server also been restored and along with the change, enabling this configuration has been simplified. Instead of having multiple entries to enable the feature and specifying a unique string, you now only have to add a single entry which is vmx.log.syslogID to your VM. The value should be a unique string identifier that the VMX associates with the VM in the syslog. For example, if I use the value of "foo", then the VMX ID will be replaced with "foo" when searching through your syslog entries.

UPDATE 2 (05/04/18) - In ESXi 6.5, 6.5 Update 1 & 6.5 Update 2, the ability to forward a VM's vmware.log to an external syslog server has also been restored and along with the change, enabling this configuration has also been simplified. Simliar to ESXi 6.7, you now only have to add a single entry which is vmx.log.syslogID to your VM. The only difference is that the unique string provided WILL NOT replace the VMX ID in the syslog entry. If you desire the original behavior, you will need to use vSphere 6.7.

To enable this feature, you will need to add the following advanced virtual machine setting:

vmx.log.destination = "syslog-and-disk"

This of course can be enabled using either the vSphere Web Client or vSphere C# Client as well as automated, take a look at this article for more details.

Here is a screenshot showing showing the contents of the vmware.log in the ESXi host's syslog which is located in /var/log/syslog:

Note: The vmware.log is only generated when a virtual machine is powered on.

You also have the option of disabling the local vmware.log from being created in the virtual machine's working directory and only forwarded to ESXi host's syslog. To do so, you would change the advanced virtual machine setting to the following:

vmx.log.destination = "syslog"

By default, the log entries will be identified by the keyword vmx and the specific virtual machine's process ID such as vmx[5313]. However, this is not very user friendly and would still require you to query the VM PID to get the virtual machine name. This can be a challenge if you are viewing the logs from a centralized syslog server such as vCenter Log Insight where you potentially could have logs being forwarded from hundreds if not thousands of ESXi hosts.

To help with this, you can specify the string in which the virtual machine will identify itself when forwarding its logs using the following advanced virtual machine setting:

vmx.log.syslogID = SOME STRING

It made the most sense to me to set this to the name of the virtual machine, so you can easily identify the source of the logs. Here is a screenshot showing the name of the virtual machine instead of the generic "vmx" string.

If you have configured your ESXi host to forward its logs to vCenter Log Insight, you can see how easy it is to view individual virtual machine logs with a click of a button isolating on the syslog source.

One caveat that I would like to mention with this solution is that you are now storing all virtual machine logs in the ESXi hosts syslog file which is also logging other things about the ESXi host. This would cause the local logs to rotate much more frequently on the ESXi host due to the verbosity when powering on and off a virtual machine. This may not be an issue if you are forwarding to a remote syslog server, but ideally it would be nice to have separate log file primarily for the virtual machine logs. In Part 2 of this article, we will take a look at how we can accomplish this by extending ESXi's logger component.

How To "Pause" (Not Suspend) A Virtual Machine In ESXi?

by // 14 Comments

Last week I received a very interesting question from a fellow blogger asking whether it was possible to "pause" (not suspend) a virtual machine running on ESXi. Today ESXi only supports the suspend operation which saves the current memory state of a virtual machine to disk. With a "pause" operation, the memory state of the virtual machine is not saved to disk, it is still preserved in the physical memory of the ESXi host. The main difference with a "pause" operation is the allocated memory is not released and this allows you to quickly resume a virtual machine almost instantly at the cost of holding onto physical memory.

The use case for this particular request was also quite interesting. The user had an NFS server that housed about 200 virtual machines that needed to be restarted and the goal was to minimize the impact to his virtual machines as much as possible. He opted out from suspending the virtual machines as it would have taken too long and decided on a more creative solution. He filled up the remainder capacity on the datastore which in effect caused all virtual machines to halt their I/O operations. Though not an ideal solution IMHO, this allowed him to restart the NFS server and then run a script for the virtual machines to retry their I/O operation once the NFS server was available again.

Based on the above scenario, he asked if it was possible to "pause" the virtual machines similar to a capability Hyper-V provides today which would have provided him a quicker way to resume the virtual machines. Thinking about the question for a bit, a virtual machine is just a VMX process running in ESXi and I wondered if this process could be paused like a UNIX/Linux process using the "kill" command. Well, it turns out, it can be!

Disclaimer: This is not officially supported by VMware, use at your own risk.

Using the kill command, you can pause the VMX process by sending the STOP signal and to resume the VMX process, you can send the CONT signal. Before getting started, you will need to identify the PID (Process ID) for the virtual machine's VMX process.

There are two methods of identifying the parent VMX PID, the easiest is using the following ESXCLI command:

esxcli vm process list


The PID for the virtual machine will be listed under the "VMX Cartel ID" and in this example I have a virtual machine called vcenter51-1 and on the right I am pinging the system to verify it is up and running. An alternative way of identifying the PID is to use "ps" by running the following command:

ps -c | grep -v grep | grep [vmname]

Note: Make sure you identify the parent PID of the virtual machine if you are using the above command as you will see multiple entries for the different VMX sub-processes.

To pause the VMX process, run the following command (substitute your PID):

kill -STOP [pid]

To resume VMX process, run the following command:

kill -CONT [pid]

Here is a screenshot of pausing and then resuming the virtual machine. You can also see where the pings stop as the virtual machine is paused and then resumed. Once the virtual machine was resumed, it operated exactly where it left off with no issues as far as I can tell.
 
Note: I have found that if you have VM monitoring enabled, there maybe issues resuming the virtual machine. This should only be done if you have VM monitoring disabled as it may not be properly aware that the VMX process being paused on purpose.

Though it is possible to pause a virtual machine, I am not sure I see too many valid use cases for this feature? Are there are use cases where this feature would actually be beneficial, feel free to leave a comment if you believe there are. For now, this is just another neat "notsupported" trick 😉