WilliamLam.com

Forwarding vCenter Server Logs to a Syslog Server

by // 24 Comments

I was recently asked if it was possible to forward vCenter Server logs to a regular syslog server and if so, how difficult would it be to setup? I had researched this topic several years back, but did not find an ideal solution as vCenter Server was only available on the Windows platform and vCenter Server itself did not provide any syslogging capabilities. With the release of vSphere 5.0, VMware introduced the VCSA (vCenter Server Appliance) and realized I never revisited this question for the VCSA.

After a bit of digging, I found that the VCSA comes installed with syslog-ng by default which is used to provide the vSphere Syslog Collector functionality as well as the local syslog client for the VCSA itself. Given this information, it was pretty trivial to source the local /var/log/vmware/vpx/vpxd.log (symlink to latest vCenter Server log as well as other important vCenter logs) and automatically forward that to a remote syslog server.

VCSA Syslog Configuration

You will need to edit the following configuration file on the VCSA - /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

# vpxd source log
source vpxd {
       file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vws.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) flags(no-parse));
};

# Remote Syslog Host
destination remote_syslog {
       udp("172.30.0.45" port (514));
};

# Log vCenter Server vpxd log remotely
log {
        source(vpxd);
        destination(remote_syslog);
};

Note: If you are interested in more details about "sourcing" a local log, take a look at this article here which I used as a reference.

Once you have saved the configuration file, you just need to restart the syslog client by running the following command:

service syslog restart

If you login to your remote syslog server, you should now see that your VCSA is forwarding it's vpxd logs over. Pretty simple, right? 🙂 You can of course forward over other vCenter Server logs by adding additional source files. The main key is that there is a symlink that automatically points to the latest log file which you map as the source file.

I am sure many of you are probably asking what about vCenter Server for Windows? Well, I did also looked into a similar solution but it's a bit more complex than just adding a few configuration entries.

Windows vCenter Server Syslog Configuration

Disclaimer: This is not supported by VMware, please use at your own risk.

There are a few challenges with the Windows version, by default there are no syslog clients installed and there is no automatic symlink to the latest vCenter Server log. Having said that, you can still get the above solution working using the free syslog-ng, but it takes a few more steps. The solution will be leveraging Cygwin, so we can run the free version of syslog-ng on a Windows system.

Step 1 - Install Cygwin and configure syslog-ng service on your vCenter Server as described in this article. You will need to add an additional package which is "Admin/Cron" that will be used in the subsequent steps. In the example, I ran syslog-ng under default system account, but if you need to run it under a different user, you may find these two articles to be helpful

  • http://linux.subogero.com/894/cron-on-cygwin/
  • http://www.davidjnice.com/articles/cygwin_cron-service.html

Step 2 - Just as before, we will need to edit /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

# vpxd source log
source vpxd {
       file("/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs/vpxd.log" follow_freq(1) flags(no-parse));
};

# Remote Syslog Host
destination log_additional_remote_syslog {
       udp("172.30.0.45" port (514));
};

# Log vCenter Server vpxd log remotely
log {
        source(vpxd);
        destination(log_additional_remote_syslog);
}; 

You will notice this time, we are accessing the Windows C drive by using the /cygdrive path

Step 3 - As mentioned earlier, there is no symlink that points to the latest vCenter Server log, which makes it difficult to map to static log file. What we can do is basically identify the latest vpxd-#.log and automatically create a symlink and that is what is being monitored by syslog-ng to forward the log. We will be using a cronjob and a very simple shell script.

You can place the script in the current home directory /home/Administrator (or whatever default user you happen to have installed Cygwin on)

Here is the shell script which I have called latest.sh:

#!/bin/bash

VC_LOG_PATH="/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs"
LATEST=$(ls -tr "/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs/" | grep "vpxd-[0-9]*.log" | grep -v ".gz" | tail -1)

if [ ! -e "${VC_LOG_PATH}/vpxd.log" ]; then
        touch "${VC_LOG_PATH}/vpxd.log"
fi

ln -sf "${VC_LOG_PATH}/${LATEST}" "${VC_LOG_PATH}/vpxd.log"

Make sure to set the script to be executable: chmod +x latest.sh

Step 4 - Create a cronjob which will run every minute (you might be able to set a longer delay depending on your environment and it's rotation frequency) by editing the following file /var/cron/tabs/Administrator or using crontab -e
Step 5 - Start or restart syslog-ng by running one of the following commands:

Start - cygrunsrv -S syslog-ng
Restart - cygrunsrv -E syslog-ng;cygrunsrv -S syslog-ng

If everything was successful, you should start seeing your vCenter Server logs from your Windows system forward to your remote syslog server. When the latest vpxd-#.log changes, the cronjob will automatically take care of re-linking to the latest vpxd-#.log to ensure you continue forwarding your vCenter Server logs.

As you can see, it is not trivial to set this up for the Windows vCenter Server as it is for the VCSA, but you now have a way to centrally store all your important vCenter Server logs for archival or analysis purposes without having to manually copy them off to a remote volume.

Few additional notes:

  • I believe the paid version of syslog-ng supports file globbing, so you do not need to setup a cronjob and just watch for all vpxd-*.log, but in this example, I went with a completely free solution
  • You might also be able to leverage Splunk to monitor vCenter Server logs as noted in this Splunkbase entry, but I have not verified and I am not sure if you have to pay for this feature in Splunk
  • Here is an easier way of forwarding vCenter Server logs on Windows using Snare by Raphael Schitz.

Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm

by // 6 Comments

I was just cleaning up one of my development labs and found that one of my VCSA (vCenter Server Appliance) which I had configured with vSphere Syslog Collector was no longer capturing logs for several of my ESXi hosts. After looking at some of the ESXi logs, I realized that I had rebooted the VCSA at one point and that caused an interruption in syslog forwarding and then knew immediately that I just needed to reload the syslog configuration via ESXCLI as noted in this VMware KB to restore log forwarding.

After restoring my syslog configurations, I had remembered a neat little trick I learned from one of the VMware TAMs about creating a vCenter Alarm to alert you when an ESXi host is no longer able to reach a remote syslog server. I thought this might be very handy alarm to have in your vCenter Server in case you hit a similar issue or having some connectivity issues with your syslog servers. By default, there is not an event on syslog connectivity but you can create a vCenter Alarm based on an eventId which shows up as "esx.problem.vmsyslogd.remote.failure" in both /var/log/hostd.log as well as /var/log/vobd.log.

Now that we know the eventId, we just need to create a vCenter Alarm which will notify us when it has a connectivity issue with it's configured syslog server.

Step 1 - Create a new alarm, in this example I am calling it "Syslog Connection Error" and you will need to specify the Alarm Type as "Host" and monitor for a specific event.

Step 2 - Next, click on Triggers and we will go ahead and paste in our eventId which is "esx.problem.vmsyslogd.remote.failure"

Step 3 - Lastly, you can configure an Action, if you wish to send an SNMP trap, run a command or send an email notification. In this example, we are just going to generate a regular vCenter Alarm event, so go ahead and just click OK to save the alarm.

To test the alarm, I just disabled the syslog-collector on the VCSA using "service syslog-collector stop" and you should see an alarm generate for any ESXi hosts forwarding it's logs to that particular syslog server.

So now when your ESXi hosts can not reach it's syslog server, you will automatically be notified and can look into the problem immediately. Now having an alarm is great ... but you might be wondering what about the need to reload the syslog configuration on all your ESXi hosts to restore syslog forwarding? This can definitely be a challenge/annoying, especially if the syslog server's connectivity is returned after some amount of time and you have hundreds of hosts.

Well luckily, you no longer have to worry about this, with the latest ESXi 5.0 patch03 that was just released, this problem has been addressed and ESXi syslog daemon will automatically start forwarding logs once connectivity has been restored to the syslog server. It is still definitely recommended that you have more than one syslog server in your environment and that they are properly being monitored. Also, do not forget with ESXi 5.0 you can now configure more than one remote syslog server, for more details take a look at this article here.

Note: After applying the patch, you will no longer be able to generate an alarm based on the eventId for syslog when using UDP. You will see something like "Hostd [290D5B90 verbose 'SoapAdapter'] Responded to service state request" in the hostd.log. The alarm will only be valid if you're using TCP or SSL protocol for syslog which have not been patched with latest p03.

If you are looking for a quick way to reload your syslog configurations, you can easily write a simple for loop to reload your ESXi hosts using the remote ESXCLI:

Here is another example using PowerCLI in-conjunction with ESXCLI:

Inactivity Timeout for the vSphere C# Client

by // 2 Comments

I recently came across an interesting VMTN thread in which I learned about a neat little feature that allows a user to configure an inactivity timeout for the vSphere C# Client. Once the timeout value has been reached, the vSphere Client will automatically disconnect from the server(vCenter 5.0 or ESXi 5.0 Server). This feature looks to have been introduced with the release of vSphere 5 and was noted in thick-client-timeout guideline in recent release of the vSphere 5 Security Hardening Guide to help reduce the risk of unauthorized access.

There are two methods you can configure the inactivity timeout value:

  • Command-line argument to vSphere Client executable
  • vSphere Client configuration file (VpxClient.exe.config)

To configure using the command-line argument, locate the vSphere Client executable on your desktop and right click and select properties. You will need to add -inactivityTimeout X, where X is the number of minutes before the vSphere Client will automatically disconnect from the server.

To configure using the vSphere Client configuration file, you will need to locate the VpxClient.exe.config (thanks to user regnak2012 for identifying the required XML entry).

Depending on whether you are running a 32bit or 64bit WindowsOS, they will be located in one of the two places:

  • 32bit - %PROGRAMFILES%\VMware\Infrastructure\Virtual Infrastructure Client\Launcher
  • 64bit - %PROGRAMFILES(x86)%\VMware\Infrastructure\Virtual Infrastructure Client\Launcher

Next, you will need to right click on VpxClient.exe.config and edit the file using an editor such as notepad. Add the following entry right above the cmdlineFallback, where X is the number of minutes for the timeout value and save then save the file.

X
To validate that inactivity timeout value works, just connect to either a vCenter Server 5.0 or ESXi 5.0 Server and wait for it to disconnect after the timeout value has been violated. In this example, I configured it for 1 minute and you should see the following screen below once it has disconnected.

Since this feature is client side only, a user can easily change or update this timeout value. One option to guarantee this configuration is by ThinApp-ing your vSphere Client, this way you can control who has access and what the inactivity timeout should be. This can really help when it comes to auditing client side system.

While looking into the vSphere Client options, I also came across a few others that are not documented but might come in handy.

Disclaimer: These are not officially supported by VMware, please use at your own risk. 

Ignore Certificate - If you have a self-signed SSL certificate when you first connect to an ESX(i) or vCenter Server, you probably have seen the following:

You can automatically ignore this message by specifying the -i command-line option OR add in the VpxClient.exe.config file.

Expand Inventory - If you want to have your entire vSphere inventory automatically expanded out for you each time you login (this may not be a good idea for very large environments) you can specify -expAll command-line option OR add in the VpxClient.exe.config file.

Disable All vSphere Plugins - If you want to prevent any vSphere Plugins from loading, you can specify -noPlugins  command-line option OR add in the VpxClient.exe.config file.