WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Forwarding Logs From The vCloud Suite To vCenter Log Insight

06.17.2013 by William Lam // 18 Comments

An exciting new product was just announced last week by VMware called vCenter Log Insight, which will be part of the vCenter Operations Management Suite when released. The announcement also includes a public beta for customers to try out the new log analytics product that allows administrators to easily get an understanding of both their physical and virtual infrastructure through the collection of log data. You can get more details on how vCenter Log Insight works by checking out this article by the Jon Herlocker, who is in the Office of CTO and focusing on vCenter Log Insight.

I had known about vCenter Log Insight for quite sometime now and like others within VMware, I had the opportunity to test drive the product early on and provide feedback to the engineering team. One of neatest thing about vCenter Log Insight, in my opinion, is the simplistic setup and the tight integration between vCenter Server and vCenter Operations Manager. During the setup of vCenter Log Insight, I was reminded about an article that I had written about forwarding vCenter Server logs to a syslog server. I thought, would it not be cool if we could forward logs from other products within the vCloud Suite to vCenter Log Insight using the same syslog-ng trick? I decided to compile a list of logs from each of the products within the vCloud Suite shared that internally and thanks to my colleague Michael White who also help vet the list by circulating it within engineering.

I then decided to create a very simple script called configurevCloudSuiteSyslog.sh that would allow users to easily configure each of the vCloud Suite products to forward their appropriate logs to vCenter Log Insight. The script is very simple to use, you just need to scp the script to one of the supported appliances within the vCloud Suite and specify the VMware solution name and the IP Address of your vCenter Log Insight Server.

Here is an example of running the script on the VCSA (vCenter Server Appliance):

Based on the VMware solution selected, the appropriate logs will be appended to /etc/syslog-ng/syslog-ng.conf to be forwarded off to your vCenter Log Insight Server. The syslog-ng client will automatically be restarted for the changes to go into effect as part of the script. In my environment, I have deployed the majority of products within the vCloud Suite installed and have configured each of them to forward their logs to vCenter Log Insight. This can be very useful from a troubleshooting perspective and being able to view and filter through all the relevant logs from a single location.

It was really interesting to see what the next "chattiest" VMware solution was from a log perspective in my environment, which turned out to be VIN after vCenter Server and ESXi host. I hope to see deeper integration between vCenter Log Insight and the rest of the vCloud Suite in future releases, but for now, if you have not tried out vCenter Log Insight, I would highly recommend you give it a try and provide any feedback you may have in the dedicated VMTN community forum.

If you are interested in the specifics logs that are being collected for each of VMware products, you can find the complete list below. Not all products from the vCloud Suite are listed here and some such as vCloud Director and vCloud Networking & Security provide native syslog configuration from the application standpoint which can be configured using either their UIs or APIs.

vCenter Operations Manager Analytics (VCOPS):

/var/log/vmware/diskadd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log 

vCenter Operations Manager UI (VCOPS):

/var/log/vmware/admin.log
/var/log/vmware/ciq-firstboot.log
/var/log/vmware/ciq.log
/var/log/vmware/diskadd.log
/var/log/vmware/lastupdate.log
/var/log/vmware/mod_jk.log
/var/log/vmware/vcops-admin.cmd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log
/var/log/vmware/diskadd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log 

vCenter Orchestrator (VCO):

/opt/vmo/app-server/server/vmo/log/boot.log
/opt/vmo/app-server/server/vmo/log/console.log
/opt/vmo/app-server/server/vmo/log/server.log
/opt/vmo/app-server/server/vmo/log/script-logs.log
/opt/vmo/configuration/jetty/logs/jetty.log 

vCenter Server Appliance (VCSA):

/var/log/vmware/vpx/vpxd.log
/var/log/vmware/vpx/vpxd-alert.log
/var/log/vmware/vpx/vws.log
/var/log/vmware/vpx/vmware-vpxd.log
/var/log/vmware/vpx/inventoryservice/ds.log 

vCloud Connector Node (VCC):

/opt/vmware/hcagent/logs/hca.log 

vCloud Connector Server (VCC):

/opt/vmware/hcserver/logs/hcs.log 

vSphere Data Protection (VDP):

/space/avamar/var/log/av_boot.rb.log
/space/avamar/var/log/dpnctl.log
/space/avamar/var/log/dpnnetutil-av_boot.log
/usr/local/avamar/var/log/dpnctl.log
/usr/local/avamar/var/log/av_boot.rb.log
/usr/local/avamar/var/log/av_boot.rb.err.log
/usr/local/avamar/var/log/dpnnetutil-av_boot.log
/usr/local/avamar/var/avi/server_log/flush.log
/usr/local/avamar/var/avi/server_log/avinstaller.log.0
/usr/local/avamar/var/vdr/server_logs/vdr-server.log
/usr/local/avamar/var/vdr/server_logs/vdr-configure.log
/usr/local/avamar/var/flr/server_logs/flr-server.log
/data01/cur/err.log
/usr/local/avamarclient/bin/logs/VmMgr.log
/usr/local/avamarclient/bin/logs/MountMgr.log
/usr/local/avamarclient/bin/logs/VmwareFlrWs.log
/usr/local/avamarclient/bin/logs/VmwareFlr.log 

vCloud Director (VCD):

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log
/opt/vmware/vcloud-director/logs/vcloud-container-info.log
/opt/vmware/vcloud-director/logs/jmx.log 

vSphere Infrastructure Navigator (VIN):

/var/log/vadm/system.log
/var/log/vadm/engine.log
/var/log/vadm/activecollector.log
/var/log/vadm/dbconfig.log
/var/log/vadm/db/postgresql.log 

vSphere Management Assistance (VMA):

/var/log/vmware/vma/vifpd.log 

vSphere Replication (VR):

/var/log/vmware/hbrsrv.log 

Categories // Uncategorized Tags // syslog, vC Log, vCenter Log Insight, vcloud suite

How To Add A Tag (Log prefix) To Syslog Entries

05.07.2013 by William Lam // 2 Comments

Last year I wrote an article on how to forward vCenter Server logs to a remote syslog server using the built in syslog-ng client in the VCSA. A few weeks back, I received an interesting email from Michael White sharing details about adding a "tag" or more specifically, adding a string prefix to each syslog entry being forwarded. This was interesting as it enables a user to easily search for a specific log entry based on a "tag" and comes really in handy when you have multiple log sources being forwarded from the same host. An example of this would be the various logs from a vCenter Server such as vpxd, vws, inventoryservice, etc. which all have their own individual logs coming from the same host.

Within the Syslog-ng client configuration, you can specify the log_prefix() option and the string you wish to prefix a given log source. The tag has a specific syntax that must contain a : (colon) and a whitespace after the string (e.g. "VC_APP: ").

Using the vCenter Server as example, we could add the following tags:
After restarting the syslog-ng client for the changes to going into effect, you can head over to your syslog server to view the updated syslog entries. In the screenshot below, we can see we have log sources from both our VC_APP (vpxd.log) and VC_IS (ds.log) entries as specified in our syslog-ng client configurations.

Note: For newer versions of syslog-ng, program_override() is used instead of log_prefix(). The syntax for that would be program_override("VC_APP").

I want to thank Michael for sharing this cool tidbit!

Categories // Uncategorized Tags // syslog, tag, VCSA

Forwarding vCenter Server Logs to a Syslog Server

08.01.2012 by William Lam // 24 Comments

I was recently asked if it was possible to forward vCenter Server logs to a regular syslog server and if so, how difficult would it be to setup? I had researched this topic several years back, but did not find an ideal solution as vCenter Server was only available on the Windows platform and vCenter Server itself did not provide any syslogging capabilities. With the release of vSphere 5.0, VMware introduced the VCSA (vCenter Server Appliance) and realized I never revisited this question for the VCSA.

After a bit of digging, I found that the VCSA comes installed with syslog-ng by default which is used to provide the vSphere Syslog Collector functionality as well as the local syslog client for the VCSA itself. Given this information, it was pretty trivial to source the local /var/log/vmware/vpx/vpxd.log (symlink to latest vCenter Server log as well as other important vCenter logs) and automatically forward that to a remote syslog server.

VCSA Syslog Configuration

You will need to edit the following configuration file on the VCSA - /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

# vpxd source log
source vpxd {
       file("/var/log/vmware/vpx/vpxd.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vpxd-alert.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vws.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/vmware-vpxd.log" follow_freq(1) flags(no-parse));
       file("/var/log/vmware/vpx/inventoryservice/ds.log" follow_freq(1) flags(no-parse));
};

# Remote Syslog Host
destination remote_syslog {
       udp("172.30.0.45" port (514));
};

# Log vCenter Server vpxd log remotely
log {
        source(vpxd);
        destination(remote_syslog);
};

Note: If you are interested in more details about "sourcing" a local log, take a look at this article here which I used as a reference.

Once you have saved the configuration file, you just need to restart the syslog client by running the following command:

service syslog restart

If you login to your remote syslog server, you should now see that your VCSA is forwarding it's vpxd logs over. Pretty simple, right? 🙂 You can of course forward over other vCenter Server logs by adding additional source files. The main key is that there is a symlink that automatically points to the latest log file which you map as the source file.

I am sure many of you are probably asking what about vCenter Server for Windows? Well, I did also looked into a similar solution but it's a bit more complex than just adding a few configuration entries.

Windows vCenter Server Syslog Configuration

Disclaimer: This is not supported by VMware, please use at your own risk.

There are a few challenges with the Windows version, by default there are no syslog clients installed and there is no automatic symlink to the latest vCenter Server log. Having said that, you can still get the above solution working using the free syslog-ng, but it takes a few more steps. The solution will be leveraging Cygwin, so we can run the free version of syslog-ng on a Windows system.

Step 1 - Install Cygwin and configure syslog-ng service on your vCenter Server as described in this article. You will need to add an additional package which is "Admin/Cron" that will be used in the subsequent steps. In the example, I ran syslog-ng under default system account, but if you need to run it under a different user, you may find these two articles to be helpful

  • http://linux.subogero.com/894/cron-on-cygwin/
  • http://www.davidjnice.com/articles/cygwin_cron-service.html

Step 2 - Just as before, we will need to edit /etc/syslog-ng/syslog-ng.conf and add the following lines at the bottom of the file (remember to replace the syslog host with your own):

# vpxd source log
source vpxd {
       file("/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs/vpxd.log" follow_freq(1) flags(no-parse));
};

# Remote Syslog Host
destination log_additional_remote_syslog {
       udp("172.30.0.45" port (514));
};

# Log vCenter Server vpxd log remotely
log {
        source(vpxd);
        destination(log_additional_remote_syslog);
}; 

You will notice this time, we are accessing the Windows C drive by using the /cygdrive path

Step 3 - As mentioned earlier, there is no symlink that points to the latest vCenter Server log, which makes it difficult to map to static log file. What we can do is basically identify the latest vpxd-#.log and automatically create a symlink and that is what is being monitored by syslog-ng to forward the log. We will be using a cronjob and a very simple shell script.

You can place the script in the current home directory /home/Administrator (or whatever default user you happen to have installed Cygwin on)

Here is the shell script which I have called latest.sh:

#!/bin/bash

VC_LOG_PATH="/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs"
LATEST=$(ls -tr "/cygdrive/c/ProgramData/VMware/VMware VirtualCenter/Logs/" | grep "vpxd-[0-9]*.log" | grep -v ".gz" | tail -1)

if [ ! -e "${VC_LOG_PATH}/vpxd.log" ]; then
        touch "${VC_LOG_PATH}/vpxd.log"
fi

ln -sf "${VC_LOG_PATH}/${LATEST}" "${VC_LOG_PATH}/vpxd.log"

Make sure to set the script to be executable: chmod +x latest.sh

Step 4 - Create a cronjob which will run every minute (you might be able to set a longer delay depending on your environment and it's rotation frequency) by editing the following file /var/cron/tabs/Administrator or using crontab -e
Step 5 - Start or restart syslog-ng by running one of the following commands:

Start - cygrunsrv -S syslog-ng
Restart - cygrunsrv -E syslog-ng;cygrunsrv -S syslog-ng

If everything was successful, you should start seeing your vCenter Server logs from your Windows system forward to your remote syslog server. When the latest vpxd-#.log changes, the cronjob will automatically take care of re-linking to the latest vpxd-#.log to ensure you continue forwarding your vCenter Server logs.

As you can see, it is not trivial to set this up for the Windows vCenter Server as it is for the VCSA, but you now have a way to centrally store all your important vCenter Server logs for archival or analysis purposes without having to manually copy them off to a remote volume.

Few additional notes:

  • I believe the paid version of syslog-ng supports file globbing, so you do not need to setup a cronjob and just watch for all vpxd-*.log, but in this example, I went with a completely free solution
  • You might also be able to leverage Splunk to monitor vCenter Server logs as noted in this Splunkbase entry, but I have not verified and I am not sure if you have to pay for this feature in Splunk
  • Here is an easier way of forwarding vCenter Server logs on Windows using Snare by Raphael Schitz.

Categories // Uncategorized Tags // syslog, VCSA, vcva

  • « Previous Page
  • 1
  • …
  • 4
  • 5
  • 6
  • 7
  • 8
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025