WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple

A preview of native syslog support in VCSA 6.0

03.30.2015 by William Lam // 29 Comments

Proper logging of VMware hosts, services and application logs are becoming more and more critical these days and their usage goes beyond just troubleshooting. In many of our customer environments, extended log retention is often mandatory to satisfy auditing and compliance requirements. Support for remote syslog has been around in ESXi for quite some time and has included several enhancements over the years, however logging for vCenter Server itself has not changed much over the years. Historically, vCenter Server started out as a Windows application and outside of standard filesystem logging there is also Microsoft Event Logs which was not really all that useful. With the release of the vCenter Server Appliance (VCSA), syslog support became more attainable, at least without additional 3rd party tools.

I can even remember when I was an administrator, I had to get creative on how to forward vCenter Server logs to a remote syslog server which I had blogged about back in 2012. Though the solution works, it was not ideal especially when you are running several dozen to several hundred vCenter Server instances like many of our customers do today. When I had discovered that there was a Common Logging initiative within VMware for vSphere 6.0, I was pretty excited and I can only guess that this also put a big smile on many of our GSS folks faces 😉

As you can imagine this was no small undertaking, especially with the organic growth of services and applications within vCenter Server. The goal was not only to support native remote syslog but to also standardize on the location, rotation, retention of all the logs and most importantly providing a consistent time stamp of events so that an administrator or 3rd party tool can easily correlate operations across multiple VMware log files. Though complete native syslog support in vCenter Server is not 100% ready just yet, much of the plumbing and foundation has already been finished and in fact you can see some of this in the latest release VCSA 6.0.

With VCSA 6.0, there is partial support for native remote syslog which is configurable through the VMware Syslog Service under the new vCenter Server System Configuration found within the vSphere Web Client.

vcenter_server_6_syslog_1
There are four settings that you will need to configure:

  • Common Log Level - * (everything), info, notice, warn, error, crit, alert & emerg
  • Host - Hostname/IP Address of a *single* remote syslog server
  • Port - Port of the remote syslog server (514 for UDP & 1514 for TCP is already opened on the VCSA firewall)
  • Protocol - Supports tcp, udp & tls

A restart is not required when configuring the syslog service and logs will automatically be forwarded to the remote syslog server which is quite nice. You can also view the health status of the syslog service and its connectivity to the remote syslog server by clicking onto the "Summary" view as seen in the screenshot below. For more information about the new syslog service, check out the official documentation here.

vcenter_server_6_syslog_2
So what exactly does partial syslog support really mean? What logs are being forwarded to a syslog server when the syslog service is enabled?

There are currently two major sets of logs that are forwarded to a remote syslog server when the new syslog service is configured:

  1. All logs from ESXi hosts that are connected to the vCenter Server will be forwarded
  2. A partial set of vCenter Server services (details in table below) will be forwarded
Service Name Service Description Service Log Location
applmgmt-audit Appliance Management /var/log/vmware/applmgmt/applmgmt-audit/applmgmt-audit-syslog.log
audispd Audit Event Dispatcher /var/log/audit/audispd/audispd-syslog.log
auditd Audit System /var/log/audit/auditd/auditd-syslog.log
rbd Auto Deploy /var/log/vmware/rbd/rbd-syslog.log
vmafdd VMware Authentication Framework /var/log/vmware/vmafdd/vmafdd-syslog.log
vmcad VMware Certificate Service /var/log/vmware/vmcad/vmcad-syslog.log
vmdird VMware Directory Service /var/log/vmware/vmdird/vmdird-syslog.log
watchdog-rhttpproxy Watchdog for Reverse HTTP Proxy service /var/log/vmware/rhttpproxy/watchdog-rhttpproxy/watchdog-rhttpproxy-syslog.log
watchdog-syslog Watchdog for Syslog service /var/log/vmware/syslog/watchdog-syslog/watchdog-syslog-syslog.log
watchdog-vmware-vpostgres Watchdog for vPostgres DB service /var/log/vmware/vpostgres/watchdog-vmware-vpostgres/watchdog-vmware-vpostgres-syslog.log
watchdog-vpxd Watchdog for vCenter Server service /var/log/vmware/vpxd/watchdog-vpxd/watchdog-vpxd-syslog.log
watchdog-vws Watchdog for vCenter Web Services service /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log

Note: The information above was extracted from /etc/vmware-syslog/custom-file-location.conf

Here is a screenshot of my vRealize Log Insight instance ingesting the logs that have been forwarded over from my VCSA 6.0:

vcenter_server_6_syslog_7
Although not all the vCenter Server services have been integrated into this new native syslog mechanism, you can see where things headed and hopefully in the not too distant future we will have full native syslog support for all application and system logs found withint vCenter Server. One thing that I really do like is that I can go to one single location to configure my remote syslog server and automatically receive all logs from the ESXi hosts being managed by that vCenter Server and forwarded to the configured syslog server. This definitely makes it operationally friendly so that you have one less thing to configure when provisioning new ESXi hosts.

One limitation that I found when configuring your remove syslog server is that there is no way to reset the values to NULL and the UI also limits the number of remote syslog server to just one, even though you can specify multiple targets. One way to get around this UI limitation is by editing the underlying configuration file which is located in /etc/vmware-syslog/syslog.conf

Here is an example of what the syslog.conf looks like for the above configuration:

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you wish to add a second or even third syslog server, you simply just need to duplicate the existing line and update the hostname or IP Address of your syslog server.

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
*.info @log2.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you are manually editing the syslog.conf, you will need to restart the syslog service by running the following command for the changes to take effect:

/etc/init.d/vmware-syslog restart

Some of you might say this is great and all, but one of the most important log files which is the vCenter Server log (vpxd.log) is not being being forwarded. How useful is this really to me? I know I definitely asked that question 🙂 Though not ideal, there is a small configuration change you can apply to easily get vpxd.log to also forward to a remote syslog server using the new syslog service.

You will need to change the vCenter Server advanced setting "config.log.outputToSyslog" property (can also be done using vSphere API) from false to true as seen in the screenshot below.

vcenter_server_6_syslog_3
The above assumes you have already configured the syslog service and for this change to go into effect, you will need to restart the vCenter Server service. This can be done using the System Configuration and under the vCenter Server Service, by just right clicking and selecting "Restart".

vcenter_server_6_syslog_4
If we now look at our vRealize Log Insight instance or whatever syslog server you are using, you should now see entries from the vpx.log being forwarded:

vcenter_server_6_syslog_6
You can also perform this change from the command-line by editing the vCenter Server configuration file at /etc/vmware-vpx/vpxd.cfg and modifying <outputToSyslog>true</outputToSyslog>

vcenter_server_6_syslog_5
Once you have saved the changes, you will need to restart the vCenter Server by running the following command:

/etc/init.d/vmware-vpxd restart

For those of you who are considering vSphere 6.0 and using the VCSA, this is something I definitely recommend checking out to help simplify the management of both your logs for vCenter Server and your ESXi hosts. I know the VMware Engineering team is working hard on making native syslog support even easier in the future and I look forward to the complete solution hopefully in the near future.

Categories // ESXi, vSphere 6.0 Tags // ESXi 6.0, syslog, vCenter Log Insight, vCenter Server, vcenter server appliance, vcsa, vcva, vmsyslog, vpx.cfg, vpxd.log, vSphere 6.0

Automate forwarding of vRealize Operations Manager logs to Syslog Server

01.06.2015 by William Lam // 3 Comments

Forwarding logs to a syslog server from vRealize Operations Manager has been greatly simplified with the latest 6.0 release. Instead of having to manually tweak syslog-ng.conf with past releases, there is now a new UI option under Administration->Support->Logs which allows you to easily configure syslog configurations. There are currently 4 loggers (Analytics, Collector, Web & SuiteAPI) that can be configured to be forwarded to a syslog server.

forward-vrealize-operations-manager-to-syslog-0
To configure using the vROps UI, you select the root logger folder and then click on the gears icon at the top. From here, you can configure the syslog server as well as the syslog port.

forward-vrealize-operations-manager-to-syslog-1
Though this is a nice UI enhancement, it would have been really nice to have been able to configure this using an API. In doing some digging, I found that you could still automate the syslog configurations by simply adding a couple of lines into the respective log4j.properties files:

vROps Logger Configuration File
Analytics /usr/lib/vmware-vcops/user/conf/analytics/log4j.properties
Collector /usr/lib/vmware-vcops/user/conf/collector/log4j.properties
Web /usr/lib/vmware-vcops/user/conf/web/log4j.properties
SuiteAPI /usr/lib/vmware-vcops/tomcat-enterprise/webapps/suite-api/WEB-INF/log4j.properties

Here is a simple shell script called configurevROpsSyslog.sh that you can run against your vROps instance to forward your logs. There are two variables SYSLOG_SERVER and SYSLOG_SERVER_PORT that will need to be updated before running the script. Here is an example of running the script that can come in handy to forwarding your logs to your favorite syslog server such as vRealize Operations Insight.

forward-vrealize-operations-manager-to-syslog-2
UPDATE (01/17/15) - Thanks to Steve Flanders for letting me know there is one additional location for configuring syslog which is the Audit logs located in under Administration->Audit. It looks like the configuration also goes into the Collector log4j.properties and I have updated my script to ensure this is also configured as part of the script.

Screen Shot 2015-01-17 at 9.43.35 PM

Categories // Automation, vRealize Suite Tags // log4j.properties, syslog, vcops, vRealize Operations Manager, vROps

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 2

07.10.2013 by William Lam // 7 Comments

In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.

To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:

[vmsyslog-logger]
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp

Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:

Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.

Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:

esxcli system syslog reload

You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:

If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.

One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.

Categories // Uncategorized Tags // syslog, vC Log, vCenter Log Insight, vmsyslog, vmware.log, vmx, vSphere 5.1

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • Next Page »

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC) across Private, Hybrid and Public Cloud

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Will this Arm SoC work with ESXi-Arm? 06/02/2023
  • Converting VirtualBox VDI (Virtual Disk Image) to VMDK for use with ESXi 8.x 05/31/2023
  • Quick Tip - How to monitor when ESXi filesystem and partitions are filling up? 05/30/2023
  • DDR5 SODIMM capable kits for ESXi 05/30/2023
  • ESXi on ASUS PN64-E1 05/24/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...