Proper logging of VMware hosts, services and application logs are becoming more and more critical these days and their usage goes beyond just troubleshooting. In many of our customer environments, extended log retention is often mandatory to satisfy auditing and compliance requirements. Support for remote syslog has been around in ESXi for quite some time and has included several enhancements over the years, however logging for vCenter Server itself has not changed much over the years. Historically, vCenter Server started out as a Windows application and outside of standard filesystem logging there is also Microsoft Event Logs which was not really all that useful. With the release of the vCenter Server Appliance (VCSA), syslog support became more attainable, at least without additional 3rd party tools.
I can even remember when I was an administrator, I had to get creative on how to forward vCenter Server logs to a remote syslog server which I had blogged about back in 2012. Though the solution works, it was not ideal especially when you are running several dozen to several hundred vCenter Server instances like many of our customers do today. When I had discovered that there was a Common Logging initiative within VMware for vSphere 6.0, I was pretty excited and I can only guess that this also put a big smile on many of our GSS folks faces 😉
As you can imagine this was no small undertaking, especially with the organic growth of services and applications within vCenter Server. The goal was not only to support native remote syslog but to also standardize on the location, rotation, retention of all the logs and most importantly providing a consistent time stamp of events so that an administrator or 3rd party tool can easily correlate operations across multiple VMware log files. Though complete native syslog support in vCenter Server is not 100% ready just yet, much of the plumbing and foundation has already been finished and in fact you can see some of this in the latest release VCSA 6.0.
With VCSA 6.0, there is partial support for native remote syslog which is configurable through the VMware Syslog Service under the new vCenter Server System Configuration found within the vSphere Web Client.
There are four settings that you will need to configure:
- Common Log Level - * (everything), info, notice, warn, error, crit, alert & emerg
- Host - Hostname/IP Address of a *single* remote syslog server
- Port - Port of the remote syslog server (514 for UDP & 1514 for TCP is already opened on the VCSA firewall)
- Protocol - Supports tcp, udp & tls
A restart is not required when configuring the syslog service and logs will automatically be forwarded to the remote syslog server which is quite nice. You can also view the health status of the syslog service and its connectivity to the remote syslog server by clicking onto the "Summary" view as seen in the screenshot below. For more information about the new syslog service, check out the official documentation here.
So what exactly does partial syslog support really mean? What logs are being forwarded to a syslog server when the syslog service is enabled?
There are currently two major sets of logs that are forwarded to a remote syslog server when the new syslog service is configured:
- All logs from ESXi hosts that are connected to the vCenter Server will be forwarded
- A partial set of vCenter Server services (details in table below) will be forwarded
| Service Name | Service Description | Service Log Location |
|---|---|---|
| applmgmt-audit | Appliance Management | /var/log/vmware/applmgmt/applmgmt-audit/applmgmt-audit-syslog.log |
| audispd | Audit Event Dispatcher | /var/log/audit/audispd/audispd-syslog.log |
| auditd | Audit System | /var/log/audit/auditd/auditd-syslog.log |
| rbd | Auto Deploy | /var/log/vmware/rbd/rbd-syslog.log |
| vmafdd | VMware Authentication Framework | /var/log/vmware/vmafdd/vmafdd-syslog.log |
| vmcad | VMware Certificate Service | /var/log/vmware/vmcad/vmcad-syslog.log |
| vmdird | VMware Directory Service | /var/log/vmware/vmdird/vmdird-syslog.log |
| watchdog-rhttpproxy | Watchdog for Reverse HTTP Proxy service | /var/log/vmware/rhttpproxy/watchdog-rhttpproxy/watchdog-rhttpproxy-syslog.log |
| watchdog-syslog | Watchdog for Syslog service | /var/log/vmware/syslog/watchdog-syslog/watchdog-syslog-syslog.log |
| watchdog-vmware-vpostgres | Watchdog for vPostgres DB service | /var/log/vmware/vpostgres/watchdog-vmware-vpostgres/watchdog-vmware-vpostgres-syslog.log |
| watchdog-vpxd | Watchdog for vCenter Server service | /var/log/vmware/vpxd/watchdog-vpxd/watchdog-vpxd-syslog.log |
| watchdog-vws | Watchdog for vCenter Web Services service | /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log |
Note: The information above was extracted from /etc/vmware-syslog/custom-file-location.conf
Here is a screenshot of my vRealize Log Insight instance ingesting the logs that have been forwarded over from my VCSA 6.0:
Although not all the vCenter Server services have been integrated into this new native syslog mechanism, you can see where things headed and hopefully in the not too distant future we will have full native syslog support for all application and system logs found withint vCenter Server. One thing that I really do like is that I can go to one single location to configure my remote syslog server and automatically receive all logs from the ESXi hosts being managed by that vCenter Server and forwarded to the configured syslog server. This definitely makes it operationally friendly so that you have one less thing to configure when provisioning new ESXi hosts.
One limitation that I found when configuring your remove syslog server is that there is no way to reset the values to NULL and the UI also limits the number of remote syslog server to just one, even though you can specify multiple targets. One way to get around this UI limitation is by editing the underlying configuration file which is located in /etc/vmware-syslog/syslog.conf
Here is an example of what the syslog.conf looks like for the above configuration:
*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
If you wish to add a second or even third syslog server, you simply just need to duplicate the existing line and update the hostname or IP Address of your syslog server.
*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
*.info @log2.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
If you are manually editing the syslog.conf, you will need to restart the syslog service by running the following command for the changes to take effect:
/etc/init.d/vmware-syslog restart
Some of you might say this is great and all, but one of the most important log files which is the vCenter Server log (vpxd.log) is not being being forwarded. How useful is this really to me? I know I definitely asked that question 🙂 Though not ideal, there is a small configuration change you can apply to easily get vpxd.log to also forward to a remote syslog server using the new syslog service.
You will need to change the vCenter Server advanced setting "config.log.outputToSyslog" property (can also be done using vSphere API) from false to true as seen in the screenshot below.
The above assumes you have already configured the syslog service and for this change to go into effect, you will need to restart the vCenter Server service. This can be done using the System Configuration and under the vCenter Server Service, by just right clicking and selecting "Restart".
If we now look at our vRealize Log Insight instance or whatever syslog server you are using, you should now see entries from the vpx.log being forwarded:
You can also perform this change from the command-line by editing the vCenter Server configuration file at /etc/vmware-vpx/vpxd.cfg and modifying <outputToSyslog>true</outputToSyslog>
Once you have saved the changes, you will need to restart the vCenter Server by running the following command:
/etc/init.d/vmware-vpxd restart
For those of you who are considering vSphere 6.0 and using the VCSA, this is something I definitely recommend checking out to help simplify the management of both your logs for vCenter Server and your ESXi hosts. I know the VMware Engineering team is working hard on making native syslog support even easier in the future and I look forward to the complete solution hopefully in the near future.
This is looking great. Will this also include the VM's vmware.log file? (Officially)
Are ESXi hosts forwarding logs to vCenter, then to a syslog server, or it's just the configuration done centrally and pushed out to all connected ESXi hosts?
VM's vmware.log is a configuration done on the VM so this would not help with that. Also to be able to forward vmware.log, ESXi host would need to be configured to send to syslog server and afaik, this would not be covered under vCenter Server Syslog Service.
The ESXi logs are sent to VC and streamed to syslog server, there's no configuration needed on the ESXi hosts themselves as this is all done through the VC Agent which lives on each of ESXi hosts.
"The ESXi logs are sent to VC and streamed to syslog server, there’s no configuration needed on the ESXi hosts themselves as this is all done through the VC Agent which lives on each of ESXi hosts"
I have configured syslog forwarding in vCenter Appliance 6.5 to remote syslog server. I have not configured Syslog.global.host on every ESXi.
I can't see that the ESXi logs are sent to VC and streamed to syslog server? I just see logs frm the vCenter itself.
Is the feature changed in 6.5?
How can I disable this feature after testing it?
Just turn the service off in the System Configuration UI 🙂
Stopping the Service is grayed out. The only thing I can do is restarting the service or editing the settings.
Strange. You can also just stop the service via CLI:
/etc/init.d/vmware-syslog stop
chkconfig vmware-syslog off <-makes sure it doesn't start back up If you need to clear the config, refer to the blog post which has details towards the end
How do you specify log retention when forwarding to Log Insight?
Great question and Log Insight expert Steve Flanders just blogged about it here http://sflanders.net/2015/04/30/log-insight-retention-period/
We already configure our ESXi hosts to forward to syslog, I don't want that traffic traversing our WAN. Is there a way to prevent ESXi host logging from being sent OR is that happening now anyway and I'm just unaware of it? Just don't want to add something new.
On our first v6.0 VCSA syslog was sending messages but not tagging with the local facility. The syslog receiver was configured for specific facility numbers so the messages were dropped. After a bunch of fiddling we rebooted the VCSA and that fixed the problem.
Great post. Saved me some time. Thanks for sharing!
One of my customer has configured Syslog for using the UDP, however he complains that he is receiving TCP traffic as well. Any idea. He is using TCPDUMP to monitor.
Anyone know how the Syslog Service is monitored in the VCSA? I have an Arcsight syslog connector using UDP for the protocol, 514 for the port and the syslogs are reaching it and searchable but the Syslog service in the vSphere web client shows critical red state with Health Message stating: "Syslog endpoint servername:514 is unreachable. If I point to another syslog server the Syslog Service health goes green making me think it is just something with Arcsight's syslog connector. I would love to just change to another syslog server but or reqirements make that not a current option. I'm trying to figure out what they use to monitor the syslog server is "reachable".
@Craig: we do have the very same problem “Syslog endpoint servername:514 is unreachable". I created a support call and will come back here with their solution (if they have any).
We now have the answer from VMware:
It works like intended. The Syslog health checks over tcp on the same port you configured to use for syslog service whether the syslog server is there or not even when you are using UDP (which is the standart for syslog) to log to your syslogserver.
"Normal" syslog servers do not listen to tcp port 524 -> it reports unreachable.
You can change the port to something your syslog server listens to in the /etc/vmware-syslog/vmware-syslog-health.properties.
cls.strata.ping.port is the entry where you can change the port...
In my opinon thats something stupid vmware made here. UDP is the world standart for syslog and why do they check tcp 514?
As Feature request they should make a better GUI where they clearify what they do here and where to change the port for strata.ping when using UDP for syslog.
I hope that helps you Craig...
Regards,
Christian
The "cls.strata.ping.port" setting worked by modifying on our external PSC appliances and VMware vCenter Server Appliances and I set them to a known TCP port listening on the server where the ArcSight syslog connector is. It would be nice if the configuration let you set that port as well instead of assuming it will be TCP 514 as you mentioned. Thanks for the response.
Regards,
Craig
Hi,
if we have configured "config.log.outputToSyslog" do we need to enable also "config.alert.log.outputToSyslog" ?
Regards,
Hi,
I followed the instructions given in "VMware Syslog Service "and configured the 3rd party syslog server (i.e Graylog server). I am able to see the log messages from vcentre to the Graylog server.
Do we have the similar instructions to send the log messages from ESXi and VDP servers as well ? [or] the instructions given in "VMware Syslog Service" will also forward the messages from ESXi and VDP as well ? [or] the instructions given in "VMware Syslog Service" is only applicable for vcentre ?
Can someone confirm on this post please ?
This unreachable behaviour (pinging on TCP 514 by default) is still present in Update 2. (VMware vCenter Server Appliance 6.0.0.20000) Build 3634794.
I do not see cls.strata.ping.port here /etc/vmware-syslog/vmware-syslog-health.properties. Assuming you guys added that value and then used a port other than 514. Just to confirm, does the entry look something like this?
$cls.strata.ping.port 80
i really like the centralized location for syslog management, but i will say, the logs that are included at first look seem extremely week. most likely i'll look to use your earlier instructions around directly sourcing the logs we want.
anyone tried editing /etc/vmware-syslog/custom-file-location.conf?
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145181. going to attempt some modes to this file and report back. hopefully we can just add whatever log files we want here. this KB references modifying log rotation.
How many ESXi servers can be supported by this syslog service? In KB 2105801, it says that there are no plans to support more than 30 ESXi servers for VCSA integrated version. Can we still deploy a Windows version individually to support a bigger size of deployment? Trying to find a max supported hosts for Windows syslog server but can not get it.
Hello William, i'm a newbie on vmware. I've the following log /var/log/audit/audit.log that is huge (more thant 3.5 GB). It is safe to delete it? there is not a retention and rotation for this log?
I did not find anything about it.
Great writeup as usual and I love your site. You say above "All logs from ESXi hosts that are connected to the vCenter Server will be forwarded". I must not understand this correctly. Does that mean I no longer have to configure syslog on my hosts and that vCenter will forward them instead? I can't find any other links, KBs or documentation that references anything about this. TIA.
Have same question, I can't see any ESXi log be transfer form vcenter to my syslog, is there somewhere need to configure?