What logs do I get when I enable syslog in VCSA 6.5?

The native remote syslog functionality in the vCenter Server Appliance (VCSA) for vSphere 6.5 introduces several new changes from vSphere 6.0. With some of the questions that I have been receiving on this topic, I figure it would be useful to take a closer look at some of the different behaviors and configuration differences. Hopefully by the end of this article, you will have a better understanding of the syslog capabilities in VCSA 6.5

Remote Syslog Configuration

In VCSA 6.0, to configure the remote syslog configuration, you needed to use the vSphere Web Client. Although this may have felt like a convenience, it also added an unnecessary dependency on both vCenter Single-Sign On (SSO) and the vSphere Web Client UI. In VCSA 6.5, the remote syslog configurations is now part of the VAMI UI (https://[VCSA]:5480) which is an out-of-band interface that can still function if either SSO or vCenter Server is down. Once you have saved your changes, the syslog client will automatically be restarted for the changes to go into effect. If you wish to disable the remote syslog functionality, simply click on the reset button.

Note: If you decide to use port 1514, I have found that you must use the TLS protocol rather than TCP or else it will not work.

Remote Syslog Client

A minor change, but syslog-ng is no longer being used within the VCSA and has been replaced by rsyslog.

vCenter Server Events

I assume most of you are probably familiar with vCenter Server Events. If not, they are simply records of user or system actions on a specific vSphere Inventory object. An example of a VC Event can be reconfiguring a VM or removing an ESXi host from a Cluster. Each VC Event contains quite a bit of information including What was changed, When it was changed and Who made the change.

There are three major changes that were introduced in VCSA 6.5 with respect to vCenter Server Events, some of which have implications on logging.

1. Enhanced VC Events in UI - Historically, when viewing VC Event that contained a configuration change, there was no additional details on what exactly was changed. The classic example is modifying a VM which produces the generic "VM was reconfigured" message. This made auditing changes in a vSphere environment quite difficult as you could not see what was changed. However, I would like to clear up a general miss-conception that although the vCenter Server UI did not provide the specific changes, the underlying vCenter Server Event system does have this information. In fact, I had even written about this topic in how you can retrieve and audit VM reconfigurations and see what exactly changed. So, the enhancement in vSphere 6.5 was to make this information available directly in the UI rather than only requiring the vSphere API to retrieve this information. As you can imagine, a very welcome feature after all these years.
2. Forward VC Events to Syslog - Customers have often asked for an easy way to not only view but to also persist VC Events which can aide in troubleshooting as well as auditing purposes. Today, you either manually parse them out of the vCenter Server (vpxd) logs which is non-trivial or programmatically extract them using the vSphere API which then requires customers to write custom code to "poll" for the VC Events. Neither of these solutions are ideal and you still have the problem of how to best store them for use later on? With VCSA 6.5, you now have the option to forward all VC Events, in a structured manner to a remote syslog server like a vRealize Log Insight or other 3rd party syslog servers. This is a great way to easily archive VC Events beyond the default vCenter Server retention period which also helps reduce the footprint of the VCDB. This also enables new actions to be taken on specific events which can be encoded into a vRealize Log Insight Content Pack for example or to trigger other workflows.
3. VC Events no longer being logged locally - With the new capability of forwarding VC Events to a remote syslog server and having the VC Events already persisted in the VCDB, the need for logging VC Events locally within the vCenter Server log (vpxd.log) has become redundant and is no longer needed. This will also help reduce the size of the vCenter Server logs which can be used to store other more important information.

Note: If you are not on vSphere 6.5 or using the latest VCSA 6.5, you still have the ability to either log or perform additional actions on specific VC Events like VM reconfigurations. Have a look at this blog post here for more details.

As noted in the documentation, VC Event logs will show up in your remote syslog server with the "Event" prefix keyword. Below is a screenshot of my vRealize Log Insight instance and highlighted in green are the VC Events being streamed from the VCSA. The entries highlighted in blue are other logs being forwarded from the VCSA which you can find more information in the next section.

Default Logs Forwarded

In addition to forwarding the VC Events when enabling the remote syslog feature, there are few additional logs (listed below) that you will also be receiving automatically from either your VCSA or PSC system.

• /var/log/messages
• /storage/log/vmware/applmgmt-audit/applmgmt-audit-syslog.log
• /storage/log/vmware/vmon/vmon-syslog.log
• /storage/log/vmware/vmdird/vmdird-syslog.log
• /storage/log/vmware/vmcad/vmcad-syslog.log
• /storage/log/vmware/rbd/rbd-syslog.log
• /storage/log/vmware/vmdnsd/vmdnsd-syslog.log
• /storage/log/vmware/vmafdd/vmafdd-syslog.log
• /storage/log/vmware/rsyslogd/rsyslogd-syslog.log

Forwarding vCenter Server Log (vpxd)

Similiar to VCSA 6.0, you will still be able to forward the vCenter Server log (vpxd.log) to a remote syslog server, which is extremely useful for troubleshooting purposes. By default, forwarding of the vpxd.log is disabled like it was in VCSA 6.0. Below are the steps to enable this in the VCSA 6.5 (which are exactly the same steps as in VCSA 6.0).

Step 1 - Go to the vCenter Server Advanced Setting and update config.log.outputToSyslog to true. You can also do this using the vSphere API for those wanting to automate this process.

Step 2 - For the changes to go into effect, you will need to restart the vCenter Server service. You can do so by navigating to System Configuration->Services->VMware vCenter Server and restart the service. You can also perform this step via Automation by using the new VAMI APIs.

If everything was successfully configured, you should now see entries from vpxd.log being forwarded to your remote syslog server (highlighted in orange) as well as the VC Events (highlighted in green) as seen in the screenshot below.

Logs forwarded by VCSA Deployment Type

Now that we have a better understanding of what logs are and can be forwarded when enabling the remote syslog feature in the VCSA. I figure it would also be useful to provide a breakdown of the specific logs that are forwarded based on each of the VCSA Deployment Types which can be either an Embedded VCSA, External PSC or External VCSA.

vRealize Log Insight vSphere Integration

For completeness sake, I figure I would also cover the vSphere Integration feature provided by vRealize Log Insight (vRLI). Rather than configuring your vCenter Server to "push" logs out to vRLI, you can have vRLI perform a "pull" using the vSphere API. One benefit to this solution is that it can support both a Windows vCenter Server as well as the VCSA, so you can be OS agonistic and have a consistent view of data being retrieved from all of your vCenter Servers.

The vSphere Integration in vRLI provides two specific capabilities:

1. Configuring the ESXi hosts that are within the vCenter Server to automatically enable syslog to vRLI
2. Retrieve only Events, Tasks and Alarm information from vCenter Server. You will not get the other logs mentioned above even if the vCenter Server is a VCSA

As you can see in the vRLI screenshot below, the log entries are quite similiar to the VC Events streamed from the VCSA, they are structured entries parsed from the vSphere API.

I personally think the vSphere Integration is a nice solution if you have both Windows vCenter Server and the VCSA and to be able to get data consistency between the two platforms from a logging standpoint. It is definitely useful if you need to quickly enable all ESXi hosts connected to the vCenter Server and have them remotely syslog to the vRLI instance. If you only have the VCSA, you would get more information by configuring the remote syslog capability in VCSA rather than using the vSphere integration feature of vRLI. This especially true if you need the vpxd.log which is generally required for troubleshooting and debugging vCenter Server issues when calling into VMware Support. The other added benefit to using the VCSA option is that structure log entries are processed directly on the VCSA rather than having to be remotely queried via the vSphere APIs, processed and then store in vRLI which would add additional load onto vRLI, especially if you need to configure additional vCenter Server instances.

Forwarding other logs from VCSA

One limitation that still exists today for the VCSA is that there are many other vCenter Server application logs (Single Sign-On as one example) that is not being captured as part of enabling remote syslogging on the VCSA. This is something that I have brought up internally and I have been told it is still being worked on. Hopefully in the near future, we will have an easy out of the box solution that will allow customers to easily forward all logs generated by either the OS or applications within the VCSA to a remote syslog server. If this capability is important to you and your organization, feel free to leave a comment if you have any additional feedback.

Having said that, I have also received several inquiries about a possible workaround in VCSA 6.5 (even if it may not be officially supported). I know several of you have even referenced my 2012 blog article where I demonstrated how to forward other logs from the VCSA to a remote syslog server. The steps outlined in that article is no longer applicable to the latest VCSA 6.5, but I did have a look at what it would take to get this working in the latest VCSA.

Please see How to forward other VCSA 6.5 logs to remote syslog server? for more details