Enhanced vCenter Server Audit Event & Logging in vSphere 6.7 Update 2

A couple of years back I had published a detailed analysis on vCenter Server's Authentication (AuthN) and Authorization (AuthZ) from an auditing and logging standpoint. This has been the go to reference for many of our customers and the posts also includes a number of log samples which I have documented in the following Github repository.

In addition to serving as a reference for our customers, it has also helped our Product and Engineering teams understand where we still had some gaps and how we could improve the overall user experience. As hinted in the recently announced vSphere 6.7 Update 2 release, which will be available soon, there are number of new auditing enhancements that have been made to both vCenter Server and the vCenter Single Sign-On (SSO) service that I think customers will really appreciate.

"Real" client IP address in Events

When you look at a login or logout Event in vCenter Server today, you may have noticed the user's client IP Address is actually of the vCenter Server rather than the actual remote client's address and the reason for this is explained here. In vSphere 6.7 Update 2, the real client IP Address is now captured and is included in all successful login/logout and failed logins. This information can now enable administrators to easily identify unauthorized access and be able to quickly track down the systems initiating the connections.

Below is a screenshot of a login/logout event for vSphere 6.7 which shows the IP Address of the vCenter Server and vSphere 6.7 Update 2 system which now shows the actual client IP Address logging into that vCenter Server. This information is accessible using both the vSphere UI and API. For customers who enable syslog on their vCenter Server Appliance (VCSA), these events will automatically be included and forwarded to their remote syslog target.

vCenter Single Sign-On Events

Historically, vCenter Server Events did not include any vCenter SSO details. If customers wanted some of this information, they would have to take additional steps to forward several SSO log files which still did not include everything. These logs were meant for troubleshooting/debugging and was not designed for auditing purposes. In vSphere 6.7 Update 2, all configuration changes or user activity that involves vCenter SSO will now have a vCenter Server Event that represents the specific operation.

The way this works is there is now a dedicated vCenter SSO Audit log at /var/log/audit/sso-events/audit_events.log for VCSA and C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\logs\audit_events.log for Windows VC which will capture all vCenter SSO related activities. Every 5 minutes, these SSO Events will then be published to vCenter Server which will then make them available in both the vSphere UI and API.

Here is a screenshot of the vSphere UI with various vCenter SSO Events that can now easily be viewed without having SSH'ing to the VCSA and manually looking at a log file.

Similarly, customers can use Automation Tools like PowerCLI or any other vSphere SDK to look up specific vCenter SSO Events such as the example below using PowerCLI.

Get-VIEvent | where {$_.EventTypeId -match "com.vmware.sso.*"} | select EventTypeId,FullFormattedMessage

For customers who enable syslog on their vCenter Server Appliance (VCSA), all vCenter SSO events will automatically be included and forwarded to their remote syslog target, as shown in the screenshot below using vRealize Log Insight. This means customers no longer have to manually configure their VCSA to forward specific SSO logs, this is now automatic for you by simply enabling syslog for your VCSA! Another reason to move to the VCSA if you are still running Windows VC.

For your convenience, below are all the different vCenter SSO Event types that can be generated. You can also get this information by simply viewing the audit_events.log which contains the logged event details and type.

Global Permission Events

As you may have guessed from the header, we also now have Global Permission Events in vSphere 6.7 Update 2! When a user creates, updates or removes a Global Permission, a new vCenter Event will be published as shown in the screenshot below. For customers who enable syslog on their vCenter Server Appliance (VCSA), all Global Permissions events will automatically be included and forwarded to their remote syslog target, no additional configuration changes required.

Here are the two Global Permission Event types:

You can also use PowerCLI or any other vSphere SDK to look up these specific events:

Get-VIEvent | where {$_.EventTypeId -match "com.vmware.cis.*"} | select EventTypeId,FullFormattedMessage

vCenter and Tagging Permission Change Events

Lastly, for any type of permission modification on a vCenter Server Inventory Object (Virtual Machine, ESXi host, Network, etc.) including vSphere Tags and Categories, a vCenter Server Event will now be published. These permission events will be available using both the vSphere UI and API. For customers who enable syslog on their vCenter Server Appliance (VCSA), all permission events will automatically be included and forwarded to their remote syslog target, no additional configuration changes required.

Here are both the new and existing vCenter Permission Event types:

Requirements

In addition to upgrading to vSphere 6.7 Update 2 to get the new auditing enhancements, the new vCenter SSO Auditing Events will only be available for an Embedded vCenter Server (Windows or Appliance). For customers running vCenter with an External Platform Services (PSC), you can converge your deployment to an Embedded vCenter by using the new Convergence CLI Tool included in vSphere 6.7 Update 1 or directly in the vSphere UI with the upcoming vSphere 6.7 Update 2 release.