Auditing/Logging vCenter Server authentication & authorization activities

Recently, I have seen an increase in the number of requests from our field and customers inquiring about logging various vCenter Server authentication and authorization activities. The topics vary from identifying which log files contain which activities to to why some of this information is not available in the vCenter Server Events UI or why they are available else where. In most of these cases, customers were also looking for a way to forward these activities to their remote syslog infrastructure for auditing and tracking purposes whether that is using vRealize Log Insight (which all vSphere customers get 25 free OSI licenses!) or some other logging solution.

Having explored this topic lightly in the past and given the amount of interests, I thought I would dive a bit deeper and look at some of the common authentication and authorization workflows and provide examples of what the log entries look like and where you can find them. However, before jumping right in, I think is is worth spending a few minutes looking at the history of authentication (commonly referred to as AuthN) and authorization (commonly referred to as AuthZ) for vCenter Server and where we had started from and where we are at today to give you the full context.

UPDATE (04/08/19) - Please take a look at this blog post here for all new auditing enhancements in vSphere 6.7 Update 2 which simplifies the consumption of vCenter and vCenter SSO auditing events.

History of vCenter Server AuthN/AuthZ

Prior to vSphere 5.1, vCenter Server handled both Authentication (AuthN) and Authorization (AuthZ). As a Client, you would connect directly to vCenter Server and the AuthN service will verify who you are whether that is a local account on the OS or an Active Directory user which required vCenter Server to be joined to your AD Domain. Once you have been authenticated, the AuthZ service will then take over and verify the privileges you have been assigned to perform specific operations within vCenter Server.

In vSphere 5.1, a new service was introduced called Single Sign-On (SSO) which now takes over for AuthN services from vCenter Server. Once authenticated, it will then allow you to connect to the vCenter Server which then handles AuthZ activities

Although it may not be apparent, one major implication is where are successful and failed authentications being logged? In the past, these would reside within vCenter Server since it handled both AuthN/Authz activities, vCenter Server even included specific authentication Events that can then be seen using the UI and/or API. However, with SSO in the picture, authentication is no longer in vCenter Server but with SSO. This is why when you have a failed login using the vSphere Web Client (Flex/H5) UI it does not show up in vCenter Server and it because the logging is done but within the SSO service (which now resides in the Platform Services Controller for more recent vCenter releases).

Now that we have some background on where to find AuthN and AuthZ activities, lets now take a closer look at what can be found in the logs. I spent most of Friday going through and documenting the following workflows in detail for both a vSphere 6.0 Update 3 and vSphere 6.5 environment. Both vCenter Servers were joined to Active Directory, so we will be able to see example log entries for both local SSO and AD users across the different operations as well as where to find it in the logs (see the detailed analysis link below the workflows).

AuthN/AuthZ Workflows

Single Sign-On Activities

• Successful SSO Login
• Successful SSO Logout
• Successful SSO Active Directory Login
• Successful SSO Active Directory Logout
• Failed SSO Login
• Failed SSO Login (User not found)
• Failed SSO Active Directory Login
• Failed SSO Active Directory Login (User not found)
• SSO User Creation
• SSO User Password Change
• SSO User Deletion
• SSO Group Creation
• SSO Group Assignment
• SSO Group Deletion
• SSO Password policy update

vCenter Server Activities

• Successful vCenter Server Login
• Successful vCenter Server Logout
• vSphere Permission Created
• vSphere Permission Updated
• vSphere Permission Deleted
• vSphere Role Creation
• vSphere Role Update
• vSphere Role Deletion

The complete analysis of the above workflows can be found in this Github repo: https://github.com/lamw/vcenter-authn-authz-log-examples

Forwarding AuthN/AuthZ activities from vSphere 6.0 & 6.5 to Syslog

For customers interested in capturing and forwarding some of the AuthN/AuthZ activities to their remote syslog infrastructure, you have a couple of options. The first is leveraging vRealize Log Insight (vRLI) which includes an out-of-the-box content pack for vCenter Server which captures some of these activities and there is no additional work required. Obviously, for the activities that are not captured, we will want them to be covered in future updates of vRLI. The second option is to forward the respective logs using the vCenter Server Appliance (VCSA) built in syslog client (Windows does not include this functionality and another reason to migrate to the VCSA) to your syslog infrastructure whether that is vRLI or some other solution.

For the given workflows above, you will be able to capture them by forwarding the following log files:

Additional Information

vCenter Server Events

Although vCenter Server has had Permission and Role Events since its inception, due to the introduction of SSO and the additional services provided by the PSC, these Events were not being triggered within vCenter Server. There have been some changes to how these Events are propagated in vSphere 6.5, so now you will be able to retrieve them using either the UI or API.

Here are six AuthN/AuthZ events that can be queried:

• PermissionAddedEvent
• PermissionUpdatedEvent
• PermissionRemovedEvent
• RoleAddedEvent
• RoleUpdatedEvent
• RoleRemovedEvent

Here is a screenshot of UI when a vSphere Permission being created:

Here is a screenshot of UI when a vSphere Permission is being updated:

Here is a screenshot of UI when a vSphere Permission being deleted:

Here is a screenshot of UI when a vSphere Role being created:

Here is a screenshot of UI when a vSphere Role is being updated:

Here is a screenshot of UI when a vSphere Role being deleted:

More Event details when using vSphere API

As mentioned, you can use the vSphere API to programmatically retrieve vCenter Server Events. Alan Renouf even wrote this nice PowerCLI article that dates back to 2010 which is still applicable today when working with vCenter Events. One thing I had noticed when using the vSphere API to extract the Events is that for some of the events, there is sone additional information provided by the API that is not included in the vSphere UI. For example, the RoleAdddedEvent actually contains the specific privileges that were used for the role creation. However, the UI does not show this information.

Here is a PowerCLI example which exercise the vSphere API to extract the RoleAddedEvent using the following snippet:

Get-VIEvent | Where { $_.GetType().Name -eq "RoleAddedEvent" }

As you can see from the screenshot, we get everything we see in the UI but in addition, we can also see the privileges that were added to the role which might be a useful thing to capture for logging and auditing purposes.

vCenter Server Login & Logout

If you recall earlier, I had mentioned that SSO handles all the AuthN operations and this is why we do not see any failed logins when checking the vCenter Server. Having said that, once a user has been successfully authenticated from SSO/PSC, they will be "logged" into vCenter Server by way of the SAML token provided by SSO to the vCenter Server. Due to this mechanism, vCenter Server logins and logouts are generated. These are exposed as vCenter Server Events respectively UserLoginSessionEvent and UserLogoutSessionEvent.

What is really neat about these two specific Events is that they not only are they available both in the UI and API, it also includes some additional metadata about each login/logout including the client IP Address (so you know where the user had initiated the connection from) as well as the User Agent which could be used to determine the type of client they were using whether thats the UI or API. See this blog post here and here for more details.

Here is a screenshot of a successful login to vCenter Server:

Here is a screenshot of a successful logout to vCenter Server:

Future of vCenter Server logging for AuthN/AuthZ

If you made it this far, I think you would agree that although there have been some nice improvements in vSphere 6.5 in terms of vCenter Server Events and the ability to forward them off to a remote syslog infrastructure, there is still more room for improvement. We want to make sure that all operations are not only logged but easily consumable without requiring users to manually forward specific log files which differ from release to release. We also want to make sure that the information captured includes full details of the what, when and who as you may have seen some of the log entries are a bit "light" on the details and requires looking else where for the information. The good news is that this work is already on-going although I can not talk about the future, I know this is an area VMware is investing in to improve the user experience as well as the quality of the data to make sure it can easily be consumed by external monitoring and logging solutions like vRealize Log Insight, vRealize Operations Manager or other third party systems using industry standards. If you have any feedback or specific feature requests, please leave a comment and I will make sure the product teams are made aware.