WilliamLam.com

Auditing/Logging vCenter Server authentication & authorization activities

by // 1 Comment

Recently, I have seen an increase in the number of requests from our field and customers inquiring about logging various vCenter Server authentication and authorization activities. The topics vary from identifying which log files contain which activities to to why some of this information is not available in the vCenter Server Events UI or why they are available else where. In most of these cases, customers were also looking for a way to forward these activities to their remote syslog infrastructure for auditing and tracking purposes whether that is using vRealize Log Insight (which all vSphere customers get 25 free OSI licenses!) or some other logging solution.

Having explored this topic lightly in the past and given the amount of interests, I thought I would dive a bit deeper and look at some of the common authentication and authorization workflows and provide examples of what the log entries look like and where you can find them. However, before jumping right in, I think is is worth spending a few minutes looking at the history of authentication (commonly referred to as AuthN) and authorization (commonly referred to as AuthZ) for vCenter Server and where we had started from and where we are at today to give you the full context.

UPDATE (04/08/19) - Please take a look at this blog post here for all new auditing enhancements in vSphere 6.7 Update 2 which simplifies the consumption of vCenter and vCenter SSO auditing events.

History of vCenter Server AuthN/AuthZ

Prior to vSphere 5.1, vCenter Server handled both Authentication (AuthN) and Authorization (AuthZ). As a Client, you would connect directly to vCenter Server and the AuthN service will verify who you are whether that is a local account on the OS or an Active Directory user which required vCenter Server to be joined to your AD Domain. Once you have been authenticated, the AuthZ service will then take over and verify the privileges you have been assigned to perform specific operations within vCenter Server.


In vSphere 5.1, a new service was introduced called Single Sign-On (SSO) which now takes over for AuthN services from vCenter Server. Once authenticated, it will then allow you to connect to the vCenter Server which then handles AuthZ activities


Although it may not be apparent, one major implication is where are successful and failed authentications being logged? In the past, these would reside within vCenter Server since it handled both AuthN/Authz activities, vCenter Server even included specific authentication Events that can then be seen using the UI and/or API. However, with SSO in the picture, authentication is no longer in vCenter Server but with SSO. This is why when you have a failed login using the vSphere Web Client (Flex/H5) UI it does not show up in vCenter Server and it because the logging is done but within the SSO service (which now resides in the Platform Services Controller for more recent vCenter releases).

[Read more...]

Auditing & Automating Disabled Protocols (TLS/SSLv3) for ESXi 6.0u3 & 6.5 using PowerCLI

by // 32 Comments

A couple of weeks back, I had received a question from one of our TAMs in regards to automating the disablement of specific TLS/SSL protocols for their ESXi 6.0 Update 3 hosts. As of vSphere 6.0 Update 3 and vSphere 6.5, customers now have the ability to completely disable TLS 1.0, TLS 1.1 and SSLv3 using the new TLS Reconfiguration Tool. Mike Foley did a nice write up here if you are interested in more details.

The TLS Reconfiguration Tool works well if you have the same version of vSphere for both your vCenter Server and ESXi host, but has challenges when you are in a mixed environment like this particular customer. In their environment, they are running vCenter Server 6.5 and ESXi 6.5 Update 3 which prevented them from using the TLS Reconfiguration Tool as this is a limitation with the tool today.

UPDATE (05/11/17) - Added support for ESXi 6.5 hosts as well

Given the TLS Reconfiguration Tool was written in Python, I was able to take a closer look at its implementation and I found that the settings that controlled the disabled protocols were just merely a few ESXi Advanced Settings which meant that this could be automated using standard vSphere Automation Tools that our customers were already familiar with. As part of this exercise, I also discovered the tool currently does NOT support disabling TLS/SSLv3 protocols for the Small Footprint CIM Broker (SFCB) service which is also required if you want to be in full compliance for a particular TLS protocol. Although there is not a direct SFCB API that allows you to manage the sfcb.cfg configuration file, there is still a way we can automate this without requiring SSH to the ESXi host which would technically be the alternative. Lastly, I was a bit surprised to see the TLS Reconfiguration Tool did not have a "query" option for listing the current disabled protocols for all ESXi hosts, but they do have it for vCenter Server itself.

To help this particular customer and others who may have specific TLS compliance requirements, I have created the following PowerCLI script called ESXiDisableProtocolConfiguration.ps1 which includes the following two functions:

  • Get-ESXiDPC - Retrieve the current disabled protocols for all ESXi hosts within a vSphere Cluster
  • Set-ESXiDPC - Configure the specific disabled protocols for all ESXi hosts within a vSphere Cluster

[Read more...]

An update on how to retrieve useful information from a vSphere login?

by // 4 Comments

There was an internal Socialcast question today in which the answer could be found in my how to identify the origin of a vSphere login article. After responding to the question, I had realized that I wrote that article almost 6 years ago and what is even more crazy is that it is still very applicable today. The article explains how you can identify a vSphere login by enabling the "trivial" logging option in vCenter Server (extremely verbose, so please use with caution). Once enabled, you can go through the vpxd.log file and find things about a user login such as the the IP Address of the client as well as the type of vSphere interface they had used to login to whether that is using the vSphere C# Client or PowerCLI for example. Although this extracted information can be very useful, the process to retrieve this is not very ideal, especially having to increase your vCenter Server logging verbosity to the extreme which can force other more critical log events to roll over.

Given that this article written back when vSphere 4.1 was still the current release, I figure I should give the process another look to see if there was a better method in retrieving this information. While quickly browsing around the SessionManager object and specifically the UserSession property, I noticed there have been quite a few enhancements that were introduced in vSphere 5.1. It looks like you can now easily retrieve things like the User Agent, IP Address of the client as well as the number of API invocations for anyone who is currently logged into a given vSphere environment. Perhaps someone internally saw my blog post and thought it would be useful to add these properties directly into the vSphere API rather than poking around in the verbose logs 😀

To exercise these new vSphere APIs, I have create a quick PowerCLI function called Get-vSphereLogins The script will iterate through all currently logged in vSphere sessions and provide the following output: Username, IP Address, API Count & Login Time. It also excludes the current session initiating the query as well as any of the VC Extension logins. Here is a screenshot of my environment using several different vSphere API interfaces to login to my vSphere environment:

retreiving-useful-information-about-vsphere-login-0
With the information above, not only can you tell who is logging in but also where (IP Address) and most importantly how (User Agent) they are logging in. One thing to be aware of is that the User Agent is not always populated and even if it is, it may not provide you with enough information on the specific interface a given user is logging in from. For example, it looks like a script written using the vSphere SDK for Python does not actually set the User Agent, so it is empty.

Here is an updated table using some of the latest vSphere interfaces to log into a vSphere 6.0 Update 2 environment and their respective observed User Agents:

Interface User Agent
vSphere C# Client VMware vSphere Client/6.0.0
vSphere Web Client VMware vim-java 1.0
vSphere MOB Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML like Gecko) Chrome/54.0.2840.71 Safari/537.36
PowerCLI PowerCLI/6.5.0
vSphere SDK for Perl VI Perl
vSphere SDK for Ruby (rbvmomi) Ruby
vSphere SDK for Python (pyvmomi) None

Note: In vSphere 6.5, the User Agent that is returned for the vSphere Web Client session looks to be using web-client/6.5.0

Finally, saving the best for last. The VMware Engineer(s) not only added these new properties into the vSphere API, but they have also made them readily available using the vSphere Web Client. To view all the session information, navigate to your vCenter Server instance and under Manage->Sessions you can get the exact same view as using the vSphere API. By default, the IP Address, User Agent & API Invocations are hidden by default. You just need to right click on the table header and add those additional field as shown in the screenshot below.

retreiving-useful-information-about-vsphere-login-1
Longer term, it would be great to see that each of the "official" VMware CLI/SDKs as well as other interfaces can uniquely identify themselves with a well defined string. This not only helps with understanding the types of tools customers are using but also helps with any types of internal audits customers may require. If you think this would be useful to have, please feel free to leave a comment or any other things you feel would be useful to include.