WilliamLam.com

New SDDC Certificate Replacement Fling

by // 11 Comments

Certificate lifecycle management is not something anyone looks forward to, it is time consuming and usually not automated. However, it is a necessity for many of our customers. The process gets even more challenging when needing replace certificates across multiple VMware products, not only careful orchestration but also properly reestablishing trust between product just adds another layer of operational complexity. Within the Integrated System Business Unit (ISBU) at VMware, which produces both the VMware Validated Design (VVD) and VMware Cloud Foundation (VCF), the team has been working on a way to simplify certificate management, not only for individual products (working with product teams) but also holistically at the VMware SDDC level.

This initially started with the development of a tool called Certificate Generation Utility (CertGen), which helps customers generate new certificates for various products within the VMware SDDC. Although it was developed for the VVD, any VMware customer who consumed products within the VVD, could also leverage this tool. We all know certificate generation can be a pain, but it is not as challenging or as complex as the actual certificate replacement process itself which is also fully documented by the VVD team here.

This is where the new Fling comes in, the SDDC Certificate Tool, which automates the manual steps outlined by the VVD and helps customers easily replace certificates that they have created (CertGen or another process) and automatically orchestrates this across the different products within the SDDC. The tool is command-line driven and uses a JSON configuration file which can contain all or a subset of the VMware SDDC products, which is great for supporting different environments and allows for easy source control. Extensive pre-checks are also built into the tool to validate the certificates themselves (e.g. expiry, chain validation, etc) also also preventing miss-match of information (e.g. SAN entries, number of nodes, etc) which then get compared against your actual environment before any changes are applied. The JSON also contains a section referred to as Service Accounts, which is merely other VMware product accounts that the tool supports to reestablish trust after replacing the certificate for given product. 

[Read more...]

Quick Tip - List all open ports on the VCSA / PSC

by // 2 Comments

The list of required ports for both a vCenter Server Appliance (VCSA) and Platform Services Controller (PSC) are pretty well documented here (6.5), here (6.0) and here (5.5) for customers who require this information to setup external connectivity within their networking infrastructure. Having said that, it is may not always be clear on what ports are actually opened as they will usually depend on the type of deployment and the services that are running. Instead, some customers have inquired about getting a list of all open ports directly from the VCSA/PSC to ensure they have the actual configuration which can be used to build firewall rules and/or for auditing purposes.

Today, the only method is to login directly into the VCSA/PSC via SSH (you could also use GuestOps API, so that SSH is NOT required) and fetching this information using iptables. Hopefully, in the future, this can be made available as part of the VAMI API since it already covers some basic inbound firewall rule capabilities. In the mean time, below are examples of how to get all the open ports for both VCSA/PSC

Run the following command to view all open ports on VCSA/PSC:

iptables -L port_filter -n --line-numbers


You will notice in the output above, there is also a chain number on the far left side which is associated with each rule. This chain number can be used to inspect the rule further and some rules include a nice alias to help you identify what the port might be used for.

For example, we can run the following to inspect chain rule #30 and find out this port is being used for syslog. If we want the port number, we simply add the -n option.

iptables -L port_filter 30
iptables -L port_filter 30 -n


Not all of the firewall rules have an alias name and even if they do, it still may not be apparent on what service is opening that particular port. We can actually look at the firewall rule definitions which are located under /etc/vmware/appliance/firewall and you will see a JSON file for each of the VCSA/PSC services that require firewall rules to be opened up. For a given port, you can just grep in this directory to identify the service that is requiring the port.

For example, if we take a look at the vmware-syslog, we see that it requires tcp/udp 514 and tcp 1514 under the "rules" array which defines the list of external ports open. You can ignore the internal ports as those are not exposed to the outside world but used by internal services. In case the services are still not clear, you can always reference the port number back to the documentation which I had linked above to get more details about the particular port.

Auditing/Logging vCenter Server authentication & authorization activities

by // 1 Comment

Recently, I have seen an increase in the number of requests from our field and customers inquiring about logging various vCenter Server authentication and authorization activities. The topics vary from identifying which log files contain which activities to to why some of this information is not available in the vCenter Server Events UI or why they are available else where. In most of these cases, customers were also looking for a way to forward these activities to their remote syslog infrastructure for auditing and tracking purposes whether that is using vRealize Log Insight (which all vSphere customers get 25 free OSI licenses!) or some other logging solution.

Having explored this topic lightly in the past and given the amount of interests, I thought I would dive a bit deeper and look at some of the common authentication and authorization workflows and provide examples of what the log entries look like and where you can find them. However, before jumping right in, I think is is worth spending a few minutes looking at the history of authentication (commonly referred to as AuthN) and authorization (commonly referred to as AuthZ) for vCenter Server and where we had started from and where we are at today to give you the full context.

UPDATE (04/08/19) - Please take a look at this blog post here for all new auditing enhancements in vSphere 6.7 Update 2 which simplifies the consumption of vCenter and vCenter SSO auditing events.

History of vCenter Server AuthN/AuthZ

Prior to vSphere 5.1, vCenter Server handled both Authentication (AuthN) and Authorization (AuthZ). As a Client, you would connect directly to vCenter Server and the AuthN service will verify who you are whether that is a local account on the OS or an Active Directory user which required vCenter Server to be joined to your AD Domain. Once you have been authenticated, the AuthZ service will then take over and verify the privileges you have been assigned to perform specific operations within vCenter Server.


In vSphere 5.1, a new service was introduced called Single Sign-On (SSO) which now takes over for AuthN services from vCenter Server. Once authenticated, it will then allow you to connect to the vCenter Server which then handles AuthZ activities


Although it may not be apparent, one major implication is where are successful and failed authentications being logged? In the past, these would reside within vCenter Server since it handled both AuthN/Authz activities, vCenter Server even included specific authentication Events that can then be seen using the UI and/or API. However, with SSO in the picture, authentication is no longer in vCenter Server but with SSO. This is why when you have a failed login using the vSphere Web Client (Flex/H5) UI it does not show up in vCenter Server and it because the logging is done but within the SSO service (which now resides in the Platform Services Controller for more recent vCenter releases).

[Read more...]