WilliamLam.com

Automating SSL Certificate Regeneration in VCSA 5.1 & 5.5 (vCenter Server Appliance)

by // 4 Comments

The VCSA (vCenter Server Appliance) provides a very simple way of regenerating the self-signed SSL Certificate by using the VAMI web management interface. This is extremely useful if you change the IP Address or hostname of your VCSA and want a proper SSL certificate with the correct common name, especially important if you are plan on using something like vCenter Orchestrator which validates this. To regenerate the SSL Certificate, you just need to login to the VAMI web interface by pointing your browser to the following address: https://[VC-IP]:5480 and under the Admin tab there is a option to "Toggle certificate setting".

After enabling this option, you will need to reboot your VCSA for the new SSL certificate to be generated. Once the VCSA is booted up, you will need to go back into the VAMI interface and disable this setting, else another SSL certificate will be generated upon the next reboot.

I was recently asked if it was possible to automate the SSL regeneration via the command-line without using the GUI which would be very useful for automated VCSA deployments. In looking into this, it turns out the process is quite simple and the present of a file within the VCSA will determine whether a certificate regeneration is required.

To enable certificate regeneration, run the following command which will "touch" (create) allow_regeneration file under /etc/vmware-vpx/ssl directory:

touch /etc/vmware-vpx/ssl/allow_regeneration

To disable certificate regeneration, you just need to remove the file after the VCSA has rebooted. Behind the scenes, this is what is happening when you are toggling the option in the VAMI interface and now you can automate this from the CLI without using the GUI!

UPDATE (09/04/13)

For the new VCSA 5.5, there is a new option that you can specify which will re-generate the SSL certificate and then delete the file without requiring manual intervention after reboot. You would still need to create the /etc/vmware-vpx/ssl/allow_regeneration file but if the contents of the file contains "only-once", it will delete the file automatically which is nice from an Automation perspective.

To re-generate the SSL certificate and automatically have it clean itself up, run the following command:

echo only-once > /etc/vmware-vpx/ssl/allow_regeneration

Extracting SSL Thumbprint from ESXi

by // 13 Comments

While browsing the VMTN forums earlier this week, I noticed an interesting request from a user who was trying to compile an inventory of the SHA1 Thumbprints for all his ESXi hosts. The challenge the user had, was that he was capturing this information manually by "looking" at the DCUI screen which is where the SHA1 Thumbprint for an ESXi host is displayed by default.

As you might have guessed, this can be very tedious and error prone by copying down this very long string by just looking at the screen. Even if you do not make a mistake copying this long string, I bet your eyes will eventually give out. Luckily, there are a few ways to retrieve this information and I will show you some methods to help automate this across all of your ESXi hosts.

UPDATE (05/22/16) - Here's how you can extract SSL Thumbprint using PowerShell

Option 1 - Retrieve SSL Thumbprint using the DCUI as shown above, this is going to be the most manual method.

Option 2 - If you have remote SSH or direct console access to ESXi Shell, you can login to your ESXi host and using openssl utility, you can retrieve the SSL Thumbprint which you can then use or copy off to a remote host.

openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout

Option 3 - You can remotely retrieve the SSL Thumbprint by leveraging just the openssl utility and you do not even need to login to the ESXi host. This not only allows you to retrieve the SSL Thumbprint from a centralized location, but you can easily automate this across all your hosts.

echo -n | openssl s_client -connect 172.30.0.252:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1

Using Option 3, you can easily wrap this in a simple "for" loop to iterate through all your ESXi hosts as long as you have either the hostname/IP Address. Here is a simple shell script that you can use to iterate through all your ESXi hosts to extract the SSL Thumbprint.

In the script above, I have a list of three ESXi hosts and it is simply going through each host and executing the two commands to extract the SSL Thumbprint and displaying it on the screen.

Option 4 - You can also retrieve the SSL Thumbprint using the vSphere API, but the property is only displayed when it is connected to a vCenter Server. There is a property on the ESXi host called sslThumbprint that is populated when querying against the vCenter Server that is managing the ESXi host. You can use the vSphere Health Check script which captures this and other useful information about your vSphere infrastructure.

As you can see, there are several options on obtaining the SSL Thumbprint for an ESXi host, you definitely do not have to manually read it off the DCUI screen. Automation FTW again! 🙂

Automating SSL Certificate Expiry Validation for vCenter Server + ESX(i) Hosts

by // 4 Comments

As many of you know, it is a best practice to replace VMware's self-signed SSL certificates that are included in the vCenter Server (Windows & VCSA) and ESX(i) hosts to prevent or help reduce MiTM (Man in The Middle) attacks. If you are looking for more details on how to replace the default SSL certificates, you should take a look at the fantastic articles written by Michael Webster who details the process, provides some troubleshooting steps and best practices for SSL certificate replacement.

Replacing the default self-signed SSL certificate is just one part of the process, but you also need to check to ensure the certificates are still valid and have not expired. If you already have a process in place or a system that automatically does this for you, that is great. If you do not, you should definitely validate that your SSL certificates are valid on a regular basis.

I recently stumbled onto a nifty open source tool called ssl-cert-check that can help with validating expiration of SSL certificates found on vCenter Server(s) and ESX(i) hosts or any other SSL enabled host for that matter. This utility is just a shell script (specifically bournce shell) wrapping the common openssl utility found on most UNIX/Linux systems and does not require any login credentials to the remote hosts to validate the SSL certificate.

To use the script, you can visit the website here and download it to a system that has openssl installed (in my home lab, I used vMA).

It took me awhile to find the script, but it's located on the right side of the screen where it says "Website". You can also download it from the command-line using wget if you have direct/proxy access to the internet:

wget http://freecode.com/urls/353b752faa208fca12bc0091c742f764 -O ssl-cert-check

Note: Don't forget to set the execute permission on the script (chmod +x ssl-cert-check) else you will get permission denied when trying to run the script.

The script can be executed interactively by specifying the -s option for server and -p for the port. You can also specify the issuer of the certificate by using the -i option. Below is a screenshot of running the ssl-cert-check against a vCenter Server:

You can also run the script in batch mode by specifying -f option which accepts a list of servers in FQDN along with the port number. Using this feature of the script, you can easily run this script against all your vCenter Server(s) and ESX(i) hosts to ensure that their SSL certificates are still valid.

If you already have a list of hosts you want to check, then you can easily create a new file with the hostname and port. Though if you do not have one handy, I wrote a quick vSphere SDK for Perl script called generateESXiHostsList.pl that helps automate the creation of the output file containing all ESX(i) hosts when connecting to a vCenter Server. To use the script, you just need to have vCLI installed on a system or use vMA.

The script accepts one options which is --output which specifies the name of the output file to be created:

If we "cat" the file out, we can see it looks like the following:

vesxi50-1.primp-industries.com 443
vesxi50-2.primp-industries.com 443
vesxi50-3.primp-industries.com 443
vesxi50-4.primp-industries.com 443

Let's now run the ssl-cert-check against the list of ESX(i) hosts using the -f option and see if we have any hosts with expired certificates:

Uh oh, it looks like we have two hosts with some problems. We can see one host that already has an expired SSL certificate and another one that will be expiring in 10 days. We better take a look at these and get them replaced soon!

There are additional options in the ssl-cert-check script including the ability to email the results or run as a nagios check. You can easily schedule a cron job to automate this script to run every week and grepping for the keyword "Expiring" to alert you of any hosts that have expiring SSL certificates. As you can see, it is not only important to replace the default self-signed SSL certificates in your environment, but you need to validate on a routinely basis your your certificates are still valid.