WilliamLam.com

How to recover VCSA 5.5 from an expired administrator account?

by // 9 Comments

Last week I wrote about a new security feature in the new VCSA 5.5 where the administrator account (root) password will now expire automatically after 90 days of powering on the VCSA if the password is not changed before then. This new enhancement is to ensures that administrative passwords are rotated routinely for good security practices. However, in the event that you forget to change the password before the expiration, you can still recover the VCSA and this article will walk you through that process.

As a lab exercise, I have configured my root password to expire in one day and purposely let it expire. If you try to login to the VAMI UI, you will get an "Unable to authenticate user" error and you will see something similar if you login to the SSH console. Ideally, this message should be a bit more descriptive to say something like the password has expired (which I have filed an internal bug for).

Requirements:

  • You will need console access to your VCSA
  • You will also need a Linux LiveCD, I personally like using KNOPPIX

Step 1 - Mount the Linux LiveCD to your VCSA and boot into the image. You will need to bring up a terminal shell. The version I am using has a menu and I just select the "shell" option.

Step 2 - Once you are in the terminal, you will need to switch to the root user by running the following command:

su -

Step 3 - Next, we need to mount the VCSA root partition which will be /dev/sda3 to /mnt directory by running the following command:

mount /dev/sda3 /mnt

Step 4 - We now need to edit /etc/shadow file on our VCSA which is located in /mnt/etc/shadow to disable the account lock. You will need to use an editor such as vi to open up the file.

You need to delete "x" in the 2nd field and the numeric value on the 5th field (if it exists, this should be the number of days for expiration, default is 90) for the root user account. The screenshot above shows what values needs to be deleted. Once you have made the changes, go ahead and save the file.

Step 5 - Reboot the VCSA and now you can login to both the VAMI UI interface as well as the SSH console.

Note: If you had the password expiration feature enabled, it has now been disabled for you to login. If you wish to re-enable it, you will need to configure it in the VAMI UI or through the CLI. Please refer to this article here for more details.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1

by // 17 Comments

Using the new vCenter Log Insight product, you can easily forward application logs from various products within the vCloud Suite for easy analysis and troubleshooting. However, one very important set of logs that we have not been able to collect in the past is the virtual machine logs (vmware.log) which are stored in the working directory of a virtual machine. These logs can be extremely useful from a VMware GSS perspective such as when a virtual machine panics, or if you need to rebuild the .VMX configuration file using these logs or for even general VM auditing purposes.

A recent conversation that I had with Daniel de Sao Jose, who works in our VMware GSS organization reminded of a neat little vSphere 5.1 feature that Daniel had shared with me awhile back. The feature allows you to configure a virtual machine to forward its vmware.log to ESXi's syslog file as well as storing them in the virtual machine's working directory. At the time, there were still a few open questions that required some additional testing and I made a note of this on my ever growing to-do list. I finally around to this and finish up the testing.

UPDATE 1 (04/25/18) - In ESXi 6.7, the ability to forward a VM's vmware.log to an external syslog server also been restored and along with the change, enabling this configuration has been simplified. Instead of having multiple entries to enable the feature and specifying a unique string, you now only have to add a single entry which is vmx.log.syslogID to your VM. The value should be a unique string identifier that the VMX associates with the VM in the syslog. For example, if I use the value of "foo", then the VMX ID will be replaced with "foo" when searching through your syslog entries.

UPDATE 2 (05/04/18) - In ESXi 6.5, 6.5 Update 1 & 6.5 Update 2, the ability to forward a VM's vmware.log to an external syslog server has also been restored and along with the change, enabling this configuration has also been simplified. Simliar to ESXi 6.7, you now only have to add a single entry which is vmx.log.syslogID to your VM. The only difference is that the unique string provided WILL NOT replace the VMX ID in the syslog entry. If you desire the original behavior, you will need to use vSphere 6.7.

To enable this feature, you will need to add the following advanced virtual machine setting:

vmx.log.destination = "syslog-and-disk"

This of course can be enabled using either the vSphere Web Client or vSphere C# Client as well as automated, take a look at this article for more details.

Here is a screenshot showing showing the contents of the vmware.log in the ESXi host's syslog which is located in /var/log/syslog:

Note: The vmware.log is only generated when a virtual machine is powered on.

You also have the option of disabling the local vmware.log from being created in the virtual machine's working directory and only forwarded to ESXi host's syslog. To do so, you would change the advanced virtual machine setting to the following:

vmx.log.destination = "syslog"

By default, the log entries will be identified by the keyword vmx and the specific virtual machine's process ID such as vmx[5313]. However, this is not very user friendly and would still require you to query the VM PID to get the virtual machine name. This can be a challenge if you are viewing the logs from a centralized syslog server such as vCenter Log Insight where you potentially could have logs being forwarded from hundreds if not thousands of ESXi hosts.

To help with this, you can specify the string in which the virtual machine will identify itself when forwarding its logs using the following advanced virtual machine setting:

vmx.log.syslogID = SOME STRING

It made the most sense to me to set this to the name of the virtual machine, so you can easily identify the source of the logs. Here is a screenshot showing the name of the virtual machine instead of the generic "vmx" string.

If you have configured your ESXi host to forward its logs to vCenter Log Insight, you can see how easy it is to view individual virtual machine logs with a click of a button isolating on the syslog source.

One caveat that I would like to mention with this solution is that you are now storing all virtual machine logs in the ESXi hosts syslog file which is also logging other things about the ESXi host. This would cause the local logs to rotate much more frequently on the ESXi host due to the verbosity when powering on and off a virtual machine. This may not be an issue if you are forwarding to a remote syslog server, but ideally it would be nice to have separate log file primarily for the virtual machine logs. In Part 2 of this article, we will take a look at how we can accomplish this by extending ESXi's logger component.

Monitoring vCenter SSO User Account Expiration

by // 2 Comments

Did you know that user accounts created in the vCenter SSO Server automatically expire by default after 365 days? If you do not update your password prior to the expiration date, in about a years time you could potentially be locked out of your vCenter SSO Server which also applies to the admin@system-domain account.

You can change the default password expiration policy by logging into the vSphere Web Client with an SSO Administrator account. Under the configuration section of "Sign-On and Discovery", there is a Password Policies tab that allows you can modify password policies. By default, this is set to 365 days. I would also recommend that after you have installed and setup your vCenter SSO Server, you add at least one user or group from your directory service such as Active Directory and assign it to the SSO Administrator group. This will ensure that you can still log in to the SSO configuration in the event the local SSO user accounts are locked out.

Even though you can change the password expiration policy, there is still no automated notification or alerting built-in for user accounts that are going to expire. The best you can do is to create a calendar event to remind you update your passwords prior to the expiration date. I am sure that many of you are anxious to add another color event to your already busy schedule 🙂

While investigating alternative options a few weeks back, the only method that I have found to retrieve the status for each SSO user is to directly connect to the vCenter SSO Database. There are two specific tables of interest, one which provides the current password policy and the other providing the last password changed date for each SSO user:

  • ims_authn_password_policy
  • ims_principal_data

Disclaimer: This "may" not be officially supported by VMware.

Instead of having you manually dig around in the SSO Database, I have created a Perl script called getSSOUserExpiration.pl which can connect to either a MSSQL or vPostgress backend SSO database. The script which will automatically list out the current password policy as well as user accounts that will be expiring in N days, where N is input provided by the user. You also have the ability to configure the script to automatically email you the results which is nice for a daily or weekly report and can be setup using a cronjob or a scheduled task. There are several configuration variables that will need to be adjusted based on your environment and these are all located within the script itself. For more details on how to setup and use the script, please take a look at the Setup and Configuration section below.

Note: To reduce any negative impact to the vCenter SSO Database, you should add or ask your DBA's to create a limited read-only account and limit access to the following tables above. You may even be able to have your DBA's create a scheduled routine for the specific queries and have that emailed to you internally.

Here is a screenshot of connecting to a vPostgres backend Database:

Here is a screenshot of connecting to a MSSQL backend database:

Here is a screenshot of what the email report looks like:

Note: The email body should contain the specific vCenter SSO Database, but I am not sure why it is not showing up in Gmail, but it does work for other email clients.

Setup and Configuration

vPostgres

To connect to a vPostgres DB, you will need to install the following two perl packages: perl-DBI and perl-DBD-Pg. In this example, I am using the vMA appliance and the zypper package installer. For more details on how to add a SLES repo, please take a look at the following article. I also assume if you are connecting to a vPostgres DB, then you are using the VCSA (vCenter Server Appliance) and by default it does not accept remote connections. We will need to also make two configuration changes to the VCSA for our script to be able to connect to the database.

Step 1 - Run the following two commands to install both perl packages:

sudo zypper in perl-DBI
sudo zypper in perl-DBD-Pg

Step 2 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/pg_hba.conf you will need to add the network in which you will be connecting from:

host    all             all             172.30.0.0/24           trust

Step 3 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/postgresql.conf you will need to add the IP Address(s) that you want vPostgres to listen for remote connection. If you use "*", it will allow all addressees:

listen_addresses = '*'

Step 4 - For the changes to go into effect, you will need to restart the vPostgres DB by running the following command:

service vmware-vpostgres restart

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB. If you need help in identifying the vCenter SSO DB credentials, please refer to this article for the details.

MSSQL

To connect to an MSSQL DB, there are a few additional steps and packages that will be required. We will be using FreeTDS which provides libraries to connect to an MSSQL DB for UNIX/Linux platforms. There was a bit of trial and error in getting the MSSQL solution working and I would like to thank Reuben Stump for his assistance. The following article was used as a reference for the setup below.

Step 1 - Run the following two commands to install the required packages:

sudo zypper in perl-DBI
sudo zypper in gcc

Step 2 - Download and extract the contents of the FreeTDS package:

wget ftp://ftp.astron.com/pub/freetds/stable/freetds-stable.tgz
tar -zxvf freetds-stable.tgz
cd freetds-0.91

Step 3 - Compile and install FreeTDS under /usr/local/freetds:

export SYBASE=/usr/local/freetds/
./configure --prefix=/usr/local/freetds
make
sudo make install

Step 4 - Add your vCenter SSO Server details into the FreeTDS configuration file located in /usr/local/freetds/etc/freetds.conf

[sso]
host = 172.30.0.239
port = 1433
tds version = 7.0

In the example above, I named my database entry "sso" but you can use any name and this will be referenced when editing the script in step 5.

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB.