WilliamLam.com

vCenter Server 6.0 Tidbits Part 6: Customizing VCSA's DCUI

by // Leave a Comment

For those of you who have deployed the latest VCSA 6.0, you may have noticed that it now includes a DCUI interface similar to that of ESXi. Just like ESXi, it provides some basic functionality such as network configuration, restarting of the management interface, enabling SSH/Shell access and viewing support logs. For some customers, there is a mandatory requirement in terms of compliance to be able to display a security or warning banner prior to logging into a system which can also include interactive console UI's like the DCUI.

With ESXi, it was possible to customize the DCUI as seen in the screenshot below (more serious customizations are used in customer environments of course). For more details, please take a look at this blog post here.

[Read more...]

How to recover VCSA 5.5 from an expired administrator account?

by // 9 Comments

Last week I wrote about a new security feature in the new VCSA 5.5 where the administrator account (root) password will now expire automatically after 90 days of powering on the VCSA if the password is not changed before then. This new enhancement is to ensures that administrative passwords are rotated routinely for good security practices. However, in the event that you forget to change the password before the expiration, you can still recover the VCSA and this article will walk you through that process.

As a lab exercise, I have configured my root password to expire in one day and purposely let it expire. If you try to login to the VAMI UI, you will get an "Unable to authenticate user" error and you will see something similar if you login to the SSH console. Ideally, this message should be a bit more descriptive to say something like the password has expired (which I have filed an internal bug for).

Requirements:

  • You will need console access to your VCSA
  • You will also need a Linux LiveCD, I personally like using KNOPPIX

Step 1 - Mount the Linux LiveCD to your VCSA and boot into the image. You will need to bring up a terminal shell. The version I am using has a menu and I just select the "shell" option.

Step 2 - Once you are in the terminal, you will need to switch to the root user by running the following command:

su -

Step 3 - Next, we need to mount the VCSA root partition which will be /dev/sda3 to /mnt directory by running the following command:

mount /dev/sda3 /mnt

Step 4 - We now need to edit /etc/shadow file on our VCSA which is located in /mnt/etc/shadow to disable the account lock. You will need to use an editor such as vi to open up the file.

You need to delete "x" in the 2nd field and the numeric value on the 5th field (if it exists, this should be the number of days for expiration, default is 90) for the root user account. The screenshot above shows what values needs to be deleted. Once you have made the changes, go ahead and save the file.

Step 5 - Reboot the VCSA and now you can login to both the VAMI UI interface as well as the SSH console.

Note: If you had the password expiration feature enabled, it has now been disabled for you to login. If you wish to re-enable it, you will need to configure it in the VAMI UI or through the CLI. Please refer to this article here for more details.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1

by // 17 Comments

Using the new vCenter Log Insight product, you can easily forward application logs from various products within the vCloud Suite for easy analysis and troubleshooting. However, one very important set of logs that we have not been able to collect in the past is the virtual machine logs (vmware.log) which are stored in the working directory of a virtual machine. These logs can be extremely useful from a VMware GSS perspective such as when a virtual machine panics, or if you need to rebuild the .VMX configuration file using these logs or for even general VM auditing purposes.

A recent conversation that I had with Daniel de Sao Jose, who works in our VMware GSS organization reminded of a neat little vSphere 5.1 feature that Daniel had shared with me awhile back. The feature allows you to configure a virtual machine to forward its vmware.log to ESXi's syslog file as well as storing them in the virtual machine's working directory. At the time, there were still a few open questions that required some additional testing and I made a note of this on my ever growing to-do list. I finally around to this and finish up the testing.

UPDATE 1 (04/25/18) - In ESXi 6.7, the ability to forward a VM's vmware.log to an external syslog server also been restored and along with the change, enabling this configuration has been simplified. Instead of having multiple entries to enable the feature and specifying a unique string, you now only have to add a single entry which is vmx.log.syslogID to your VM. The value should be a unique string identifier that the VMX associates with the VM in the syslog. For example, if I use the value of "foo", then the VMX ID will be replaced with "foo" when searching through your syslog entries.

UPDATE 2 (05/04/18) - In ESXi 6.5, 6.5 Update 1 & 6.5 Update 2, the ability to forward a VM's vmware.log to an external syslog server has also been restored and along with the change, enabling this configuration has also been simplified. Simliar to ESXi 6.7, you now only have to add a single entry which is vmx.log.syslogID to your VM. The only difference is that the unique string provided WILL NOT replace the VMX ID in the syslog entry. If you desire the original behavior, you will need to use vSphere 6.7.

To enable this feature, you will need to add the following advanced virtual machine setting:

vmx.log.destination = "syslog-and-disk"

This of course can be enabled using either the vSphere Web Client or vSphere C# Client as well as automated, take a look at this article for more details.

Here is a screenshot showing showing the contents of the vmware.log in the ESXi host's syslog which is located in /var/log/syslog:

Note: The vmware.log is only generated when a virtual machine is powered on.

You also have the option of disabling the local vmware.log from being created in the virtual machine's working directory and only forwarded to ESXi host's syslog. To do so, you would change the advanced virtual machine setting to the following:

vmx.log.destination = "syslog"

By default, the log entries will be identified by the keyword vmx and the specific virtual machine's process ID such as vmx[5313]. However, this is not very user friendly and would still require you to query the VM PID to get the virtual machine name. This can be a challenge if you are viewing the logs from a centralized syslog server such as vCenter Log Insight where you potentially could have logs being forwarded from hundreds if not thousands of ESXi hosts.

To help with this, you can specify the string in which the virtual machine will identify itself when forwarding its logs using the following advanced virtual machine setting:

vmx.log.syslogID = SOME STRING

It made the most sense to me to set this to the name of the virtual machine, so you can easily identify the source of the logs. Here is a screenshot showing the name of the virtual machine instead of the generic "vmx" string.

If you have configured your ESXi host to forward its logs to vCenter Log Insight, you can see how easy it is to view individual virtual machine logs with a click of a button isolating on the syslog source.

One caveat that I would like to mention with this solution is that you are now storing all virtual machine logs in the ESXi hosts syslog file which is also logging other things about the ESXi host. This would cause the local logs to rotate much more frequently on the ESXi host due to the verbosity when powering on and off a virtual machine. This may not be an issue if you are forwarding to a remote syslog server, but ideally it would be nice to have separate log file primarily for the virtual machine logs. In Part 2 of this article, we will take a look at how we can accomplish this by extending ESXi's logger component.