vcenter

Customizing vCenter Alarm Email Subject and Body

11.20.2019 by William Lam // 13 Comments

One of the automated actions that can be configured when a vCenter Server Alarm is triggered is to send an email notification. Over the years, I have seen a number of requests and questions about customizing the email and whether an email template exists. I personally have not used this feature much which has been around since the introduction of vCenter Server and mainly because I have always worked in an environment where we had dedicated monitoring tools that provide notifications including emails.

Most recently, I noticed an increase number of questions around this topic and I was curious on whether a solution exists today or if this is still a gap today? A quick Google search landed me on this 2013 VMTN thread which included several workarounds that customers have found. However, the only viable "supported" and "persisted" option at the time within that thread was to use the vSphere API/PowerCLI to customize the alarm action.

While going through this exercise myself, I found that our vSphere UI has had some enhancements since that 2013 thread and I thought it was worth sharing an update in 2019 on how customers can customize both the email subject and body for vCenter Alarms. One thing to note is that there is no generic email template that can be edited, the email customizations are applied on a per-Alarm action basis and this is applicable for both vCenter Server running in a traditional on-premises environment as well as for VMware Cloud on AWS or Dell EMC.

[Read more...]

Quick Tip - Steps to shutdown/startup VSAN Cluster w/vCenter running on VSAN Datastore

07.08.2014 by William Lam // 11 Comments

I know Cormac Hogan already wrote about this topic awhile ago, but there was a question that was recently brought up that included a slight twist which I thought it would be useful to share some additional details. The question that was raised: How do you properly shutdown an entire VSAN Cluster when vCenter Server itself is also running on the VSAN Datastore? One great use case for VSAN in my opinion is a vSphere Management Cluster that would contain all your basic infrastructure VMs including vCenter Server which can be bootstrapped onto a VSAN Datastore. In the event that you need to shutdown the entire VSAN Cluster which may also include your vCenter Server, what is the exact procedure?

To help answer this question, I decided to perform this operation in my own lab which contains a 3-Node (physical) VSAN Cluster that had several VMs running on the VSAN Datastore including the vCenter Server VM that was managing the VSAN Cluster.

Below are the steps that I took to properly shutdown down a VSAN Cluster as well as powering everything back on.

UPDATE (4/27) - Added instructions for shutting down a VSAN 6.0 Cluster when vCenter Server is running on top of VSAN.

Shutdown VSAN Cluster (VSAN 6.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM, that will be the last VM you shutdown.

Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

Step 3 - Ensure that there are no vSAN Components being resync'ed before proceeding to the next step. You can find this information by going to the vSAN Cluster and under Monitor->vSAN->Resyncing Components as shown in the screenshot below.

Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

Step 5 - Next, you will need to place ALL ESXi hosts into Maintenance Mode. However, you must perform this operation through one of the CLI methods that supports setting the VSAN mode when entering Maintenance Mode. You can either do this by logging directly into the ESXi Shell and running ESXCLI locally or you can invoke this operation on a remote system using ESXCLI.

Here is the ESXCLI command that you will need to run and ensure that "No Action" option is selected when entering Maintenance Mode:

esxcli system maintenanceMode set -e true -m noAction

Step 5 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Shutdown VSAN Cluster (VSAN 1.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM.

Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

Step 3 - Place all ESXi hosts into Maintenance Mode except for the ESXi host that is currently running the vCenter Server. Ensure you de-select "Move powered-off and suspend virtual machines to other hosts in the Cluster" as well as selecting the "No Data Migration" option since we do not want any data to be migrated as we are shutting down the entire VSAN Cluster.

Note: Make sure you do not shutdown any of the ESXi hosts during this step because the vCenter Server VSAN Components are distributed across multiple hosts. If you do this, you will be unable to properly shutdown the vCenter Server VM because its VSAN components will not available.

Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

Step 6 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Startup VSAN Cluster

Step 1 - Power on all the ESXi hosts that is part of the VSAN Cluster.

Step 2 - Once all the ESXi hosts have been powered on, you can then login to the ESXi host that contains your vCenter Server. If you took my advice earlier from the shutdown procedure, then you can login to the first ESXi host and power on your vCenter Server VM.

Note: You can perform steps 2-4 using the vSphere C# Client but you can also do this using either the API or simply calling vim-cmd from the ESXi Shell. To use vim-cmd, you need to first search for the vCenter Server VM by running the following command:

vim-cmd vmsvc/getallvms

You will need to make a note of the Vmid and in this example, our vCenter Server has Vmid of 6

Step 3 - To power on the VM, you can run the following command and specify the Vmid:

vim-cmd vmsvc/power.on [VMID]

Step 4 - If you would like to know when the vCenter Server is ready, you can check the status of VMware Tools as that should give you an indication that system is up and running. To do so, you can run the following command and look for the VMware Tools status:

vim-cmd vmsvc/get.guest [VMID]

Step 5 - At this point, you can now login to the vSphere Web Client and take all of your ESXi hosts out of Maintenance Mode and then power on the rest of your VMs.

As you can see the process to shutdown an entire VSAN Cluster even with vCenter Server running on the VSAN Datastore is fairly straight forward. Once you are comfortable with the procedure, you can even automate this entire process using the vSphere API/CLI, so you do not even need a GUI to perform these steps. This might even be a good idea if you are monitoring a UPS and have an automated way of sending remote commands to shutdown your infrastructure.

How to recover VCSA 5.5 from an expired administrator account?

09.10.2013 by William Lam // 9 Comments

Last week I wrote about a new security feature in the new VCSA 5.5 where the administrator account (root) password will now expire automatically after 90 days of powering on the VCSA if the password is not changed before then. This new enhancement is to ensures that administrative passwords are rotated routinely for good security practices. However, in the event that you forget to change the password before the expiration, you can still recover the VCSA and this article will walk you through that process.

As a lab exercise, I have configured my root password to expire in one day and purposely let it expire. If you try to login to the VAMI UI, you will get an "Unable to authenticate user" error and you will see something similar if you login to the SSH console. Ideally, this message should be a bit more descriptive to say something like the password has expired (which I have filed an internal bug for).

Requirements:

You will need console access to your VCSA
You will also need a Linux LiveCD, I personally like using KNOPPIX

Step 1 - Mount the Linux LiveCD to your VCSA and boot into the image. You will need to bring up a terminal shell. The version I am using has a menu and I just select the "shell" option.

Step 2 - Once you are in the terminal, you will need to switch to the root user by running the following command:

su -

Step 3 - Next, we need to mount the VCSA root partition which will be /dev/sda3 to /mnt directory by running the following command:

mount /dev/sda3 /mnt

Step 4 - We now need to edit /etc/shadow file on our VCSA which is located in /mnt/etc/shadow to disable the account lock. You will need to use an editor such as vi to open up the file.

You need to delete "x" in the 2nd field and the numeric value on the 5th field (if it exists, this should be the number of days for expiration, default is 90) for the root user account. The screenshot above shows what values needs to be deleted. Once you have made the changes, go ahead and save the file.

Step 5 - Reboot the VCSA and now you can login to both the VAMI UI interface as well as the SSH console.

Note: If you had the password expiration feature enabled, it has now been disabled for you to login. If you wish to re-enable it, you will need to configure it in the VAMI UI or through the CLI. Please refer to this article here for more details.