VCSA 6.5 – Page 9

How to enable vCenter Server High Availability (VCHA) in vSphere 6.5 w/single ESXi host?

11.16.2016 by William Lam // 5 Comments

One of the big new features that was introduced in vSphere 6.5, exclusively for vCenter Server Appliance (VCSA), is the vCenter Server High Availability (VCHA) capablilty. Feidhlim O'Leary has an excllent blog post covering what VCHA provides as well as a couple of demo videos on how it works, definitely worth checking out! After upgrading one of my home lab enviornments to vSphere 6.5, I wanted to try out this feature from an educational standpoint and specifically around using new VCHA vSphere APIs.

Like most vSphere Home Labbers, I have limited hardware and if you try to enable VCHA with only a single ESXi host, you will see the following error:

This operation would violate a virtual machine affinity/anti-affinity rule.

As you might expect, VCHA will automatically provision affinity rules to ensure that the active, passive and witness node are not all running on the same physical ESXi host. For a production deployment this is completely valid but for lab and testing purposes, this might be a tough requirement to satisfy. I was hoping there might be an override option and searching for the word "ha" in the vCenter Server Advanced Settings lead me to an interesting property called config.vpxd.vcha.drsAntiAffinity. This discovery was purely by luck and I had noticed it was set to true by default, so I decided to change it to false and see what would happen.

To my surprise, changing this setting worked and I was able to successfully enable VCHA in my lab with all three nodes just running on a single ESXi host 😀

An alternative solution would be to deploy a 3-Node Nested ESXi cluster which would not require this modification, but my physical ESXi host was limited on memory, only 16GB and would have been a lot slower.

VCSA alarm for VCDB space utilization in vSphere 6.5

11.10.2016 by William Lam // 4 Comments

With prior releases of the vCenter Server Appliance (VCSA), there was little to no visibility to the underlying vCenter Server Database (VCDB) which uses an embedded vPostgres Database. This was especially true for being able to get basic storage utilization of the VCDB including the breakdown of the different data types being stored. More importantly, there was no easy way to even monitor the storage utilization of the VCDB to help prevent the rare case where the VCDB could be filling up for whatever reason.

In vSphere 6.5, there have been huge amount of improvements to provide customers with greater visibility into the VCDB. Not only can customers get granular into the specific types of data being consumed: Stats, Events, Alarm & Tasks (SEAT), Transaction Log & VC Inventory within the VCDB, but this information can also be easily accessed both from a UI as well as API (using the VAMI REST API) standpoint. The Virtual Appliance Management Interface, better known as the VAMI for the VCSA has received a huge face lift in vSphere 6.5. As you can see from the screenshot below, there is now a Database section which gives you the current utilization of your VCDB. In addition, you can also see how this utilization trends over time for the various data types.

From a reporting and visibility standpoint, this is great but how do you go about operationalizing this data and ensuring that you do not run into situation where your VCDB is out of space or is close to being out of space? Another improvement that has been made to the VCSA 6.5 is that there is now a default vCenter Server Database Health alarm that will monitor the space utilization of your VCDB.

The way in this work is that system will check the VCDB space utilization every 15minutes with the following trigger events defined:

If the current storage utilization is at 80%, a Warning alarm will be triggered
If the current storage utilization is 95%, an Error alarm will be triggered and the action is to shutdown the vCenter Server application to protect the database

These default triggers can be changed by simply editing the following vCenter Server advanced settings: vpxd.vdb.space.errorPercent and vpxd.vdb.space.warningPercent (restart of VC service is not required).

Customers can also extend these alarms to send an additional email and/or SNMP trap to their monitoring system so that not only is this visible in the vSphere Web Client but the appropriate administrators can also be notified. The above is just one of the many improvements the VCSA 6.5 has received and I definitely recommend customers spend some time looking at what is now available in the VAMI UI as well as being able to pull this information using our new VAMI REST API.

Updates to VMDK partitions & disk resizing in VCSA 6.5

11.07.2016 by William Lam // 9 Comments

Similiar to the vCenter Server Appliance (VCSA) 6.0 release, the new VCSA 6.5 is also composed of multiple virtual machine disks (VMDKs). Each VMDK maps to a specific function and OS partition within the VCSA. There are now a total of 12 VMDKs, two of which are new in vSphere 6.5: vSphere Update Manager (VUM) and Image Builder. The following table provides a break down of the VMDKs in VCSA 6.5 compared to VCSA 6.0:

Disk	6.0 Size	6.5 Size	Purpose	Mount Point
VMDK1	12GB	12GB	/ and Boot	/ and Boot
VMDK2	1.2GB	1.8GB	VCSA's RPM packages	N/A as it is not mounted after install
VMDK3	25GB	25GB	Swap	SWAP
VMDK4	25GB	25GB	Core	/storage/core
VMDK5	10GB	10GB	Log	/storage/log
VMDK6	10GB	10GB	DB	/storage/db
VMDK7	5GB	15GB	DBLog	/storage/dblog
VMDK8	10GB	10GB	SEAT (Stats Events and Tasks)	/storage/seat
VMDK9	1GB	1GB	Net Dumper	/storage/netdump
VMDK10	10GB	10GB	Auto Deploy	/storage/autodeploy
VMDK11	N/A (Previously InvSrvc 5GB)	10GB	Image Builder	/storage/imagebuilder
VMDK12	N/A	100GB	Update Manager	/storage/updatemgr

In addition to the VMDK/partition changes, there are a couple of enhancements when needing to increase disk capacity in the VCSA. Just like in VCSA 6.0, you will still be able to hot-extend any one of the VMDKs while the system is still running.

The first change is that instead of the old vpxd_servicecfg command which is used expand the logical volume(s) making the new storage capacity available the OS/application, it has been replaced with the following command: /usr/lib/applmgmt/support/scripts/autogrow.sh
The second change is that instead of having to perform the above command using only SSH which may be disabled by default. There is now a new Virtual Appliance Management Interface (VAMI) REST API that can be called remotely: POST /appliance/system/storage/resize
The final difference is that in previous releases, you could only resize the Embedded VCSA or External VCSA node, but not the Platform Services Controller (PSC) node. In 6.5, this has changed and you can apply this method on any one of the VCSA nodes. Thanks to Blair for reminding me on this one!

Lets walk through an example of increasing the Net Dumper partition (VMDK9) and exercising this new VAMI API.

Step 1 - Login to VCSA using SSH to run a quick "df -h" to check the current size of your Net Dumper partition which by default will be 1GB as seen in the screenshot below.

Step 2 - Next, we will increase the VMDK to 5GB. In this example, I am using the vSphere Web Client but if you wanted to completely automate this process end-to-end, you can use the vSphere API/PowerCLI to perform this operation.

Step 3 - To quickly try out the new VAMI API, we will use the new vSphere API Explorer that is included in the VSCA 6.5. Simply open a web browser and enter the following URL: https://[VCSA-HOSTNAME]/apiexplorer Select the "appliance" API and then click on the login button and enter your vCenter Server credentials.

Step 4 - Scroll down to the POST /appliance/system/storage/resize operation and expand it. To call this API, just click on the "Try it out" button. If the operation completely successfully, you should see a 200 response as shown in the screenshot below.

Step 3 and 4 can also be called directly through PowerCLI using the new CIS cmdlets (Connect-CisServer & Get-CisService) which exposes the new VAMI APIs. Below is a quick snippet that performs the exact same operation:

Connect-CisServer -Server 192.168.1.150 -User *protected email* -Password VMware1!
$diskResize = Get-CisService -Name 'com.vmware.appliance.system.storage'
$diskResize.resize()

Step 5 - Lastly, we can now log back into the VCSA and re-run the "df -h" command to verify we can see the new storage capacity.