SEAT

VCSA alarm for VCDB space utilization in vSphere 6.5

11.10.2016 by William Lam // 4 Comments

With prior releases of the vCenter Server Appliance (VCSA), there was little to no visibility to the underlying vCenter Server Database (VCDB) which uses an embedded vPostgres Database. This was especially true for being able to get basic storage utilization of the VCDB including the breakdown of the different data types being stored. More importantly, there was no easy way to even monitor the storage utilization of the VCDB to help prevent the rare case where the VCDB could be filling up for whatever reason.

In vSphere 6.5, there have been huge amount of improvements to provide customers with greater visibility into the VCDB. Not only can customers get granular into the specific types of data being consumed: Stats, Events, Alarm & Tasks (SEAT), Transaction Log & VC Inventory within the VCDB, but this information can also be easily accessed both from a UI as well as API (using the VAMI REST API) standpoint. The Virtual Appliance Management Interface, better known as the VAMI for the VCSA has received a huge face lift in vSphere 6.5. As you can see from the screenshot below, there is now a Database section which gives you the current utilization of your VCDB. In addition, you can also see how this utilization trends over time for the various data types.

From a reporting and visibility standpoint, this is great but how do you go about operationalizing this data and ensuring that you do not run into situation where your VCDB is out of space or is close to being out of space? Another improvement that has been made to the VCSA 6.5 is that there is now a default vCenter Server Database Health alarm that will monitor the space utilization of your VCDB.

The way in this work is that system will check the VCDB space utilization every 15minutes with the following trigger events defined:

If the current storage utilization is at 80%, a Warning alarm will be triggered
If the current storage utilization is 95%, an Error alarm will be triggered and the action is to shutdown the vCenter Server application to protect the database

These default triggers can be changed by simply editing the following vCenter Server advanced settings: vpxd.vdb.space.errorPercent and vpxd.vdb.space.warningPercent (restart of VC service is not required).

Customers can also extend these alarms to send an additional email and/or SNMP trap to their monitoring system so that not only is this visible in the vSphere Web Client but the appropriate administrators can also be notified. The above is just one of the many improvements the VCSA 6.5 has received and I definitely recommend customers spend some time looking at what is now available in the VAMI UI as well as being able to pull this information using our new VAMI REST API.

vCenter Server Database retention purge schedule

11.08.2016 by William Lam // 5 Comments

The size of your vCenter Server Database is largely based on the amount events/tasks and performance statistics that you retain for your vSphere environment. You can view and edit these settings by going to the vCenter Server "General" settings as shown in the screenshot below (documentation here and here):

A common misconception when changing any one of these retention policies, especially when decreasing the amount of data to be retained, is that the existing data would be purged immediately to comply with the new settings. This is actually not the case and for data that is applicable for removal, there are a set of purge jobs that run on a specific schedule to perform the clean up. Below is the schedule in which these database jobs run for each of the data types:

Performance Statistics:

Daily Level - Once every 30 minutes starting at 00:00 (e.g. 00:00, 00:30, 01:00, etc.)
Weekly Level - Once every 2 hours starting at 01:45 (e.g. 01:45, 03:45, 05:45, etc. )
Monthly & Yearly Level - Once a day at 02:15

Events and Tasks:

Once a day at 00:15

For customers that are looking for immediate results and reclaim storage from within their VCDB, you can take a look at the following VMware KB 1025914 which outlines the specific instructions. This can especially be useful if you are looking to perform a Windows vCenter Server to vCenter Server Appliance Migration and wish to reduce the overall amount of data that is being copied over from your existing environment.

Multiple VMDKs in VCSA 6.0?

03.09.2015 by William Lam // 10 Comments

One thing you might notice after deploying the new VCSA 6.0 is that it now includes 11 VMDKs. If you are like me, you are probably asking why are there so many? If you look at past releases of the VCSA, it only contained two VMDKS. The first disk was used for both the OS and the various VMware applications like vCenter Server, vSphere Web Client, etc. and the second disk was where all the application data was stored such as the VCDB, SSODB, Logs, etc.

There were several challenges with this design, one issue was that you could not easily increase the disk capacity for a particular application component. If you needed more storage for the VCDB but not for your logs or other applications, you had no choice but to increase the entire volume. In fact, this was actually a pretty painful process because a logical volume manager (LVM) was also not used. This meant that you needed to stop the vCenter Server service, add a new disk, format it and then copy all the data from the old volume to the new. Another problem with the old design is that you can not apply Storage QoS on important data such as the VCDB which you may want on a faster tier of storage or putting your Log data on slower and cheaper tier of storage by leveraging something like VM Storage Policies which works on a per VMDK basis.

For these reasons, VCSA 6.0 is now comprised of 11 individual VMDKs as seen in the screenshot below.

Here is a useful table that I have created which provides the mappings of each of the VDMKs to their respective functions.

Disk	Size	Purpose	Mount Point
VMDK1	12GB	/ and Boot	/ and /boot
VMDK2	1.2GB	Temp Mount	/tmp/mount
VMDK3	25GB	Swap	SWAP
VMDK4	25GB	Core	/storage/core
VMDK5	10GB	Log	/storage/log
VMDK6	10GB	DB	/storage/db
VMDK7	5GB	DBLog	/storage/dblog
VMDK8	10GB	SEAT (Stats Events and Tasks)	/storage/seat
VMDK9	1GB	NetDumper	/storage/netdump
VMDK10	10GB	AutoDeploy	/storage/autodeploy
VMDK11	5GB	Inventory Service	/storage/invsvc

In addition, increasing disk capacity for a particular VMDK has been greatly simplified as the VCSA 6.0 now uses LVM to manage each of the partitions. You can now, on the fly increase disk space for a particular volume while the vCenter Server is still running and the changes will go live immediately. You can refer to this article here for the process as it is a simple two step process.

Here are some useful commands to get more details of the filesystem structure in the new VCSA.

lsblk

lsscsi