WilliamLam.com

VCSA alarm for VCDB space utilization in vSphere 6.5

11.10.2016 by William Lam // 4 Comments

With prior releases of the vCenter Server Appliance (VCSA), there was little to no visibility to the underlying vCenter Server Database (VCDB) which uses an embedded vPostgres Database. This was especially true for being able to get basic storage utilization of the VCDB including the breakdown of the different data types being stored. More importantly, there was no easy way to even monitor the storage utilization of the VCDB to help prevent the rare case where the VCDB could be filling up for whatever reason.

In vSphere 6.5, there have been huge amount of improvements to provide customers with greater visibility into the VCDB. Not only can customers get granular into the specific types of data being consumed: Stats, Events, Alarm & Tasks (SEAT), Transaction Log & VC Inventory within the VCDB, but this information can also be easily accessed both from a UI as well as API (using the VAMI REST API) standpoint. The Virtual Appliance Management Interface, better known as the VAMI for the VCSA has received a huge face lift in vSphere 6.5. As you can see from the screenshot below, there is now a Database section which gives you the current utilization of your VCDB. In addition, you can also see how this utilization trends over time for the various data types.

From a reporting and visibility standpoint, this is great but how do you go about operationalizing this data and ensuring that you do not run into situation where your VCDB is out of space or is close to being out of space? Another improvement that has been made to the VCSA 6.5 is that there is now a default vCenter Server Database Health alarm that will monitor the space utilization of your VCDB.

The way in this work is that system will check the VCDB space utilization every 15minutes with the following trigger events defined:

If the current storage utilization is at 80%, a Warning alarm will be triggered
If the current storage utilization is 95%, an Error alarm will be triggered and the action is to shutdown the vCenter Server application to protect the database

These default triggers can be changed by simply editing the following vCenter Server advanced settings: vpxd.vdb.space.errorPercent and vpxd.vdb.space.warningPercent (restart of VC service is not required).

Customers can also extend these alarms to send an additional email and/or SNMP trap to their monitoring system so that not only is this visible in the vSphere Web Client but the appropriate administrators can also be notified. The above is just one of the many improvements the VCSA 6.5 has received and I definitely recommend customers spend some time looking at what is now available in the VAMI UI as well as being able to pull this information using our new VAMI REST API.

vCenter Server Database retention purge schedule

11.08.2016 by William Lam // 5 Comments

The size of your vCenter Server Database is largely based on the amount events/tasks and performance statistics that you retain for your vSphere environment. You can view and edit these settings by going to the vCenter Server "General" settings as shown in the screenshot below (documentation here and here):

A common misconception when changing any one of these retention policies, especially when decreasing the amount of data to be retained, is that the existing data would be purged immediately to comply with the new settings. This is actually not the case and for data that is applicable for removal, there are a set of purge jobs that run on a specific schedule to perform the clean up. Below is the schedule in which these database jobs run for each of the data types:

Performance Statistics:

Daily Level - Once every 30 minutes starting at 00:00 (e.g. 00:00, 00:30, 01:00, etc.)
Weekly Level - Once every 2 hours starting at 01:45 (e.g. 01:45, 03:45, 05:45, etc. )
Monthly & Yearly Level - Once a day at 02:15

Events and Tasks:

Once a day at 00:15

For customers that are looking for immediate results and reclaim storage from within their VCDB, you can take a look at the following VMware KB 1025914 which outlines the specific instructions. This can especially be useful if you are looking to perform a Windows vCenter Server to vCenter Server Appliance Migration and wish to reduce the overall amount of data that is being copied over from your existing environment.

An update on how to retrieve useful information from a vSphere login?

11.07.2016 by William Lam // 4 Comments

There was an internal Socialcast question today in which the answer could be found in my how to identify the origin of a vSphere login article. After responding to the question, I had realized that I wrote that article almost 6 years ago and what is even more crazy is that it is still very applicable today. The article explains how you can identify a vSphere login by enabling the "trivial" logging option in vCenter Server (extremely verbose, so please use with caution). Once enabled, you can go through the vpxd.log file and find things about a user login such as the the IP Address of the client as well as the type of vSphere interface they had used to login to whether that is using the vSphere C# Client or PowerCLI for example. Although this extracted information can be very useful, the process to retrieve this is not very ideal, especially having to increase your vCenter Server logging verbosity to the extreme which can force other more critical log events to roll over.

Given that this article written back when vSphere 4.1 was still the current release, I figure I should give the process another look to see if there was a better method in retrieving this information. While quickly browsing around the SessionManager object and specifically the UserSession property, I noticed there have been quite a few enhancements that were introduced in vSphere 5.1. It looks like you can now easily retrieve things like the User Agent, IP Address of the client as well as the number of API invocations for anyone who is currently logged into a given vSphere environment. Perhaps someone internally saw my blog post and thought it would be useful to add these properties directly into the vSphere API rather than poking around in the verbose logs 😀

To exercise these new vSphere APIs, I have create a quick PowerCLI function called Get-vSphereLogins The script will iterate through all currently logged in vSphere sessions and provide the following output: Username, IP Address, API Count & Login Time. It also excludes the current session initiating the query as well as any of the VC Extension logins. Here is a screenshot of my environment using several different vSphere API interfaces to login to my vSphere environment:

With the information above, not only can you tell who is logging in but also where (IP Address) and most importantly how (User Agent) they are logging in. One thing to be aware of is that the User Agent is not always populated and even if it is, it may not provide you with enough information on the specific interface a given user is logging in from. For example, it looks like a script written using the vSphere SDK for Python does not actually set the User Agent, so it is empty.

Here is an updated table using some of the latest vSphere interfaces to log into a vSphere 6.0 Update 2 environment and their respective observed User Agents:

Interface	User Agent
vSphere C# Client	VMware vSphere Client/6.0.0
vSphere Web Client	VMware vim-java 1.0
vSphere MOB	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML like Gecko) Chrome/54.0.2840.71 Safari/537.36
PowerCLI	PowerCLI/6.5.0
vSphere SDK for Perl	VI Perl
vSphere SDK for Ruby (rbvmomi)	Ruby
vSphere SDK for Python (pyvmomi)	None

Note: In vSphere 6.5, the User Agent that is returned for the vSphere Web Client session looks to be using web-client/6.5.0

Finally, saving the best for last. The VMware Engineer(s) not only added these new properties into the vSphere API, but they have also made them readily available using the vSphere Web Client. To view all the session information, navigate to your vCenter Server instance and under Manage->Sessions you can get the exact same view as using the vSphere API. By default, the IP Address, User Agent & API Invocations are hidden by default. You just need to right click on the table header and add those additional field as shown in the screenshot below.

Longer term, it would be great to see that each of the "official" VMware CLI/SDKs as well as other interfaces can uniquely identify themselves with a well defined string. This not only helps with understanding the types of tools customers are using but also helps with any types of internal audits customers may require. If you think this would be useful to have, please feel free to leave a comment or any other things you feel would be useful to include.