Search Results for: vsphere events

VCSA alarm for VCDB space utilization in vSphere 6.5

11.10.2016 by William Lam // 4 Comments

With prior releases of the vCenter Server Appliance (VCSA), there was little to no visibility to the underlying vCenter Server Database (VCDB) which uses an embedded vPostgres Database. This was especially true for being able to get basic storage utilization of the VCDB including the breakdown of the different data types being stored. More importantly, there was no easy way to even monitor the storage utilization of the VCDB to help prevent the rare case where the VCDB could be filling up for whatever reason.

In vSphere 6.5, there have been huge amount of improvements to provide customers with greater visibility into the VCDB. Not only can customers get granular into the specific types of data being consumed: Stats, Events, Alarm & Tasks (SEAT), Transaction Log & VC Inventory within the VCDB, but this information can also be easily accessed both from a UI as well as API (using the VAMI REST API) standpoint. The Virtual Appliance Management Interface, better known as the VAMI for the VCSA has received a huge face lift in vSphere 6.5. As you can see from the screenshot below, there is now a Database section which gives you the current utilization of your VCDB. In addition, you can also see how this utilization trends over time for the various data types.

From a reporting and visibility standpoint, this is great but how do you go about operationalizing this data and ensuring that you do not run into situation where your VCDB is out of space or is close to being out of space? Another improvement that has been made to the VCSA 6.5 is that there is now a default vCenter Server Database Health alarm that will monitor the space utilization of your VCDB.

The way in this work is that system will check the VCDB space utilization every 15minutes with the following trigger events defined:

If the current storage utilization is at 80%, a Warning alarm will be triggered
If the current storage utilization is 95%, an Error alarm will be triggered and the action is to shutdown the vCenter Server application to protect the database

These default triggers can be changed by simply editing the following vCenter Server advanced settings: vpxd.vdb.space.errorPercent and vpxd.vdb.space.warningPercent (restart of VC service is not required).

Customers can also extend these alarms to send an additional email and/or SNMP trap to their monitoring system so that not only is this visible in the vSphere Web Client but the appropriate administrators can also be notified. The above is just one of the many improvements the VCSA 6.5 has received and I definitely recommend customers spend some time looking at what is now available in the VAMI UI as well as being able to pull this information using our new VAMI REST API.

An update on how to retrieve useful information from a vSphere login?

11.07.2016 by William Lam // 4 Comments

There was an internal Socialcast question today in which the answer could be found in my how to identify the origin of a vSphere login article. After responding to the question, I had realized that I wrote that article almost 6 years ago and what is even more crazy is that it is still very applicable today. The article explains how you can identify a vSphere login by enabling the "trivial" logging option in vCenter Server (extremely verbose, so please use with caution). Once enabled, you can go through the vpxd.log file and find things about a user login such as the the IP Address of the client as well as the type of vSphere interface they had used to login to whether that is using the vSphere C# Client or PowerCLI for example. Although this extracted information can be very useful, the process to retrieve this is not very ideal, especially having to increase your vCenter Server logging verbosity to the extreme which can force other more critical log events to roll over.

Given that this article written back when vSphere 4.1 was still the current release, I figure I should give the process another look to see if there was a better method in retrieving this information. While quickly browsing around the SessionManager object and specifically the UserSession property, I noticed there have been quite a few enhancements that were introduced in vSphere 5.1. It looks like you can now easily retrieve things like the User Agent, IP Address of the client as well as the number of API invocations for anyone who is currently logged into a given vSphere environment. Perhaps someone internally saw my blog post and thought it would be useful to add these properties directly into the vSphere API rather than poking around in the verbose logs 😀

To exercise these new vSphere APIs, I have create a quick PowerCLI function called Get-vSphereLogins The script will iterate through all currently logged in vSphere sessions and provide the following output: Username, IP Address, API Count & Login Time. It also excludes the current session initiating the query as well as any of the VC Extension logins. Here is a screenshot of my environment using several different vSphere API interfaces to login to my vSphere environment:

With the information above, not only can you tell who is logging in but also where (IP Address) and most importantly how (User Agent) they are logging in. One thing to be aware of is that the User Agent is not always populated and even if it is, it may not provide you with enough information on the specific interface a given user is logging in from. For example, it looks like a script written using the vSphere SDK for Python does not actually set the User Agent, so it is empty.

Here is an updated table using some of the latest vSphere interfaces to log into a vSphere 6.0 Update 2 environment and their respective observed User Agents:

Interface	User Agent
vSphere C# Client	VMware vSphere Client/6.0.0
vSphere Web Client	VMware vim-java 1.0
vSphere MOB	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML like Gecko) Chrome/54.0.2840.71 Safari/537.36
PowerCLI	PowerCLI/6.5.0
vSphere SDK for Perl	VI Perl
vSphere SDK for Ruby (rbvmomi)	Ruby
vSphere SDK for Python (pyvmomi)	None

Note: In vSphere 6.5, the User Agent that is returned for the vSphere Web Client session looks to be using web-client/6.5.0

Finally, saving the best for last. The VMware Engineer(s) not only added these new properties into the vSphere API, but they have also made them readily available using the vSphere Web Client. To view all the session information, navigate to your vCenter Server instance and under Manage->Sessions you can get the exact same view as using the vSphere API. By default, the IP Address, User Agent & API Invocations are hidden by default. You just need to right click on the table header and add those additional field as shown in the screenshot below.

Longer term, it would be great to see that each of the "official" VMware CLI/SDKs as well as other interfaces can uniquely identify themselves with a well defined string. This not only helps with understanding the types of tools customers are using but also helps with any types of internal audits customers may require. If you think this would be useful to have, please feel free to leave a comment or any other things you feel would be useful to include.

Nested ESXi Enhancements in vSphere 6.5

10.19.2016 by William Lam // 15 Comments

As many of you have probably heard by now, vSphere 6.5 was just announced at VMworld Barcelona this week and it is packed with a ton of new features and capabilities which you can read more about here. One area that is near and dear to me and has not been covered are some of the Nested ESXi enhancements that have been made in vSphere 6.5.

To be clear, VMware has NOT changed its support stance in vSphere 6.5. Both Nested ESXi as well as general Nested Virtualization is still NOT officially supported. Okay, so thats out of the way now, lets see what is new?

Paravirtual SCSI (PVSCSI) support
GuestOS Customization
Pre-vSphere 6.5 enablement on vSphere 6.0 Update 2
Virtual NVMe support

Lets take a closer look at each of these enhancements.

Paravirtual SCSI (PVSCSI) support

When vSphere 6.0 Update 2 was released, I had hinted that PVSCSI support might be a possibility in the near future. I am happy to announce that this is now possible with running Nested ESXi and having the guestOS as ESXi 6.5. In vSphere 6.5, a new GuestOS type has been introduced called vmkernel65 which is optimized for running ESXi 6.5 in a VM as shown in the screenshot below.

As you can see from the VM Virtual Hardware configuration screen below, both the PVSCSI and VMXNET3 adapter are now the recommended default when creating a Nested ESXi VM for running ESXi 6.5.

Similiar to the VMXNET3 driver, the PVSCSI driver is automatically bundled within the version of VMware Tools for Nested ESXi. That is to say, the drivers are included in the default ESXi image itself and is ONLY activated when ESXi detects that it is running inside of a VM. From a user standpoint, this means there are no additional configurations or installations that is required. You simply select ESXi 6.5 as the GuestOS and install ESXi as you normally would and this will automatically be enabled for you.

The only requirement to leverage this new capability is that BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. The underlying physical ESXi host can either be ESXi 6.0 or ESXi 6.5. In addition to new virtual hardware defaults, I have also found that the new ESXi 6.5 GuestOS type now uses EFI firmware over the legacy BIOS compared to previous ESXi 6.x/5.x/4.x GuestOS types.

For customers who wish to push their storage IO a bit more for Nested ESXi guests, this is a great addition, especially with lower overhead when using a PVSCSI adapter.

GuestOS Customization

One of the very last capability that has been missing from Nested ESXi is the ability to perform a simple GuestOS customization when cloning or deploying a VM from template running Nested ESXi. Today, you can deploy my Nested ESXi Virtual Appliance which basically provides you with the ability to customize your deployment but would it not be great if this was native in the platform? Well, I am pleased to say this is now possible!

When you go and clone or deploy a VM from template that is a Nested ESXi VM, you will now have the option to select the Customize guest OS option. As you can see from the screenshot below, you can now create a new Customization Spec which is based on the Linux customization spec. The customization only covers networking configuration (IP Address, Netmask, Gateway and Hostname) and only applies it to the first VMkernel interface, all others will be ignored. The thinking here is that once you have your Nested ESXi VM on the network, you can then fully manage and configure it using the vSphere API rather than re-creating the same functionality just for cloning.

To use this new Nested ESXi GuestOS customization, there are two things you will need to do:

Perform two configuration changes within the Nested ESXi VM which will prepare them for cloning. You can find the configuration changes described in my blog post here
Ensure BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. This means that your underlying physical vSphere infrastructure can be running either vSphere 6.0 Update 2 or vSphere 6.5

You can monitor the progress of the guest customization by going to the VM's Monitor->Tasks & Events using the vSphere Web Client or vSphere API if you are doing this programmatically. Below is a screenshot of a successful Nested ESXi guest customization. If there are any errors, you can take a look at /var/log/vmware-imc/toolsDeployPkg.log within the cloned Nested ESXi VM to determine what went wrong.

I know this will be a very welcome capability for customers who extensively use the guest customization feature or if you just want to quickly clone an existing Nested ESXi VM that you have already configured.

Pre-vSphere 6.5 enablement in vSphere 6.0 Update 2

By now, you probably have figured out what this last enhancement is all about 🙂 It is exactly as it sounds, we are enabling customers to try out ESXi 6.5 by running it as a Nested ESXi VM on your existing vSphere 6.0 environment and specifically the Update 2 release (this includes both vCenter Server as well as ESXi). Although this has always been possible with past releases of vSphere running newer versions, we are now pre-enabling ESXi 6.5 specific Nested ESXi capabilities in the latest release of vSphere 6.0 Update 2. This means when vSphere 6.5 is generally available, you will be able to test drive some of the new Nested ESXi 6.5 capabilities that I had mentioned on your existing vSphere infrastructure. This is pretty darn cool if you ask me!?

Virtual NVMe support

I had a few folks ask on whether the upcoming Virtual NVMe capability in vSphere 6.5 would be possible with Nested ESXi and the answer is yes. Please have a look at this post here for more details.

For those of you who use Nested ESXi, hopefully you will enjoy these new updates! As always, if you have any feedback or requests, feel free to leave them on my blog and I will be sure to share it with the Engineering team and who knows, it might show up in the future just like some of these updates which have been requested from customers in the past 😀