WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Hardware Options
    • Hardware Reviews
    • Lab Deployment Scripts
    • Nested Virtualization
    • Homelab Podcasts
  • VMware Nostalgia
  • Apple

Retrieving ESXTOP Performance Data Using the vSphere 5.1 API

01.15.2013 by William Lam // 5 Comments

In vSphere 5.1, VMware introduced a new managed object called the Service Manager. The Service Manager is a generic object that wraps the execution of a single command and it requires a specific set of inputs to invoke a particular service command. This is particularly interesting as it allows users to access both the ESXTOP and vscsiStatsinterface using the vSphere API. Prior to vSphere 5.1, to use ESXTOP you would need to either login to the ESXi Shell to run the local ESXTOP command or connect remotely using the RESXTOP utility which is only available on a Linux system. For vScsiStats, you would need to login to the ESXi Shell as a remote version of this tool does not exist. The Service Manager used to be a private interface, an interesting tidbit is that some of you may have already interacted with this interface without even realizing it if you have used PowerCLI's Get-Esxtop cmdlet. In this article I will show you how to programmatically access ESXTOP using the vSphere API.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.


Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

Both the ESXTOP and vscsiStats services are only available when connecting directly to an ESXi 5.1 host, it is not available when connecting to a vCenter Server. If we browse over to the vSphere MOB, we can clearly see the two services:

The first step is to get a reference to the ESXTOP service via the Service Manager and to invoke an operation for ESXTOP, you will need to use the ExecuteSimpleCommand. For ESXTOP, there are three valid operations:

  • CounterInfo
  • FetchStats
  • FreeStats

To demonstrate the ESXTOP interface, I have written a sample vSphere SDK for Perl script called getEsxtop.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

CounterInfo

This operation only needs to be invoked once and it will provide you with the list of available counters and their associated properties and data types for a given ESXi host. Here is an example of this using the "getcounters" operation implemented in the script:

Each line represents a specific counter type followed by each property name and their data type. For example, the first line is for the Server counter and has the following properties and types:

Property Name Type
MinFetchIntervalInUsec U64
IsVMVisor B
TimeStampInUsec U64
Time S64

Here is a quick diagram to help you visualize the hierarchy of all the ESXTOP counters and their relationships with one another:

Note: This diagram was created using yuml.me and here is the raw text in JSON format if you are interested.

FetchStats

This operation fetches a single snapshot of ALL the ESXTOP statistics which contains two pieces of information:

  • The topology of the counter instances
  • The actual counter instances values

The first section is denoted by ==NUM-OF-OBJECTS== which contains either inventory data that does not change or counter instance structure which describes the relationship between the different counter instances. Here is an example of the first section using the "getstats" operation implemented in the script:

If we take a look at the second line as an example |PCPU|LCPU,24|Core,12|Package,2| we can see that PCPU counter contains 24 LCPU that you would need to then enumerate as well as inventory information describing the CPU's logical cores and physical socket.

To view the enumerated counter instances and their instance values, we need to look in the second portion of the data which is denoted by ==COUNTER-VALUE== within the output. Here is a screenshot of this section and we can see the enumerated LCPU's (24 in total as denoted earlier) and their associated instance values:

Remember you will need to correlate with the counter definitions that was extracted earlier from the "getcounters" and this will help you build up the data. I do have to say it can be a bit confusing when you first look at the raw data, but as you start to play with it a bit more, it will start to make sense. Two useful references that can help with parsing the data is the ESXTOP bible and an article that Luc Dekens wrote awhile back exploring the Get-Esxtop cmdlet which I mentioned earlier leverages this exact interface.

FreeStats

Lastly, once you are done collecting the ESXTOP data, you will need to run the "freestats" operation and this will release any server side resources used during the collection. When this command is invoked, it will free up all resources even for past collections where you might have forgotten to perform this last step. There is no output from this operation as you can see from the example screenshot below:

Even though it is nice to see the ESXTOP interface be accessible via the vSphere API, it is not the easiest interface to use and is definitely geared more towards a developer. For extracting general performance data, I would still recommend using the Performance Manager managed object or one of the above mentioned command-line interfaces. In the next article, I go into more detail about the vscsiStats interface and how to consume it using the vSphere API.

Categories // Automation Tags // api, ESXi 5.1, esxtop, resxtop, service manager, simplecommand, vSphere 5.1, vsphere sdk for perl

Extracting Information from VIN (vSphere Infrastructure Navigator) Part 1

11.06.2012 by William Lam // 4 Comments

vSphere Infrastructure Navigator or VIN for short is a really useful tool that helps you map out and visualize application services and dependencies between your virtual machines. This is extremely helpful when trying to identify what applications or ports a given virtual machine is running and how it is related or dependent on other virtual machines within your virtual infrastructure.

One thing that I found lacking in VIN today is the ability to export or extract some of this useful information for external use such as a CMDB (Configuration Management Database) as one potential use case. This would allow users to always have an up-to-date CMDB that does not rely on a manual process of adding CI (Configuration Items) and services to your CMDB which can potentially lead to incorrect or more importantly, out of date information (I have seen this happen in the past as an System Admin).

While deploying the latest release of VIN 1.2 in my home lab, I decided to poke around in the VIN appliance and I noticed a web application called Jolokia (not the hot pepper) that was running. After a quick Google, I came across this Jolokia tutorial and with some trial and error, I was able to validate the steps against VIN and was able to access the data VIN was discovering about the virtual machines.

In part 1 of this article, I will show how to get setup and start exploring the data VIN is collecting and in part 2, I will provide a sample script which can be used to automate the extraction of some of this data found in VIN. In the example below, I will be leveraging the vMA appliance, you could use another system that can install the Jolokia client packages, please refer to tutorial for more details.

Disclaimer: This is not officially supported by VMware, use at your own risk.

VIN Appliance Setup:

Before we begin, we need to configure a few things on the VIN appliance which will allow us to connect remotely which is disabled by default.

Step 1 - Edit the following configuration file /opt/vadm-engine/webapps/jolokia/WEB-INF/classes/jolokia-access.xml and specify the remote IP Address you wish to connect to VIN from. By default, VIN is configured to only allow connections from localhost (127.0.0.1).

Step 2 - Run the following script called /opt/vadm-engine/bin/disable_security.sh which disables some of the security configurations such as firewalls, JMX security, etc. which is required to connect to Jolokia interface remotely.

Step 3 - Finally, for the changes to go into effect, you will need to restart the VIN discovery engine by running the following command: /etc/init.d/vadm-engine restart

Step 4 - To verify the configuration above, go ahead and login to the remote host of the IP Address that you added to the jolokia-access.xml file and perform the following curl command: curl http://[VIN_IP_ADDRESS]:8080/jolokia and you should get back a reply which confirms you can communicate with the Jolokia service on your VIN host remotely.

Note: You can also take the URL and paste that into a web browser and it should also respond with the same information. If you do use a browser, make sure you add the IP Address of the desktop that is running the browser to jolokia-access.xml file.

Installing Jmx4Perl on vMA:

 
Step 5 - To explore the data in VIN, we will be installing Jmx4Perl as noted in the Jolokia tutorial which includes an interactive utility called j4psh which will make it easier to view the information in VIN. You will need to install Jmx4Perl via CPAN or manually install the following Perl packages in this order:

  • JSON-2.53
  • Term-Clui-1.66
  • Config-General-2.51
  • Term-ShellUI-0.92
  • Module-Find-0.11
  • jmx4perl-1.06

Luckily, you do not need to manually do this as I wrote a quick shell script called setupJMX4PERL.sh which will download the packages above and install Jmx4Perl for you. All you need to do is just execute the script and then select "yes" when prompted.

Step 6 - Once Jmx4Perl has been installed, you should now have a utility called j4psh which we will use to connect to the Jolokia service running on your VIN appliance remotely from vMA. Run the following command: j4psh http://[VIN_IP_ADDRESS]:8080/jolokia 

Note: This can take awhile, in my lab environment it took up to 10 minutes before it connected. Please be patient as it could take longer, I can only assume some amount of data is being cached on the client.

Once you have successfully connected, you should now have a prompt displaying the VIN address. There are multiple namespaces or MBeans and we will be looking at the com.vmware.vadm which is the VIN application. To change into this specific Mbean, run the following command: cd com.vmware.vadm as shown in the screenshot above. Once you have switched to VIN's MBean, you can then run the ls command, similar to UNIX/Linux ls (list) command. You can also just type help for all available commands.

What you see listed above in the screenshot are the different sub-MBeans that are available in the com.vmware.vadm Mbean and we will take a look at a few of these to show you how to navigate your way around.

Exploring VIN MBeans:

Note: Please be very careful when executing operations, I would highly recommend you do this in a lab environment to ensure you do not impact your VIN system.

The first MBean we will take a look at is name=inceptionConfigurationMBean,type=ConfigurationMBean and as you might have guessed from the name it is related to some type of configuration. Go ahead and cd into that Mbean and then type ls.

We can see from this Mbean, there are a bunch of attributes that we can access. To view an attribute, we just run the cat command and the name of the attribute. In this example, I am checking the value of vc.credentials.host which should contain the address of the vCenter Server that the VIN Server is currently registered with. Before proceeding to the next Mbean, you will need to cd back out to main MBean, to do so, just type cd .. as you would on a UNIX/Linux machine to change to previous directory.

The next Mean we will take a look at is name=vcInventory,type=VcConnector and as you might have guess this is dealing with the vCenter inventory. Go ahead and cd to this Mbean and type ls.

For this Mbean, when we perform the list command, we are only shown operations or methods for this Mbean. The operations shown on the left will display the required parameters and we can see they all require a string input except for the getAllVms() so let's go ahead and execute that and see what happens (of course we can probably guess, a list of all VMs will be displayed). To execute an operation, you will need to run the exec command, go ahead and run the following command: exec getAllVms as shown in the screenshot above. As you can see from the screenshot, we are listing all VMs that have been discovered by VIN including a nice summary of each virtual machine.

Let's go ahead and run through two additional operations that accept parameters such as getVmByName and getVmByIp. You will use the exec command but now you will need specify an additional parameter based on the operation. The screenshot below demonstrates calling both of these operations and specifying the appropriate input parameter:

The next two operations will be located in the same Mbean which deals with the application services found on each VM as well as any external VM dependencies. The first operation is name=ApplicationService,type=ApplicationService and we will be looking specifically at the findApplicationComponentsByInfrastructureElemenBusinessKeys operation which accepts the MoRef Id of the VM in question which must be in JSON format (this took me awhile to figure out). From the previous operations above, you will need a MoRef Id of a VM you are interested in seeing what application services are running and run the following command: exec findInfrastructureElementsByBusinessKeys '[{"moid":"vm-232"}]' and be sure to replace the MoRef Id with your own.

As you can see from the output, it contains quite a bit of information including the three services this VM is running and one of these services is vCenter Operations. If you hop over to the vSphere Web Client and find this VM, you should see the same set of services listed.

The second operation that we will take a look at is called findOutgoingDependentInfrastructureElements and it also accepts MoRef Id as input and noticed the help option shows it requiring a single entry as well as an array entry of the MoRef Id, again in JSON format. To view the dependency of this VM to other VMs in your virtual infrastructure, run the following command:  exec findOutgoingDependentInfrastructureElements '{"moid":"vm-385"}' '[{"moid":"vm-385"}]' and be sure to replace the MoRef Id with your own.

From the output, you can see the VM's that are depended on by this VM as well as other information.

I have not explored all the Mbeans and operations, but I highly encourage you to browse around and see what you can find. Just be careful when executing operations that you are performing READ operations and not WRITE operations as you can impact your VIN system. In part 2 of this article, I will show you how to easily export some of this useful information found in VIN using a simple script.

Categories // Automation Tags // infrastructure navigator, jmx, jmx4perl, jolokia, notsupported, vIN

Lossless OVF Export in vSphere 5.1 & vCloud Director 5.1

09.27.2012 by William Lam // 5 Comments

Did you know in vSphere 5.1, the new vSphere Web Client now provides a lossless OVF export feature for virtual machines? Previous to this, when you exported a virtual machine only the basic configurations were captured to ensure the virtual machine could be re-imported into another environment that supports OVF. The virtual machine's "virtual hardware personality" such as the BIOS UUID, MAC addresses, VMware specific advanced settings (Extra Configurations), boot order, PCI slot numbers, etc. were not captured as part of the export.

One of the main reasons for doing this is to ensure greater portability of the virtual machine across different environments. By having some of these specific properties tied to the virtual machine, it could limit where a virtual machine could be imported to. However, if you wish to preserve some of these settings during an export, you now have a new advanced option in vSphere 5.1 to specify additional configurations to capture for an OVF export.

When you select the "Enable advanced options" box, you will be provided with a list of configurations that you can preserve and export with the virtual machine.

Note: As the warning message states, only enable the configurations that you absolutely need, else you should stick to the defaults to ensure you have the best portability for your virtual machine export.

There is also a similar feature in vCloud Director 5.1 for vAppTemplates. When you initiate a download of a vAppTemplate, you now have an additional check box to "Preserve identity information" of the vAppTemplate prior to export. This is just a single check box for vCloud Director, you can not specify which configurations you want to preserve.

All these new OVF features can also be accessed with the latest ovftool 3.0.1 release which includes several new enhancements as well as support for both vSphere 5.1 and vCloud Director 5.1. You can find more details in the ovftool 3.0.1 user guide, but here is a summary of what's new in this release.

What Is New in OVF Tool 3.0.1

  • Support for vSphere 5.1 and 5.0.  
  • Support for vCloud Director 5.1, 1.5, and partial support for vCloud Director 1.0
  •    The following new ovftool options: --annotation, --exportFlags, --shaAlgorithm, --sourcePEM, --targetPEM, --vCloudTemplate, --sourceSSLThumbprint, --targetSSLThumbprint, --fencedMode, --noSSLVerify, --vService.

Feature Highlights

  • Full support for vCloud Director 1.5 and 5.1
  • Includes full OVF 1.0, and 1.1 support and backward-compatible mode for importing existing OVF 0.9 packages
  • Supports both import and generation of OVA packages (OVA is part of the OVF standard, and contains all the files of a virtual machine or vApp in a single file.)
  • Directly converts between any vSphere, vCloud Director, VMX, or OVF source format to any vSphere, vCloud Director, VMX, or OVF target format
  • Accesses OVF sources using HTTP, HTTPS, or FTP, or from a local file 
  • Deploys and exports vApp configurations on vSphere 4.0 targets and later and on vCloud Director 1.5 targets and later
  • Provides options to power on a VM or vApp after deployment, and to power off a virtual machine or vApp before exporting (caution advised)
  • Show information about the content of any source in probe mode
  • Provides context sensitive error messages for vSphere and vCloud Director sources andtargets, showing possible completions for common errors, such as an incomplete vCenter inventory path or missing datastore and network mappings
  • Provides an optional output format to support scripting when another program calls OVFTool
  • Uses new optimized upload and download API (optimized for vSphere 4.0 and later)
  • Signs OVF packages and validates OVF package signatures
  • Validates XML Schema of OVF 1.0 and OVF 1.1 descriptors
  • Import and export of OVF packages into a vApprun1.0 workspace. For more information about vApprun, see http://labs.vmware.com/flings/vapprun.

Categories // Automation, OVFTool Tags // export, lossless, ova, ovf, ovftool, vSphere 5.1, vsphere web client

  • « Previous Page
  • 1
  • …
  • 212
  • 213
  • 214
  • 215
  • 216
  • …
  • 224
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Ultimate Lab Resource for VCF 9.0 06/25/2025
  • VMware Cloud Foundation (VCF) on ASUS NUC 15 Pro (Cyber Canyon) 06/25/2025
  • VMware Cloud Foundation (VCF) on Minisforum MS-A2 06/25/2025
  • VCF 9.0 Offline Depot using Synology 06/25/2025
  • Deploying VCF 9.0 on a single ESXi host? 06/24/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025