WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Using the vSphere API in vCenter Server to collect ESXTOP & vscsiStats metrics

02.15.2017 by William Lam // 1 Comment

Back in 2013, I wrote two articles here and here on how to use the Service Manager API that was introduced in vSphere 5.1 to remotely collect ESXTOP and vscsiStats metrics. At the time, I was told that this API was only available when connecting directly to an ESXi host using the vSphere API. This was still a huge improvement over the previous method which basically required customers to run the commands locally. For ESXTOP, there was resxtop but it was not available for all platforms and some customers still prefer to use the ESXi Shell's version. When I had learned about this API, I was really hoping I could collect both ESXTOP & vscsiStats metrics using vCenter Server which would remove the need to have direct connectivity to each ESXi host.

Last week, an Engineer came across one of my blog posts related to the Service Manager APIs which helped him with a problem he was trying to solve. In the email conversation, I then came to learn from the Engineer that the the Service Manager API can be used from vCenter Server and going directly to the ESXi host was not necessary. It turns out that the QueryServiceList() method which accepts an array of "location" expects a special keyword prefix appended to the list of ESXi hosts that you wish to use the local Service Manager instances on.

The special keyword prefix is "vmware.host." and this is appended to either the Hostname or IP Address of the ESXi hosts being managed by the vCenter Server. For example, in my environment I have an ESXi host (192.168.1.50) that is managed by my VC and so the location string for that host should be "vmware.host.192.168.1.50". If the method was successfully called, you should get back the two service instances for ESXTOP and vscsiStats for each of the ESXi hosts where you can then perform the metric collection.

I have created two sample pyvmomi scripts which exercises the Service Manager API for ESXTOP and vscsiStats:

  • service_manager_esxtop_in_vc.py
  • service_manager_vscsistats_in_vc.py

Note: For more details on how to use Service Manager API to collect ESXTOP and vscsiStats, please refer to this post here and here.

[Read more...]

Categories // Automation, vSphere Tags // esxtop, service manager, vscsiStats, vSphere API

Retrieving vscsiStats Using the vSphere 5.1 API

01.17.2013 by William Lam // Leave a Comment

In my previous article, I talked about the new Service Manager API that was introduced in vSphere 5.1 and how you can retrieve ESXTOP performance data using this new vSphere API. In this article I will show you how to collect vscsiStats data using this same interface. If you are not familiar or have not used vscsiStats before, I would highly recommend you take a look at the Using vscsiStats for Storage Performance Analysis as it goes over some of the basics of vscsiStats and how it works.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.

Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

The first step is to get a reference to the vscsiStats service via the Service Manager (must connect directly to an ESXi 5. host, this is not supported when connecting to vCenter Server) and to invoke an operation for vscsiStats, you will need to use the ExecuteSimpleCommand. For vscsiStats, there are four valid operations:

  • StartVscsiStats
  • FetchAllHistograms
  • ResetVscsiStats
  • StopVscsiStats

To demonstrate the vscsiStats interface, I have written a sample vSphere SDK for Perl script called getVscsiStats.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

StartVscsiStats

This operation starts the vscsiStats collection for ALL virtual machines residing on your ESXi hosts. This is exactly the same operation if you were to only specify the -s option to the vscsiStats command-line. Here is a screenshot of the "start" operation implemented in the script:

You should see a response of OK from the output and this would indicate the vscsiStats collection has started.

FetchAllHistograms

This operation fetches ALL the vscsiStats histogram data similar to specifying the -p All option in the vscsiStats command-line. The output contains the following:

The tag denotes the details about each Virtual Machine:

  • VM Display Name
  • VM VMX Configuration Path
  • VM BIOS UUID
  • vCenter Server UUID

This is then followed by the tag which provides the VMDK name in the format of scsi:X:Y and within each virtual disk section it will contain 13 tags which represents each of the statistics type and their associated values:

  1. VSCSIVsi_DistanceHistogram: Histogram: distance (in LBNs) between successive commands
  2. VSCSIVsi_DistanceLast16Histogram: Histogram: distance (in LBNs) between each command from the closest of previous 16
  3. VSCSIVsi_DistanceReadsHistogram: Histogram: distance (in LBNs) between successive Read commands
  4. VSCSIVsi_DistanceWritesHistogram: Histogram: distance (in LBNs) between successive Write commands
  5. VSCSIVsi_IoLatencyHistogram: Histogram: latency of IOs in Microseconds (us)
  6. VSCSIVsi_IoLatencyReadsHistogram: Histogram: latency of Read IOs in Microseconds (us)
  7. VSCSIVsi_IoLatencyWritesHistogram: Histogram: latency of Write IOs in Microseconds (us)
  8. VSCSIVsi_IoLengthHistogram: Histogram: IO lengths of commands
  9. VSCSIVsi_IoLengthReadsHistogram: Histogram: IO lengths of Read commands
  10. VSCSIVsi_IoLengthWritesHistogram: Histogram: IO lengths of Write commands
  11. VSCSIVsi_OutstandingIOsHistogram: Histogram: number of outstanding IOs when a new IO is issued
  12. VSCSIVsi_OutstandingIOsReadsHistogram: Histogram: number of outstanding Read IOs when a new Read IO is issued
  13. VSCSIVsi_OutstandingIOsWritesHistogram: Histogram: number of outstanding Write IOs when a new Write IO is issued

Here is a screenshot of the "getstats" operation implemented in the script:

Note: In comparing the output between the vscsiStats command-line and this interface, I found the following three statistics are not available:

  • Histogram: latency of IO interarrival time in Microseconds (us)
  • Histogram: latency of IO interarrival time for Reads in Microseconds (us)
  • Histogram: latency of IO interarrival time for Writes in Microseconds (us)

 

ResetVscsiStats

This operation will reset the vscsiStats collection similar to the -r option in the vscsiStats command-line. Here is a screenshot of the "reset" operation implemented in the script:

StopVscsiStats 

This operation will stop the vscsiStats collection similar to the -x option in the vscsiStats command-line. Make sure you perform this operation once you are done retrieving your vscsiStats data. Here is a screenshot of the "stop" operation implemented in the script:

In addition to the four operations, you can also save the output to a file by specifying the --output option along with the name of the file. vscsiStats is an extremely useful tool to help vSphere administrators profile their virtual machine's IO workload and now you can easily collect this information using the vSphere API. Some really cool things you can do with this data is to create some nifty graphs such as the ones here and here.

Categories // Automation Tags // api, ESXi 5.1, service manager, simplecommand, vscsiStats, vSphere 5.1, vsphere sdk for perl

How to obtain GID and LWID from esxtop?

11.20.2010 by William Lam // 11 Comments

Last week I remember seeing a tweet from Duncan Epping regarding the use of VMware vsish to obtain the VMX Cartel ID, also known as the LWID (Leader World Id) for a virtual machine world. This VMX Cartel ID is then used to obtain the GID (Group Id) which was documented by Duncan on his esxtop page to limit the view in esxtop to a particular VM of interest:

VMWID=`vm-support -x | grep |awk '{gsub("wid=", "");print $1}'`
VMXCARTEL=`vsish -e cat /vm/$VMWID/vmxCartelID`
vsish -e cat /sched/memClients/$VMXCARTEL/SchedGroupID

As you can see from the above, Duncan utilizes the vm-support to obtain the WID (World ID) and then using vsish to query for the VMX Cartel ID (LWID). Finally, using LWID, he was able to obtain the GID (Group ID). This example only applies to ESXi, since classic ESX does not include vsish tool.

Here is an example of where to locate the values in esxtop under CPU section:

I remember during some of my skunk work adventures, there were other methods of obtaining these IDs. Due to my VCAP-DCD cramming last week, I did not get a change to further investigate. Now that I have some time, I thought share some of these other methods for obtaining the GID and LWID in both ESX and ESXi.

ESXi

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

~ # /sbin/vmdumper -l
wid=16881 pid=-1 cfgFile="/vmfs/volumes/4cdeeb09-1ad4c18a-5ff9-003048d9586a/scofield/scofield.vmx" uuid="42 34 e6 bc f1 83 c2 db-fb fb 08 73 b7 0c 26 40" displayName="scofield" vmxCartelID=16880

ps - The VMX Cartel ID is actually both the PID and CID within process status in ESXi

~ # ps -Cc | grep "scofield" | grep -v grep | awk '{print $2}'
16880

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

~ # /sbin/vscsiStats -l | grep "scofield" | awk '{print $4}' | sed 's/,//g'
16880

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

~ # /sbin/esxcli vms vm list | grep -A3 "scofield" | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
16880

Obtaining GID

Method1 - Using vscsiStats and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method2 - Using esxcli and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method3 - Using esxcli and vsish to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');/sbin/vsish -e cat /sched/memClients/${VMX_CARTEL}/SchedGroupID
4639

Note: You just need to substitute VM_NAME variable with the display name of of the virtual machine you are interested in. There are actually multiple commands being executed in this one line. If your VM has spaces, make sure you put quotes around it

ESX

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

[root@himalaya ~]# /usr/lib/vmware/bin/vmdumper -l
wid=29250 pid=-1 cfgFile="/vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/STA202G/STA202G.vmx" uuid="42 30 d1 75 c5 3e 81 2a-14 15 1f 86 bb 5b b9 e5" displayName="STA202G" vmxCartelID=29249

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

[root@himalaya ~]# /usr/lib/vmware/bin/vscsiStats -l | grep "STA202G" | awk '{print $4}' | sed 's/,//g'
29249

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

[root@himalaya ~]# /usr/sbin/esxcli vms vm list | grep -A3 STA202G | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
29249

Obtaining GID

Method1 - Using esxcli and esxcfg-info to extract VM's GID

[root@himalaya ~]# VM_NAME=STA202G ;VMX_CARTEL=$(/usr/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
197

Method2 - Using sched-stats to extract GID

[root@himalaya ~]# VM_NAME=STA202G;VMX_CARTEL=$(/usr/lib/vmware/bin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');/usr/bin/sched-stats -t groups | grep "vm.${VMX_CARTEL}" | awk '{print $1}'
197

Method3 - Using /proc information from sched

[root@himalaya ~]# VM_NAME=STA202G ;grep "${VM_NAME}" /proc/vmware/sched/drm-stats | awk '{print $1}'
197

As you can see, there are more than one way to obtain the same exact values and I am sure there are probably a few others. For command simplicity, I would probably recommend method #3 for ESXi and method #3 for ESX. As with anything, be careful when you using these methods as they are not really supported by VMware unless directed by their support organization.

Categories // Uncategorized Tags // esxcli, esxtop, gid, lwid, vmdumper, vscsiStats, vsish

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025