WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Required vSphere Privilege for Read-Only RESXTOP View

06.25.2013 by William Lam // 1 Comment

Yesterday I received a question about the specific vSphere privilege that is required to view RESXTOP data on an ESXi host. The reason for this request was to create a restricted role for a group of users who only needed to have access to RESXTOP performance data. I did not know the answer off the top of my head, but it was a pretty easy to narrow down the specific privilege with a quick test in my lab.

Through the process of elimination, it turns out you just need the Global.Service managers privilege to view only RESXTOP data. It may not seem intuitive, but the Service Manager is responsible for providing vSphere API access to both RESXTOP as well as vScsiStats interfaces which I have written about here.

In my lab, I created a new role called resxtop and then associated the role with the user(s) within the vSphere inventory. You can centrally manage this using vCenter Server or you can do this directly on an ESXi host, but you will need to ensure the role is create on each and every single ESXi host along with it's user association.

Categories // Uncategorized Tags // esxtop, permission, privilege, resxtop, service manager

Retrieving ESXTOP Performance Data Using the vSphere 5.1 API

01.15.2013 by William Lam // 5 Comments

In vSphere 5.1, VMware introduced a new managed object called the Service Manager. The Service Manager is a generic object that wraps the execution of a single command and it requires a specific set of inputs to invoke a particular service command. This is particularly interesting as it allows users to access both the ESXTOP and vscsiStatsinterface using the vSphere API. Prior to vSphere 5.1, to use ESXTOP you would need to either login to the ESXi Shell to run the local ESXTOP command or connect remotely using the RESXTOP utility which is only available on a Linux system. For vScsiStats, you would need to login to the ESXi Shell as a remote version of this tool does not exist. The Service Manager used to be a private interface, an interesting tidbit is that some of you may have already interacted with this interface without even realizing it if you have used PowerCLI's Get-Esxtop cmdlet. In this article I will show you how to programmatically access ESXTOP using the vSphere API.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.


Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

Both the ESXTOP and vscsiStats services are only available when connecting directly to an ESXi 5.1 host, it is not available when connecting to a vCenter Server. If we browse over to the vSphere MOB, we can clearly see the two services:

The first step is to get a reference to the ESXTOP service via the Service Manager and to invoke an operation for ESXTOP, you will need to use the ExecuteSimpleCommand. For ESXTOP, there are three valid operations:

  • CounterInfo
  • FetchStats
  • FreeStats

To demonstrate the ESXTOP interface, I have written a sample vSphere SDK for Perl script called getEsxtop.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

CounterInfo

This operation only needs to be invoked once and it will provide you with the list of available counters and their associated properties and data types for a given ESXi host. Here is an example of this using the "getcounters" operation implemented in the script:

Each line represents a specific counter type followed by each property name and their data type. For example, the first line is for the Server counter and has the following properties and types:

Property Name Type
MinFetchIntervalInUsec U64
IsVMVisor B
TimeStampInUsec U64
Time S64

Here is a quick diagram to help you visualize the hierarchy of all the ESXTOP counters and their relationships with one another:

Note: This diagram was created using yuml.me and here is the raw text in JSON format if you are interested.

FetchStats

This operation fetches a single snapshot of ALL the ESXTOP statistics which contains two pieces of information:

  • The topology of the counter instances
  • The actual counter instances values

The first section is denoted by ==NUM-OF-OBJECTS== which contains either inventory data that does not change or counter instance structure which describes the relationship between the different counter instances. Here is an example of the first section using the "getstats" operation implemented in the script:

If we take a look at the second line as an example |PCPU|LCPU,24|Core,12|Package,2| we can see that PCPU counter contains 24 LCPU that you would need to then enumerate as well as inventory information describing the CPU's logical cores and physical socket.

To view the enumerated counter instances and their instance values, we need to look in the second portion of the data which is denoted by ==COUNTER-VALUE== within the output. Here is a screenshot of this section and we can see the enumerated LCPU's (24 in total as denoted earlier) and their associated instance values:

Remember you will need to correlate with the counter definitions that was extracted earlier from the "getcounters" and this will help you build up the data. I do have to say it can be a bit confusing when you first look at the raw data, but as you start to play with it a bit more, it will start to make sense. Two useful references that can help with parsing the data is the ESXTOP bible and an article that Luc Dekens wrote awhile back exploring the Get-Esxtop cmdlet which I mentioned earlier leverages this exact interface.

FreeStats

Lastly, once you are done collecting the ESXTOP data, you will need to run the "freestats" operation and this will release any server side resources used during the collection. When this command is invoked, it will free up all resources even for past collections where you might have forgotten to perform this last step. There is no output from this operation as you can see from the example screenshot below:

Even though it is nice to see the ESXTOP interface be accessible via the vSphere API, it is not the easiest interface to use and is definitely geared more towards a developer. For extracting general performance data, I would still recommend using the Performance Manager managed object or one of the above mentioned command-line interfaces. In the next article, I go into more detail about the vscsiStats interface and how to consume it using the vSphere API.

Categories // Automation Tags // api, ESXi 5.1, esxtop, resxtop, service manager, simplecommand, vSphere 5.1, vsphere sdk for perl

What Are the Shadow of vmnic in ESXTOP?

01.04.2013 by William Lam // 6 Comments

In ESXi 5.1, you might have noticed something new called Shadow of vmnic under the USED-BY column in the Network view of ESXTOP.

I initially heard about this from a few followers on twitter and I was curious myself since I could not find any documentation regarding this topic. It took a bit of digging but it turns out these shadow vmnics is actually related to the new VDS Health Check feature released in vSphere 5.1. A shadow port is created automatically for every uplink in your ESXi host and is used for transmitting and receiving health check related packets for each uplink. In ESXTOP, you can monitor the statistics for these shadow ports which can be used to debug/troubleshoot the network health check feature and this is why they are present.

One thing to note, these shadow ports are created regardless of whether or not your ESXi host is connected to a VDS or if you have the network health check features enabled. This is by design and there are four VMkernel modules that are responsible for the network health check feature:

  • vlanmtucheck
  • teamcheck
  • heartbeat
  • healthchk

Disclaimer: Do not modify or disable any of the above VMkernel modules else you can potentially disable network connectivity to your ESXi host (trust me, I know).

After some investigation and testing in my lab, I found that I could unload the above VMkernel modules and the shadow vmnics entries would no longer appear in ESXTOP. To do this, you will need an ESXi 5.1 host which is not running any virtual machines and you will need to run the following commands in this exact order (as there are module dependencies):

vmkload_mod -u vlanmtucheck
vmkload_mod -u teamcheck
vmkload_mod -u heartbeat
vmkload_mod -u healthchk

Once you have successfully unloaded all four VMkernel modules, if you open up ESXTOP, you should no longer see the shadow vmnics. To restore the shadow vmnics, you can either reboot (since the unload is not persistent) OR you can run the following commands in this exact order:

vmkload_mod heartbeat
vmkload_mod teamcheck
vmkload_mod vlanmtucheck

Note: By loading the heartbeat VMkernel module, it also loads the healthchk module.

Categories // Uncategorized Tags // ESXi, ESXi 5.1, esxtop, healthcheck, shadow, vds, vmnic

  • « Previous Page
  • 1
  • 2
  • 3
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025