WilliamLam.com

Hidden vCenter Debugging Performance Metrics

by // 1 Comment

While extracting the new performance metrics in vSphere 5 for a blog post, I came across a metric type that I had never noticed before, vcDebugInfo. These performance metrics seems to deal with some of the internal performance/counters in vCenter such as lock statistics, MoRef (Managed Object Reference) counts, etc. Majority of these metrics are available in either collection level 1 or 4 which is the highest level containing all statistics. VMware's best practice is to only enable collection level 1 or 2, 3 and 4 should only be enabled under VMware supervision for debugging purposes which is most likely when some of these stats may come in handy. So be warn, these are probably not supported by VMware

I found it interesting that these metrics are hidden from the vSphere Client UI, but they can easily be extracted when going through the vSphere API. It is just amazing on all the goodies you can find when going through the APIs 🙂

Metric Stat Level Description
vcDebugInfo
maximum.millisecond.activationlatencystats 4 The latency of an activation operation in vCenter
minimum.millisecond.activationlatencystats 4 The latency of an activation operation in vCenter
summation.millisecond.activationlatencystats 1 The latency of an activation operation in vCenter
maximum.number.activationstats 4 Activation operations in vCenter
minimum.number.activationstats 4 Activation operations in vCenter
summation.number.activationstats 1 Activation operations in vCenter
maximum.millisecond.hostsynclatencystats 4 The latency of a host sync operation in vCenter
minimum.millisecond.hostsynclatencystats 4 The latency of a host sync operation in vCenter
summation.millisecond.hostsynclatencystats 1 The latency of a host sync operation in vCenter
maximum.number.hostsyncstats 4 The number of host sync operations in vCenter
minimum.number.hostsyncstats 4 The number of host sync operations in vCenter
summation.number.hostsyncstats 1 The number of host sync operations in vCenter
maximum.number.inventorystats 4 vCenter inventory statistics
minimum.number.inventorystats 4 vCenter inventory statistics
summation.number.inventorystats 1 vCenter inventory statistics
maximum.number.lockstats 4 vCenter locking statistics
minimum.number.lockstats 4 vCenter locking statistics
summation.number.lockstats 1 vCenter locking statistics
maximum.number.lrostats 4 vCenter LRO statistics
minimum.number.lrostats 4 vCenter LRO statistics
summation.number.lrostats 1 vCenter LRO statistics
maximum.number.miscstats 4 Miscellaneous statistics
minimum.number.miscstats 4 Miscellaneous statistics
summation.number.miscstats 1 Miscellaneous statistics
maximum.number.morefregstats 4 Managed object reference counts in vCenter
minimum.number.morefregstats 4 Managed object reference counts in vCenter
summation.number.morefregstats 1 Managed object reference counts in vCenter
maximum.number.scoreboard 4 Object counts in vCenter
minimum.number.scoreboard 4 Object counts in vCenter
summation.number.scoreboard 3 Object counts in vCenter
maximum.number.sessionstats 4 The statistics of client sessions connected to vCenter
minimum.number.sessionstats 4 The statistics of client sessions connected to vCenter
summation.number.sessionstats 1 The statistics of client sessions connected to vCenter
maximum.number.systemstats 4 The statistics of vCenter as a running system such as thread statistics and heap statistics
minimum.number.systemstats 4 The statistics of vCenter as a running system such as thread statistics and heap statistics
summation.number.systemstats 1 The statistics of vCenter as a running system such as thread statistics and heap statistics
maximum.number.vcservicestats 4 vCenter service statistics such as events, alarms, and tasks
minimum.number.vcservicestats 4 vCenter service statistics such as events, alarms, and tasks
summation.number.vcservicestats 1 vCenter service statistics such as events, alarms, and tasks

New Performance Metrics In vSphere 5

by // 2 Comments

I recently had to look at some performance metrics in my vSphere 5 lab and I was curious if VMware had documented all the new performance metrics. I headed over to the vSphere 5 API reference guide and to my surprise, they were exactly the same as the vSphere 4 API reference guide. Though looking at the vSphere Client, it was obvious there were new performance metrics for features such as Storage DRS that did not exists in vSphere 4.

Using a similar method in a previous post about Power performance metrics, I extracted all the new metrics in vSphere 5 and created the following table that includes the metric name (rollup,units and internal name), collection level and description of the metric. There are a total of 129 new performance metrics that include Storage DRS and HBR (Host Based Replication).

Hopefully this will be fixed in the API documentation when vSphere 5 GA's as I recalled providing the same feedback during the beta program. 

Metric Stat Level Description
cpu
average.MHz.capacity.provisioned 3 Capacity in MHz of the physical CPU cores
average.MHz.capacity.entitlement 1 CPU resources devoted by the ESX scheduler to virtual machines and resource pools
average.MHz.capacity.usage 3 CPU usage in MHz during the interval
average.MHz.capacity.demand 2 The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
average.percent.capacity.contention 2 Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)
average.number.corecount.provisioned 2 The number of physical cores provisioned to the entity
average.number.corecount.usage 2 The number of virtual processors running on the host
average.percent.corecount.contention 1 Time the VM is ready to run, but is unable to run due to co-scheduling constraints
average.MHz.capacity.demand 2 The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
average.percent.latency 2 Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)
latest.MHz.entitlement 2 CPU resources devoted by the ESX scheduler
average.MHz.demand 2 The amount of CPU resources a VM would use if there were no CPU contention or CPU limit
summation.millisecond.costop 2 Time the VM is ready to run, but is unable to due to co-scheduling constraints
summation.millisecond.maxlimited 2 Time the VM is ready to run, but is not run due to maxing out its CPU limit setting
summation.millisecond.overlap 3 Time the VM was interrupted to perform system services on behalf of that VM or other VMs
summation.millisecond.run 2 Time the VM is scheduled to run
datastore
latest.millisecond.maxTotalLatency 3 Highest latency value across all datastores used by the host
average.KBps.throughput.usage 2 usage
average.millisecond.throughput.contention 2 contention
summation.number.busResets 2 busResets
summation.number.commandsAborted 2 commandsAborted
summation.number.commandsAborted 2 commandsAborted
summation.number.busResets 2 busResets
latest.number.datastoreReadBytes 2 Storage DRS datastore bytes read
latest.number.datastoreWriteBytes 2 Storage DRS datastore bytes written
latest.number.datastoreReadIops 1 Storage DRS datastore read I/O rate
latest.number.datastoreWriteIops 1 Storage DRS datastore write I/O rate
latest.number.datastoreReadOIO 1 Storage DRS datastore outstanding read requests
latest.number.datastoreWriteOIO 1 Storage DRS datastore outstanding write requests
latest.number.datastoreNormalReadLatency 2 Storage DRS datastore normalized read latency
latest.number.datastoreNormalWriteLatency 2 Storage DRS datastore normalized write latency
latest.number.datastoreReadLoadMetric 4 Storage DRS datastore metric for read workload model
latest.number.datastoreWriteLoadMetric 4 Storage DRS datastore metric for write workload model
latest.number.datastoreMaxQueueDepth 1 Storage I/O Control datastore maximum queue depth
disk
average.KBps.throughput.usage 3 Aggregated disk I/O rate
average.millisecond.throughput.contention 3 Average amount of time for an I/O operation to complete
summation.number.scsiReservationConflicts 2 Number of SCSI reservation conflicts for the LUN during the collection interval
average.percent.scsiReservationCnflctsPct 2 Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval
average.kiloBytes.capacity.provisioned 3 provisioned
average.kiloBytes.capacity.usage 2 usage
average.percent.capacity.contention 1 contention
hbr
average.number.hbrNumVms 4 Current Number of Replicated VMs
average.KBps.hbrNetRx 4 Average amount of data received per second
average.KBps.hbrNetTx 4 Average amount of data transmitted per second
managementAgent
average.MHz.cpuUsage 3 Amount of Service Console CPU usage
mem
average.kiloBytes.capacity.provisioned 3 Total amount of memory configured for the VM
average.kiloBytes.capacity.entitlement 1 Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
average.kiloBytes.capacity.usable 2 Amount of physical memory available for use by virtual machines on this host
average.kiloBytes.capacity.usage 1 Amount of physical memory actively used
average.percent.capacity.contention 2 Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory
average.kiloBytes.capacity.usage.vm 2 vm
average.kiloBytes.capacity.usage.vmOvrhd 2 vmOvrhd
average.kiloBytes.capacity.usage.vmkOvrhd 2 vmkOvrhd
average.kiloBytes.capacity.usage.userworld 2 userworld
average.kiloBytes.reservedCapacity.vm 2 vm
average.kiloBytes.reservedCapacity.vmOvhd 2 vmOvhd
average.kiloBytes.reservedCapacity.vmkOvrhd 2 vmkOvrhd
average.kiloBytes.reservedCapacity.userworld 2 userworld
average.percent.reservedCapacityPct 3 Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations
average.kiloBytes.consumed.vms 2 Amount of physical memory consumed by VMs on this host
average.kiloBytes.consumed.userworlds 2 Amount of physical memory consumed by userworlds on this host
average.percent.latency 2 Percentage of time the VM is waiting to access swapped or compressed memory
average.kiloBytes.entitlement 2 Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
average.kiloBytes.lowfreethreshold 2 Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping
none.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
average.KBps.llSwapInRate 2 Rate at which memory is being swapped from host cache into active memory
average.KBps.llSwapOutRate 2 Rate at which memory is being swapped from active memory to host cache
average.kiloBytes.overheadTouched 4 Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM
average.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
maximum.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
minimum.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
none.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
average.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
maximum.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
minimum.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
none.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
average.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
maximum.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
minimum.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
net
average.KBps.throughput.provisioned 2 Provisioned pNic I/O Throughput
average.KBps.throughput.usable 2 Usable pNic I/O Throughput
average.KBps.throughput.usage 3 Average vNic I/O rate
summation.number.throughput.contention 2 Count of vNic packet drops
average.number.throughput.packetsPerSec 2 Average rate of packets received and transmitted per second
average.KBps.throughput.usage.vm 3 Average pNic I/O rate for VMs
average.KBps.throughput.usage.nfs 3 Average pNic I/O rate for NFS
average.KBps.throughput.usage.vmotion 3 Average pNic I/O rate for vMotion
average.KBps.throughput.usage.ft 3 Average pNic I/O rate for FT
average.KBps.throughput.usage.iscsi 3 Average pNic I/O rate for iSCSI
average.KBps.throughput.usage.hbr 3 Average pNic I/O rate for HBR
average.KBps.bytesRx 2 Average amount of data received per second
average.KBps.bytesTx 2 Average amount of data transmitted per second
summation.number.broadcastRx 2 Number of broadcast packets received during the sampling interval
summation.number.broadcastTx 2 Number of broadcast packets transmitted during the sampling interval
summation.number.multicastRx 2 Number of multicast packets received during the sampling interval
summation.number.multicastTx 2 Number of multicast packets transmitted during the sampling interval
summation.number.errorsRx 2 Number of packets with errors received during the sampling interval
summation.number.errorsTx 2 Number of packets with errors transmitted during the sampling interval
summation.number.unknownProtos 2 Number of frames with unknown protocol received during the sampling interval
power
summation.joule.energy 3 Total energy used since last stats reset
average.percent.capacity.usagePct 3 Current power usage as a percentage of maximum allowed power
average.watt.capacity.usable 2 Current maximum allowed power usage
average.watt.capacity.usage 2 Current power usage
storageAdapter
latest.millisecond.maxTotalLatency 3 Highest latency value across all storage adapters used by the host
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
average.percent.OIOsPct 3 The percent of I/Os that have been issued but have not yet completed
average.number.outstandingIOs 2 The number of I/Os that have been issued but have not yet completed
average.number.queued 2 The current number of I/Os that are waiting to be issued
average.number.queueDepth 2 The maximum number of I/Os that can be outstanding at a given time
average.millisecond.queueLatency 2 Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
average.KBps.throughput.usage 4 The storage adapter's I/O rate
storagePath
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
latest.millisecond.maxTotalLatency 3 Highest latency value across all storage paths used by the host
summation.number.busResets 2 Number of SCSI-bus reset commands issued during the collection interval
summation.number.commandsAborted 2 Number of SCSI commands aborted during the collection interval
average.KBps.throughput.usage 2 Storage path I/O rate
sys
latest.second.osUptime 4 Total time elapsed, in seconds, since last operating system boot-up
vcResources
average.kiloBytes.buffersz 4 buffersz
average.kiloBytes.cachesz 4 cachesz
average.number.diskreadsectorrate 4 diskreadsectorrate
average.number.diskwritesectorrate 4 diskwritesectorrate
virtualDisk
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
average.KBps.throughput.usage 2 Virtual disk I/O rate
summation.number.commandsAborted 2 commandsAborted
summation.number.busResets 2 busResets
latest.number.readOIO 2 Average number of outstanding read requests to the virtual disk during the collection interval
latest.number.writeOIO 2 Average number of outstanding write requests to the virtual disk during the collection interval
latest.number.readLoadMetric 2 Storage DRS virtual disk metric for the read workload model
latest.number.writeLoadMetric 2 Storage DRS virtual disk metric for the write workload model

2 Hidden Virtual Machine Gems in the vSphere 5 API

by // 3 Comments

I was recently going through the new vSphere 5 API reference guide and I had stumbled across two new interesting virtual machine features that I did not see any mention of in the vSphere 5 beta documentation.

The first feature is the support for a new e1000e virtual ethernet adapter which provides support for PCI-Express adapters. This is a new virtual device type that has been added in vSphere 5 and is only supported on virtual machines running on Virtual Hardware 8.

The interesting caveat about this feature is that it is not available as an option through the vSphere Client when adding a virtual network adapter to a virtual machine. You only have the option of pcnet,vmxnet2,vmxnet3 or e1000 but not the new e1000e. Since the feature is in the vSphere API, I updated an old vSphere SDK for Perl script called vmNICManagement.pl to support the update of an existing virtual network adapter to an e1000e.

Here is an example of a VM that is running Virtual Hardware 8 with a normal e1000 virtual network adapter which we will then convert into an e1000e adapter.

We then run the script and select the virtual network adapter to update and the operation "updatenictype" and specify the nictype which is e1000e in this example

Once the virtual network adapter has been updated, you can view the virtual machine's settings once again and you will see the new e1000e adapter.

There are also two additional device types that have been introduced in vSphere 5: VirtualUSBXHCIController which is virtual USB Extensible Host Controller Interface (USB 3.0) and VirtualHdAudioCard (check out Kendrick Coleman's post here)

The second feature is the ability configure the boot device order for a virtual machine which is available through the vSphere API but not through the vSphere Client. This feature does not actually change the BIOS boot order device but provides the ability to create an ordered list of preferred boot order devices. Once this list has been exhausted, then it will default to the BIOS list of boot order devices. This feature also requires the virtual machine to be running Virtual Hardware 8.

When viewing the configurations of a virtual machine using the vSphere Client, you have a few options to configure when it comes to the "Boot Options". The only new feature from vSphere 5 that has been exposed in the vSphere Client is the ability to select boot firmware which now includes EFI support. No where in the vSphere Client is there an option to control the boot device order.

The only way to view this new feature is using the vSphere API and using the vSphere MOB we can see that by default this feature is not enabled/configured.

I of course decided to write a simple vSphere SDK for Perl script called updateVMBootOrder.pl which allows you to configure a list of preferred boot devices. The options support cdrom, floppy, hard disk and virtual nic and the last two also require specifying the specific hard disk or virtual nic as a virtual machine can be configured with several.

The script supports a "list" operation which will query an existing virtual machine to check whether boot options have been configured, if so, they will be listed in order they were defined in.

We have confirmed that no boot options have been configured for this VM. If you decide to select a hard disk or virtual network adapter, you will need to run some additional queries to identify the "deviceKey" which is used to identify a particular virtual device. If you select cdrom or floppy, then these additional steps are not necessary.

To view the virtual network adapter device keys, you will need to run the "listnic" operation.

To view the hard disk device keys, you will need to run the "listdisk" operation.

Now let's say you want to configure the following boot device order:

  1. cdrom
  2. ethernet
  3. disk

You will need to use the "update" operation and then specify the devices and their order using --bootorder flag and also the respective --nickey and --diskkey.

If we now re-run the "list" operation, we will see the configured boot devices for this virtual machine.

We can also verify by checking the vSphere MOB for this particular virtual machine and the bootOptions parameter should now be populated.

If you would like to manually add this to a virtual machine's .vmx configuration file you can, but it is definitely not recommended. The following entries map to the example above:

As you can see it pays off to poke around in the vSphere API reference documentation 😀 You should try it sometime!