WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Using the vSphere API to remotely generate ESXi performance support bundles

06.14.2016 by William Lam // 2 Comments

This is a follow-up post from my previous article on how to run a script using a vCenter Alarm action in the vCenter Server Appliance (VCSA). What I had demonstrated was using a pyvmomi (vSphere SDK for Python) script to be triggered automatically to generate a VMware Support Bundle for all ESXi hosts for a given vSphere Cluster. The one caveat that I had mentioned in the blog post was that the solution slightly differed from the original request which was to create an ESXi performance support bundle. However, the vSphere API only supports the creation of generic VMware Support Bundle which may not be as useful if you are only interested in collecting more granular performance stats for troubleshooting purposes.

After publishing the article, I had thought about the problem a bit more and realized there is still a way to solve the original request. Before going into the solution, I wanted to quickly cover how you can generate an ESXi Performance Support Bundle which can be done either directly in the ESXi Shell using something like the following:

vm-support -p -d 60 -i 5 -w /vmfs/volumes/datastore1

or you can actually use a neat little trick which I had blogged about here back in 2011 where you can simply open a web browser and run the following:

https://esxi-1.primp-industries.com/cgi-bin/vm-support.cgi?performance=true&interval=5&duration=60

Obviously, the first option is not ideal as you would need to SSH (generally disabled for good security practices) to each and every ESXi host and then manually run the command and copy the support bundle off of each system. The second option still requires going to each and every ESXi host, however it does not require ESXi Shell or SSH access. This is still not ideal from an Automation standpoint, especially if these ESXi hosts are already being managed by vCenter Server.

However, the second option is what gave me the lightbulb idea! I had recalled a couple of years back that I had blogged about a way to efficiently transfer files to a vSphere Datastore using the vSphere API. The solution leveraged a neat little vSphere API method called AcquireGenericServiceTicket() which is part of the vCenter Server sessionManager. Using this method, we can request a ticket for a specific file with a one time HTTP request to connect directly to an ESXi host. This means, I can connect to vCenter Server using the vSphere API, retrieve all ESXi hosts from a given vSphere Cluster and request a one time ticket to remotely generate an ESXi performance support bundle and then download it locally to the VCSA (or any other place that you can run the pyvmomi sample).

Download the pyvmomi script: generate_esxi_perf_bundle_from_vsphere_cluster.py

Here is the sample log output when triggering this script from vCenter Alarm in the VCSA:

2016-06-08 19:29:42;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-08 19:29:42;INFO;Creating directory /storage/log/esxi-support-logs to store support bundle
2016-06-08 19:29:42;INFO;Requesting Session Ticket for 192.168.1.190
2016-06-08 19:29:42;INFO;Waiting for Performance support bundle to be generated on 192.168.1.190 to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-08 19:33:19;INFO;Requesting Session Ticket for 192.168.1.191
2016-06-08 19:33:19;INFO;Waiting for Performance support bundle to be generated on 192.168.1.191 to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

I have also created a couple more scripts exercising some additional use cases that I think customers may also find useful. Stay tuned for those additional articles later this week.

UPDATE (06/16/16) - There was another question internally asking whether other types of ESXi Support Bundles could also be generated using this method and the answer is yes. You simply just need to specify the types of manifests you would like to collect, such as HungVM for example.

To list the available Manifests and their respective IDs, you can manually perform this operation once by opening browser specifying the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listmanifests=true

Screen Shot 2016-06-16 at 5.38.08 AM
To list the available Groups and their respective IDs, you can specify the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listgroups=true

Screen Shot 2016-06-16 at 5.45.56 AM
Here is an example URL constructed using some of these params:

https://192.168.1.149/cgi-bin/vm-support.cgi?manifests=HungVM:Coredump_VM%20HungVM:Suspend_VM&groups=Fault%20Hardware%20Logs%20Network%20Storage%20System%20Userworld%20Virtual&vm=FULL_PATH_TO_VM_VMX

The following VMware KB 2005715 may also be useful as it provides some additional examples on using these additional parameters.

Categories // Automation, ESXi, vSphere Tags // ESXi, performance, python, pyVmomi, support bundle, vm-support, vm-support.cgi, vSphere API

Quick Tip - iPerf now available on ESXi

03.15.2016 by William Lam // 25 Comments

The other day I was looking to get a baseline of the built-in ethernet adapter of my recently upgraded vSphere home lab running on the Intel NUC. I decided to use iPerf for my testing which is a commonly used command-line tool to help measure network performance. I also found a couple of articles from well known VMware community members: Erik Bussink and Raphael Schitz on this topic as well which were quite helpful. Erik's article here outlines how to run the iPerf Client/Server using a pair of Virtual Machines running on top of two ESXi hosts. Although the overhead of the VMs should be negligible, I was looking for a way to benchmark the ESXi hosts directly. Raphael's article here looked promising as he found a way to create a custom iPerf VIB which can run directly on ESXi.

I was about to download the custom VIB and I had just remembered that the VSAN Health Check plugin in the vSphere Web Client also provides some proactive network performance tests to be run on your environment. I was curious on what tool was being leveraged for this capability and in doing a quick search on the ESXi filesystem, I found that it was actually iPerf. The iPerf binary is located in /usr/lib/vmware/vsan/bin/iperf and looks to have been bundled as part of ESXi starting with the vSphere 6.0 release from what I can tell.

UPDATE (10/19/23) - As of ESXi 8.x, you may run into following error when attempting to start running iperf:

iperf3: running in appDom(30): ipAddr = ::, port = 5201: Access denied by vmkernel access control policy

To workaround this, you can run the following command:

esxcli system secpolicy domain set -n appDom -l disabled

and re-enable (use "enforcing" as the value) once you have completed your iperf test both on the server and client ESXi host.

UPDATE (09/20/22) - As of ESXi 7.0 Update 3 20036589 (possibly earlier) and later, you no longer need to make a copy of the iperf3 utility. You can simply run it from /usr/lib/vmware/vsan/bin/iperf3 and you also do NOT have to lower the security by changing ESXi Advanced Setting execInstalledOnly to FALSE

UPDATE (10/02/18) - It looks like iPerf3 is now back in both ESXi 6.5 Update 2 as well as the upcoming ESXi 6.7 Update 1 release. You can find the iPerf binary under /usr/lib/vmware/vsan/bin/iperf3

One interesting thing that I found when trying to run iPerf in "server" mode is that you would always get the following error:

bind failed: Operation not permitted

The only way I found to fix this issue was to basically copy the iPerf binary to another file like iperf3.copy which it would then allow me to start iPerf in "server" mode. You can do so by running the following command on the ESXi Shell:

cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf3.copy

Running iPerf in "Client" mode works as expected and the copy is only needed when running in "server" mode. To perform the test, I used both my Apple Mac Mini and the Intel NUC which had ESXi running with no VMs.

I ran the iPerf "Server" on the Intel NUC by running the following command:

/usr/lib/vmware/vsan/bin/iperf3.copy -s -B [IPERF-SERVER-IP]

Note: If you have multiple network interfaces, you can specify which interface to use with the -B option and passing the IP Address of that interface.

I ran the iPerf "Client" on the Mac Mini by running the following command and specifying the address of the iPerf "Server":

/usr/lib/vmware/vsan/bin/iperf3 -n 800M -c [IPERF-SERVER]

I also disabled the ESXi firewall before running the test, which you can do by running the following command:

esxcli network firewall set --enabled false

Here is a screenshot of my iPerf test running between my Mac Mini and Intel NUC. Hopefully this will come in handy for anyone needing to run some basic network performance tests between two ESXi hosts without having to setup additional VMs.

esxi-iperf

Categories // ESXi, vSphere 6.0 Tags // ESXi, iperf, network, performance, vSphere 6.0 Update 1, vSphere 6.0 Update 2

Simulating vSphere Performance Metrics using VCSIM

04.01.2014 by William Lam // 7 Comments

A really useful tool that I leverage from time to time is VCSIM (vCenter Simulator) which can be found within the VCSA (vCenter Server Appliance). VCSIM allows you to easily "simulate" a custom vSphere Inventory that can be used for a variety of use cases including custom reports using the vSphere API/CLIs. There are additional capabilities that VCSIM provides and one that I have not explored much is the simulation of vSphere Performance Metrics. Having received a couple of inquires regarding VCSIM and performance metrics, I figure this would be a good opportunity to explore this feature in a bit more detail.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Before getting starting, you should familiarize yourself with VCSIM by reading these two articles here and here.

By default when running VCSIM, there are no Performance Metrics. If you wish to include it, you will need to ensure the following two lines are added to your VCSIM configuration file:

<perfCounterInfo>vcsim/model/PerfCounterInfo.xml</perfCounterInfo>
<metricMetadata>vcsim/model/metricMetadata.cfg</metricMetadata>

The first file contains the performance metric definitions that are supported and the second file will specify which metrics will be simulated. To demonstrate the performance metric capabilities of VCSIM, I will be using the following configuration files which you can just copy/paste:

vghetto-perf-vcsim.cfg - This will be our VCSIM configuration file

<simulator>
  <enabled>true</enabled>
  <initInventory>vcsim/model/vghetto-perf-inventory.cfg</initInventory>
  <hostConfigLocation>vcsim/model/hostConfig</hostConfigLocation>
  <perfCounterInfo>vcsim/model/PerfCounterInfo.xml</perfCounterInfo>
  <metricMetadata>vcsim/model/metricMetadata.cfg</metricMetadata>
  <datastore>
     <global>1</global>
     <cluster>2</cluster>
     <local>1</local>
     <prefix>vghetto</prefix>
  </datastore>
</simulator>

vghetto-perf-inventory.cfg - This will be our VCSIM inventory configuration file

<config>
  <inventory>
    <dc>1</dc>
    <host-per-dc>0</host-per-dc>
    <vm-per-host>0</vm-per-host>
    <poweron-vm-per-host>0</poweron-vm-per-host>
    <cluster-per-dc>1</cluster-per-dc>
    <host-per-cluster>2</host-per-cluster>
    <rp-per-cluster>1</rp-per-cluster>
    <vm-per-rp>3</vm-per-rp>
    <poweron-vm-per-rp>3</poweron-vm-per-rp>
    <dv-portgroups>0</dv-portgroups>
  </inventory>
  <prefix>vGhetto-</prefix>
  <worker-threads>1</worker-threads>
  <synchronous>true</synchronous>
</config>

To ensure everything is working, we can start VCIM by issuing the following command on the VCSA:

vmware-vcsim-start /etc/vmware-vpx/vcsim/model/vghetto-perf-vcsim.cfg

If everything is working, you should be able to login using the vSphere Web/C# Client to view the small inventory we just created. If you are able to see the inventory, then go ahead and stop VCSIM by issusing the following command:

vmware-vcim-stop false

Note: To ensure our inventory is not destroyed each time, you should pass in the 'false' flag else it will automatically be deleted each time. This is useful if you want to preserve your inventory on subsequent reboots.

For all Performance Metric configurations, you will only need to edit the metricMetadata.cfg file. VCSIM supports four different types of Stats model:

  1. Constant
  2. Linear
  3. Square
  4. Triangle

We will take a look at Stats model for 2-4 since the Constant is not all that interesting 🙂 For our examples, we will take a look at each Stats model for the datastore.datastoreIops.average metrics.

Linear Stats Model:

vcsim-perf-linear

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

Square Stats Model:

vcsim-perf-square

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Square</Type>
      <Values>0,10,0,20,0</Values>
      <Periods>300,300,600,600,300</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

Triangle Stats Model:

vcsim-perf-triangle

<Metric id="datastore.datastoreIops.average">
  <Instance id="Default"/>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

I thought this was pretty neat that the VCSIM developers included a couple of Stats Models that could be leveraged right out of the box! As you can see, it is pretty easy to enable various performance metrics simply by identifying the metric(s) you are interested in and specifying the Stats Model and then starting up VCSIM. The other neat thing that I have been asked about before is can VCSIM simulate performance metrics for specific vSphere entities? I originally thought the answer was no until I started to play with the performance metric simulator a bit more and realize there is a List property that you can use to specify the specific objects in which you want data to be displayed.

Here is an example of of the same performance metric we have been looking at but only enabling it for two Datastores:

vcsim-perf-list

<Metric id="datastore.datastoreIops.average">
  <Instance id="List">
    <List>vghettoDS_vGhetto-DC0_C0_0,vghettoDS_vGhetto-DC0_C0_1</List>
    <StatsModel>
      <Type>Triangle</Type>
      <Values>0,20,10,30,0</Values>
      <Periods>600,300,600,900</Periods>
    </StatsModel>
  </Instance>
  <Instance id="HostDatastore"/>
</Metric>

 Note: One thing I noticed while playing with the performance metric simulator is that some times the object in the UI is blank when using the vSphere Web Client. If I use the vSphere C# Client, it is perfectly fine and that is also true if you are using the vSphere API to query for these metrics.

Hopefully this was a good overview of how the VCSIM performance metrics feature works. I know there are a couple of internal folks who have used VCSIM injunction with vCenter Operations and I am also curious to see what other neat uses cases exists for the performance metrics feature. Also, if you have created a really cool metricMetadata.cfg configuration file, feel free to share with the rest of the community!

Categories // Not Supported, VCSA, vSphere Tags // performance, VCSA, vcsim, vcva

  • 1
  • 2
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • VMware Flings is now available in Free Downloads of Broadcom Support Portal (BSP) 05/19/2025
  • VMUG Connect 2025 - Minimal VMware Cloud Foundation (VCF) 5.x in a Box  05/15/2025
  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...