vm-support

This is a follow-up post from my previous article on how to run a script using a vCenter Alarm action in the vCenter Server Appliance (VCSA). What I had demonstrated was using a pyvmomi (vSphere SDK for Python) script to be triggered automatically to generate a VMware Support Bundle for all ESXi hosts for a given vSphere Cluster. The one caveat that I had mentioned in the blog post was that the solution slightly differed from the original request which was to create an ESXi performance support bundle. However, the vSphere API only supports the creation of generic VMware Support Bundle which may not be as useful if you are only interested in collecting more granular performance stats for troubleshooting purposes.

After publishing the article, I had thought about the problem a bit more and realized there is still a way to solve the original request. Before going into the solution, I wanted to quickly cover how you can generate an ESXi Performance Support Bundle which can be done either directly in the ESXi Shell using something like the following:

vm-support -p -d 60 -i 5 -w /vmfs/volumes/datastore1

or you can actually use a neat little trick which I had blogged about here back in 2011 where you can simply open a web browser and run the following:

https://esxi-1.primp-industries.com/cgi-bin/vm-support.cgi?performance=true&interval=5&duration=60

Obviously, the first option is not ideal as you would need to SSH (generally disabled for good security practices) to each and every ESXi host and then manually run the command and copy the support bundle off of each system. The second option still requires going to each and every ESXi host, however it does not require ESXi Shell or SSH access. This is still not ideal from an Automation standpoint, especially if these ESXi hosts are already being managed by vCenter Server.

However, the second option is what gave me the lightbulb idea! I had recalled a couple of years back that I had blogged about a way to efficiently transfer files to a vSphere Datastore using the vSphere API. The solution leveraged a neat little vSphere API method called AcquireGenericServiceTicket() which is part of the vCenter Server sessionManager. Using this method, we can request a ticket for a specific file with a one time HTTP request to connect directly to an ESXi host. This means, I can connect to vCenter Server using the vSphere API, retrieve all ESXi hosts from a given vSphere Cluster and request a one time ticket to remotely generate an ESXi performance support bundle and then download it locally to the VCSA (or any other place that you can run the pyvmomi sample).

Download the pyvmomi script: generate_esxi_perf_bundle_from_vsphere_cluster.py

Here is the sample log output when triggering this script from vCenter Alarm in the VCSA:

2016-06-08 19:29:42;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-08 19:29:42;INFO;Creating directory /storage/log/esxi-support-logs to store support bundle
2016-06-08 19:29:42;INFO;Requesting Session Ticket for 192.168.1.190
2016-06-08 19:29:42;INFO;Waiting for Performance support bundle to be generated on 192.168.1.190 to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-08 19:33:19;INFO;Requesting Session Ticket for 192.168.1.191
2016-06-08 19:33:19;INFO;Waiting for Performance support bundle to be generated on 192.168.1.191 to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

I have also created a couple more scripts exercising some additional use cases that I think customers may also find useful. Stay tuned for those additional articles later this week.

UPDATE (06/16/16) - There was another question internally asking whether other types of ESXi Support Bundles could also be generated using this method and the answer is yes. You simply just need to specify the types of manifests you would like to collect, such as HungVM for example.

To list the available Manifests and their respective IDs, you can manually perform this operation once by opening browser specifying the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listmanifests=true

To list the available Groups and their respective IDs, you can specify the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listgroups=true

Here is an example URL constructed using some of these params:

https://192.168.1.149/cgi-bin/vm-support.cgi?manifests=HungVM:Coredump_VM%20HungVM:Suspend_VM&groups=Fault%20Hardware%20Logs%20Network%20Storage%20System%20Userworld%20Virtual&vm=FULL_PATH_TO_VM_VMX

The following VMware KB 2005715 may also be useful as it provides some additional examples on using these additional parameters.

When an issue arises in your vSphere environment and you engage with VMware GSS, one of the first task you will mostly like be asked to perform is to generate a support log bundle and submit it. You can generate a support bundle using the vSphere Web/C# Client as seen in the screenshot below.

Last week I recieved a question from one of our development teams about generating support log bundles for both vCenter Server and ESXi hosts, but using the vSphere API instead of the UI. This functionality is available as part of the DiagnosticManager and the method you will want to call is GenerateLogBundles_Task() which allows you to specify which systems to collect support logs from. This by far is the easiest way if you just want to collect all logs and I will go into more detail in a bit on what I mean by this.

When invoking the GenerateLogBundles_Task(), support logs will be generated on each of the respective sub-systems which includes vCenter Server and the ESXi hosts. Once the log bundles have been created, the API will return a list of download URLs to the support log bundles. To demonstrate this functionality, I have created a sample vSphere SDK for Perl script called generateLogBundle.pl

The script assumes you want to collect ALL logs for both vCenter Server and ESXi hosts being managed by the given vCenter Server.

While working on the sample code, one thing I had noticed in the vSphere Web/C# Client is that you have the ability to select the specific logs to export.

I was looking through the vSphere API but I did not find a way to specify this level of granularity in the logs. I was of course curious and wondered if this was using some sort of private vSphere API (which I hoped was not the case). I decided to pull out one of my favorite tools when I get stuck with a question around the vSphere API which is Onyx. After manually generating the support log bundle and selecting a subset of the logs to export, I found out this process actually makes a call to a CGI script on the ESXi host called vm-support.cgi.

The list of available logs to export is generated by making a GET request to the following URL: https://[ESXI-HOSTNAME]/cgi-bin/vm-support.cgi?listmanifests=1

The output is an XML file that contains the manifest of the available logs to export in the log support bundle. To specify the specific logs to collect, you must provide the "id" property as seen in the screenshot above. In this example, I would like to collect the following: System:Base, VirtualMachines:GuestState, Storage:VSAN and VirtualMachines:diskinfo

To generate a support bundle from the ESXi host that contains the above, you will need to issue a GET request using the following URL: https://[ESXI-HOSTNAME]/cgi-bin/vm-support.cgi?manifests=System:Base VirtualMachines:GuestState Storage:VSAN VirtualMachines:diskinfo

This will instruct the ESXi host to generate the log bundle that contains the following manifest log files. To include others, you just need to append to the URL. You can easily test this out by pasting this URL into a web browser and providing your ESXi credentials and the logs will be available for download. Going back to my original observation, this is actually how vCenter Server provides the ability to export specific logs from each of the ESXi hosts when generating a support bundle.

If you would like to use the vSphere API to provide this same capability, you just need to leverage the AcquireGenericServiceTicket() API method which is available as part of the SessionManager and specify the URL for the support bundle. This will allow you to request a support log bundle from within vCenter Server without having to go directly to each of the ESXi hosts manually. You can see a very similar implementation of this API by taking a look at my How to efficiently transfer files to a datastore in vCenter using the vSphere API and specifically the efficent-upload-files-to-datastore.pl sample code.

Using the vSphere API to remotely generate ESXi performance support bundles

How to generate specific support log bundles for vCenter & ESXi using vSphere API?