I was recently reminded of an excellent VMworld 2017 session that given by Ravi Soundararajan, a Principal Engineer at VMware working in our vCenter Server Performance Team. In his session, vCenter Server Performance Deep Dive, Ravi provides some great insights into things to consider that may have an impact on vCenter Server performance. In addition, he also covered a few additional topics, one of which that comes up every so often which around auditing vSphere API usages for a given user. Below are links to both the recording as well as the deck.
If you were not able to watch Ravi's session live, I highly recommend giving the session a watch and downloading the deck as it contains a ton of useful nuggets!
After re-watching Ravi's session on auditing vSphere API usage, I thought it would be cool to automate the manual process he had outlined. With that, I created a PowerShell script called vSphereAPIUsage.ps1 which contains a single function called Get-vSphereAPIUsage. This script requires access to the vpxd.log which a user will need to download from vCenter Server by either running a VC Support bundle or manually retrieving it from the vCenter Server. In addition, you will need to also provide the user session ID that you wish to query. In Ravi's session, he pointed users to the vpxd-profiler.log but I had found that this can also be found within the vpxd.log which saves users from having to look at another file.
Once you have downloaded the vpxd.log locally on your system, go ahead and open it up with your favorite text editor. I highly recommend Microsoft Visual Studio Code, if you do not have one handy or prefer something beyonds notepad or vi. You will need to search for the particular user you wish to perform the query and the string to search for should look like the following (replace with your SSO or AD domain and username)
[Auth]: User VSPHERE.LOCAL\Administrator
I would also recommend searching from the bottom up as you may want the last login from this particular user. Once you have identified the line, you then need to go up three lines until you see "vim.SessionManager.loginByToken" entry and to the right of that (highlighted in green) is the session ID that you need to make a note of. You can also use the opID value to ensure the session ID is in fact related to this login as you may have other log entries in between.
After making a note of the session ID, you can simply call the Get-vSphereAPIUsage and provide it the full path to the downloaded vpxd.log and the session ID that you had retrieved above. Here is an example using the session ID from the screenshot above:
Get-vSphereAPIUsage -VpxdLogFile "C:\Users\lamw\Dropbox\vpxd.log" -SessionId "52bb9a98-598d-26e9-46d0-ee85d3912646"
The results of the script is a tally of all the different vSphere APIs that have been invoked by this particular session/user and its frequency from lowest to highest. In the example above, I had created a new Datacenter entity, created a couple of Clusters, created several VMs, powered on/off and created/deleted snapshot. These operations were all invoked using the vSphere H5 Client, so there will be other vSphere APIs that are in-directly used by the UI such as inventory lookup that may show up. Hopefully this script will come in handy for those that are interested in this information and beats going through the vpxd.log line by line 🙂
Lastly, Ravi also mentioned that you can use the vSphere Flex/H5 Client to get useful information for a given vCenter Server Session such as the client IP Address as well as the number of API invocations. These details can also be retrieved by using the vSphere API itself, have a look at this article here which provides more details.
Ravi S says
Erich Popp says
as usual a very good and interesting post.
the script do have two typos:
Lines 40 and 41 should read: $VpxdLogFile
Btw: do you have a in depth docoumentation (public) about vpxd-profiler.log?
I'm hunting for this because i do have a high CPU usage on a vCenter Appliance setup
Great article as usual! Curious if all LROs are API calls? We're in a situation where we want to track all API call against our ENV in a time series, as well as break it out by user. Currently trying to figure out what actually indicates an API call in the vpxd.log
Great Article ...