WilliamLam.com

Deployment models for vSphere Content Library

by // 7 Comments

When talking to customers about vSphere Content Library deployments, one question I normally get is how best deploy Content Library for optimal workload deployment, especially in scenarios where remote or branch offices are involved? There are two main deployment models for vSphere Content Library as the title has alluded to. The main difference between the two is whether you have a single vCenter Server or if you have multiple vCenter Servers, with each managing its own vSphere infrastructure?

Lets refer to the single vCenter Server case as Scenario 1 and the multi-vCenter Server case as Scenario 2 and below are the two scenarios outlined with additional details.

Scenario 1 (Single vCenter Server):

In this scenario, which is a fairly common deployment for many smaller to mid-size organizations, where you only have a single or very few vCenter Server(s). They are used to manage several remote locations which only consists of ESXi hosts running at each of the locations and storage local to the site is available. In addition, there are several expected behaviors of the content itself which I have formulated into the following table below:

Content Management Content Distribution Content Deployment
Centrally Managed Sync across the WAN Workloads stored and deployed locally

For this type of an environment, you would first setup a published library which stores all the content that you wish to distribute across your remote sites. Next, you would create subscriber library(s) consuming the published library, but instead of storing the replicated content locally, it is actually stored at each of the remote locations and their respective vSphere Datastore(s). This ensures that content is synchronized from our published library out to each of the remote locations, but when content is requested for deployment, the traffic is local to the site rather than going across the WAN.


In the above scenario, since there is only a single vCenter Server, if it ever becomes unavailable then provisioning and management to the remote location will also be unavailable. This is the expected behavior regardless if Content Library is configured.

[Read more...]

How to audit vSphere API usage?

by // 4 Comments

I was recently reminded of an excellent VMworld 2017 session that given by Ravi Soundararajan, a Principal Engineer at VMware working in our vCenter Server Performance Team. In his session, vCenter Server Performance Deep Dive, Ravi provides some great insights into things to consider that may have an impact on vCenter Server performance. In addition, he also covered a few additional topics, one of which that comes up every so often which around auditing vSphere API usages for a given user. Below are links to both the recording as well as the deck.

If you were not able to watch Ravi's session live, I highly recommend giving the session a watch and downloading the deck as it contains a ton of useful nuggets!

After re-watching Ravi's session on auditing vSphere API usage, I thought it would be cool to automate the manual process he had outlined. With that, I created a PowerShell script called vSphereAPIUsage.ps1 which contains a single function called Get-vSphereAPIUsage. This script requires access to the vpxd.log which a user will need to download from vCenter Server by either running a VC Support bundle or manually retrieving it from the vCenter Server. In addition, you will need to also provide the user session ID that you wish to query. In Ravi's session, he pointed users to the vpxd-profiler.log but I had found that this can also be found within the vpxd.log which saves users from having to look at another file.

Once you have downloaded the vpxd.log locally on your system, go ahead and open it up with your favorite text editor. I highly recommend Microsoft Visual Studio Code, if you do not have one handy or prefer something beyonds notepad or vi. You will need to search for the particular user you wish to perform the query and the string to search for should look like the following (replace with your SSO or AD domain and username)

[Auth]: User VSPHERE.LOCAL\Administrator

I would also recommend searching from the bottom up as you may want the last login from this particular user. Once you have identified the line, you then need to go up three lines until you see "vim.SessionManager.loginByToken" entry and to the right of that (highlighted in green) is the session ID that you need to make a note of. You can also use the opID value to ensure the session ID is in fact related to this login as you may have other log entries in between.


After making a note of the session ID, you can simply call the Get-vSphereAPIUsage and provide it the full path to the downloaded vpxd.log and the session ID that you had retrieved above. Here is an example using the session ID from the screenshot above:

Get-vSphereAPIUsage -VpxdLogFile "C:\Users\lamw\Dropbox\vpxd.log" -SessionId "52bb9a98-598d-26e9-46d0-ee85d3912646"


The results of the script is a tally of all the different vSphere APIs that have been invoked by this particular session/user and its frequency from lowest to highest. In the example above, I had created a new Datacenter entity, created a couple of Clusters, created several VMs, powered on/off and created/deleted snapshot. These operations were all invoked using the vSphere H5 Client, so there will be other vSphere APIs that are in-directly used by the UI such as inventory lookup that may show up. Hopefully this script will come in handy for those that are interested in this information and beats going through the vpxd.log line by line 🙂

Lastly, Ravi also mentioned that you can use the vSphere Flex/H5 Client to get useful information for a given vCenter Server Session such as the client IP Address as well as the number of API invocations. These details can also be retrieved by using the vSphere API itself, have a look at this article here which provides more details.

Translating vSAN VM Object IDs (UUID to VM and VM to UUID)

by // 2 Comments

I was working on one of my vSAN Clusters a few weeks back and I had noticed a bunch of vSAN Objects being listed under the "Other" category within the vSAN Virtual Objects Health view as shown in the screenshot below.


I could not figure out what files or VMs these vSAN objects were actually associated to and it was especially strange since all VMs that were deployed on my vSAN Cluster were already properly showing up under this view and I could not account for these "Other" vSAN Objects. I had reached out to a few folks to see if anyone knew how to identify these objects and the only suggestion I had received back was try to run this python vSAN Health Status script located on one of the ESXi hosts participating in the vSAN Cluster to see if it provided what I needed.

The script is located at /usr/lib/vmware/vsan/bin/vsan-health-status.pyc and you run it like the following:

python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc > /tmp/output

The above command just runs the script and stores its output (which is quite extensive) to /tmp/output. Once the script finishes, you can then open up the file using vi and search for the specific vSAN Object UUID in question. I was able to eventually identify what these vSAN Object UUIDs were mapped to (more on this later), but the overall experience was not ideal and it required SSH access to ESXi host which most customers disable by default. In addition, the process was pretty manual and tedious if you wanted to check multiple vSAN Object UUIDs.

So what did I do, well I looked for a better way of course! It turns out the output produced by vsan-health-status.pyc is actually all available using the vSAN Management API. Not only can you obtain this information programmatically and remotely but you can also retrieve this information by simply going to vCenter Server rather than having to directly connect to an ESXI host which was huge negative for me regarding the previous solution.

[Read more...]