WilliamLam.com

Can the VCSA 6.5 forward to multiple syslog targets?

by // 2 Comments

I had a couple folks ping me recently asking whether the latest vCenter Server Appliance (VCSA) 6.5 release supports forwarding to multiple syslog targets? Currently today, only a single syslog target is officially supported which can be configured using the VAMI UI. I know this is something our customers have been asking about and I know this is something the VC Engineering team is considering.

Having said that, it is possible to configure additional syslog targets on the VCSA, but please be aware this is not officially supported. A couple of these customers understood the support impact and were still interested in a solution as some of their environments mandated multiple redundant syslog targets and using a syslog forwarder/relay was not an option for them.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

When configuring syslog forwarding from the VAMI UI, the configurations are all written to /etc/vmware-syslog/syslog.conf on the VCSA.

With this information, if we want to add additional targets (which can be of the same configuration or different), you simply append additional targets to the syslog configuration file. For example, if I have two syslog targets 192.168.30.110 and 192.168.30.111 and I wish to use the default log level, TCP and 514, I would use the following:

*.* @@192.168.30.110:514;RSYSLOG_SyslogProtocol23Format
*.* @@192.168.30.111:514;RSYSLOG_SyslogProtocol23Format

Once you have saved your changes, you will need to restart the rsyslog service for the change to go into effect. To do so, run the following two commands on the VCSA:

systemctl stop rsyslog
systemctl start rsyslog

One additional thing to note is that the VAMI UI will only show the very last syslog target within the configuration file but if you monitor syslog servers, you will see that logs are indeed being forward to all servers that have been configured in the syslog configuration file.

Deployment models for vSphere Content Library

by // 7 Comments

When talking to customers about vSphere Content Library deployments, one question I normally get is how best deploy Content Library for optimal workload deployment, especially in scenarios where remote or branch offices are involved? There are two main deployment models for vSphere Content Library as the title has alluded to. The main difference between the two is whether you have a single vCenter Server or if you have multiple vCenter Servers, with each managing its own vSphere infrastructure?

Lets refer to the single vCenter Server case as Scenario 1 and the multi-vCenter Server case as Scenario 2 and below are the two scenarios outlined with additional details.

Scenario 1 (Single vCenter Server):

In this scenario, which is a fairly common deployment for many smaller to mid-size organizations, where you only have a single or very few vCenter Server(s). They are used to manage several remote locations which only consists of ESXi hosts running at each of the locations and storage local to the site is available. In addition, there are several expected behaviors of the content itself which I have formulated into the following table below:

Content Management Content Distribution Content Deployment
Centrally Managed Sync across the WAN Workloads stored and deployed locally

For this type of an environment, you would first setup a published library which stores all the content that you wish to distribute across your remote sites. Next, you would create subscriber library(s) consuming the published library, but instead of storing the replicated content locally, it is actually stored at each of the remote locations and their respective vSphere Datastore(s). This ensures that content is synchronized from our published library out to each of the remote locations, but when content is requested for deployment, the traffic is local to the site rather than going across the WAN.


In the above scenario, since there is only a single vCenter Server, if it ever becomes unavailable then provisioning and management to the remote location will also be unavailable. This is the expected behavior regardless if Content Library is configured.

[Read more...]

How to audit vSphere API usage?

by // 4 Comments

I was recently reminded of an excellent VMworld 2017 session that given by Ravi Soundararajan, a Principal Engineer at VMware working in our vCenter Server Performance Team. In his session, vCenter Server Performance Deep Dive, Ravi provides some great insights into things to consider that may have an impact on vCenter Server performance. In addition, he also covered a few additional topics, one of which that comes up every so often which around auditing vSphere API usages for a given user. Below are links to both the recording as well as the deck.

If you were not able to watch Ravi's session live, I highly recommend giving the session a watch and downloading the deck as it contains a ton of useful nuggets!

After re-watching Ravi's session on auditing vSphere API usage, I thought it would be cool to automate the manual process he had outlined. With that, I created a PowerShell script called vSphereAPIUsage.ps1 which contains a single function called Get-vSphereAPIUsage. This script requires access to the vpxd.log which a user will need to download from vCenter Server by either running a VC Support bundle or manually retrieving it from the vCenter Server. In addition, you will need to also provide the user session ID that you wish to query. In Ravi's session, he pointed users to the vpxd-profiler.log but I had found that this can also be found within the vpxd.log which saves users from having to look at another file.

Once you have downloaded the vpxd.log locally on your system, go ahead and open it up with your favorite text editor. I highly recommend Microsoft Visual Studio Code, if you do not have one handy or prefer something beyonds notepad or vi. You will need to search for the particular user you wish to perform the query and the string to search for should look like the following (replace with your SSO or AD domain and username)

[Auth]: User VSPHERE.LOCAL\Administrator

I would also recommend searching from the bottom up as you may want the last login from this particular user. Once you have identified the line, you then need to go up three lines until you see "vim.SessionManager.loginByToken" entry and to the right of that (highlighted in green) is the session ID that you need to make a note of. You can also use the opID value to ensure the session ID is in fact related to this login as you may have other log entries in between.


After making a note of the session ID, you can simply call the Get-vSphereAPIUsage and provide it the full path to the downloaded vpxd.log and the session ID that you had retrieved above. Here is an example using the session ID from the screenshot above:

Get-vSphereAPIUsage -VpxdLogFile "C:\Users\lamw\Dropbox\vpxd.log" -SessionId "52bb9a98-598d-26e9-46d0-ee85d3912646"


The results of the script is a tally of all the different vSphere APIs that have been invoked by this particular session/user and its frequency from lowest to highest. In the example above, I had created a new Datacenter entity, created a couple of Clusters, created several VMs, powered on/off and created/deleted snapshot. These operations were all invoked using the vSphere H5 Client, so there will be other vSphere APIs that are in-directly used by the UI such as inventory lookup that may show up. Hopefully this script will come in handy for those that are interested in this information and beats going through the vpxd.log line by line 🙂

Lastly, Ravi also mentioned that you can use the vSphere Flex/H5 Client to get useful information for a given vCenter Server Session such as the client IP Address as well as the number of API invocations. These details can also be retrieved by using the vSphere API itself, have a look at this article here which provides more details.