WilliamLam.com

Heads Up - Potential missing vCenter Server Events due to sequence ID overflow

by // 2 Comments

We had a few users who had reported issues with using the VMware Event Broker Appliance (VEBA) solution where VEBA was not receiving any events from the connected vCenter Server. This was really puzzling for the team to debug because the user clearly saw events in both the vSphere UI as well as using vSphere Automation Clients like PowerCLI.

After a bit of debugging with a few of our users (huge thanks to Michael Gasch for driving this), we discovered that in certain environments, the generated sequence number that is used for the vCenter Event ID has overflowed and causes the value to have a negative number. To further complicate the debugging, there are actually two ways of fetching vCenter Server Events using the vSphere API. The first is to just look at the LatestPage property, which will return the most recent events and not care about event ID and the second is to use CreateCollectorForEvents() which is more of an event stream and it does care about the event ID being non-negative. You can probably guess which vSphere API VEBA was using, not only because of our check-pointing feature but LatestPage could lose events from a client request point of view for chatty environments.

[Read more...]

Listing all VMware Horizon Events

by // Leave a Comment

One very useful feature in the latest VMware Horizon (2106) release is the introduction of a new Audit Events API, that will finally allow users to retrieve all Horizon events in a programmatic manner. Previously, the only method of extracting the Horizon events was to unfortunately query the SQL Server database directly.

Since VMware Horizon is not a solution I normally work with, I was actually curious on how many events are produced by Horizon? Back in 2019, I shared how to retrieve and list all vCenter Server Events and ended up publishing a Github repo that folks can quickly view and search for a specific event, which is especially useful if you are using the VMware Event Broker Appliance (VEBA) solution.

After speaking with Chris Halstead, one of the most knowledgable person on VMware Horizon that I know, a list of all known Horizon events has never been published.

[Read more...]

Why does Deploy OVF Template operation show vpxd-extension-[uuid]?

by // Leave a Comment

A question that I had received awhile ago from a customer was how to identify the specific user(s) who have deployed an OVF/OVA? Customers can easily do this by leveraging vCenter Serve Events, which are extremely rich with information that can help answer this and many other questions you might have in your vSphere enviornment.


The first challenge that you will find is that an OVF/OVA import operation is mapped to a generic vCenter TaskEvent, which will require some additional filtering. Secondly, depending on the method that was used to deploy the OVF/OVA such as the vSphere UI or using Automation tools like OVFTool or PowerCLI, you will also slightly diffrent behaviors in terms of the vCenter Server Events that are emitted.

If you deploy an OVF/OVA using the vSphere UI, you may have noticed there are actually two vCenter tasks which are displayed and running simultaneously as shown in the screenshot above. The first task is called "Import OVF package" and you will see that this task is associated with the actual user who initiated the import. The second task called "Deploy OVF template" is then associated with a vCenter system account that handles the actual deployment which will show up with a vpxd-extension-[uuid] username. This occurs because the user who is performing the import is not interacting directly with with vCenter Server, but rather through the vSphere UI which uses a system account to then communicate the operation to vCenter Server.

For this reason, when an OVF/OVA is imported through the vSphere UI, you will need to look at the TaskEvent and filter for the initial import operation. If an OVF/OVA is imported using the vSphere API using something like OVFTool, PowerCLI or any other vSphere SDK, then you will only see the Deploy OVF Template operation and the user associated with that operation is the person who initiated the import.

Using PowerCLI and the Get-VIEvent cmdlet, we can easily filter out these two types of TaskEvents.

Get-VIEvent | where {$_.GetType().Name -eq "TaskEvent" -and $_.FullFormattedMessage -eq "Task: Import OVF package"}

Info                 : VMware.Vim.TaskInfo
Key                  : 2036862
ChainId              : 2036862
CreatedTime          : 4/23/2021 9:30:36 AM
UserName             : vsphere.local\william
Datacenter           : VMware.Vim.DatacenterEventArgument
ComputeResource      : VMware.Vim.ComputeResourceEventArgument
Host                 :
Vm                   :
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : Task: Import OVF package
ChangeTag            :

Get-VIEvent | where {$_.GetType().Name -eq "TaskEvent" -and $_.FullFormattedMessage -eq "Task: Import OVF package"}

Info                 : VMware.Vim.TaskInfo
Key                  : 2036869
ChainId              : 2036869
CreatedTime          : 4/23/2021 9:31:01 AM
UserName             : VSPHERE.LOCAL\vpxd-extension-767f8016-870d-4a98-a457-8247454fa759
Datacenter           : VMware.Vim.DatacenterEventArgument
ComputeResource      : VMware.Vim.ComputeResourceEventArgument
Host                 :
Vm                   :
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : Task: Deploy OVF template
ChangeTag            :