Exploring the new vSphere Privilege Recorder in vSphere 8.0 Update 1

Determining the minimum vSphere privileges that is required to perform a given vSphere operation (UI/API) has been a huge customer challenge to say the least. Strategies have included guessing along with trial and error by creating a custom vSphere Role and slowly removing privileges until you have identified the minimum required privileges. If you are familiar with the vSphere API and know exactly which API methods and properties are consumed, then you can use the vSphere API Reference since every method and property includes the specific privilege required in the documentation, but this method is pretty tedious and time consuming.

If only we had a way to record all the vSphere privilege that was used for a specific set of operation(s) in vCenter Server ... 🤔

Apparently I missed the initial news, but this was actually one of the new features that was introduced in vSphere 8.0 Update 1 called the vSphere Privilege Recorder! 😆

UPDATE (07/25/24) - Looks like the PowerCLI team has productized this capability with a new cmdlet introduced in PowerCLI 13.3 called Get-VIPrivilegeReport

The vSphere Privilege Recorder feature is only consumable when using the vSphere API, specifically using the vCenter REST API (List Privilege Check) and it is disabled by default. As mentioned in the vSphere Privilege Recorder Documentation, to enable this feature you need to add two vCenter Server Advanced Settings: config.vpxd.privCheck.bufferSize and config.vpxd.privCheck.cleanupInterval

While the documentation is a bit light on the details, I came to learn that the recommended value for the buffer size, which is the number of privileges that is stored in memory should be between 5000000 to 10000000, based on our internal scale testing. The clean up interval is simply the interval in which the privilege recordings will be cleaned up, so depending on how large of an environment, you might want the clean up interval to be longer so that you can retrieve the privileges before they are removed. To disable the vSphere Privilege Recording, simply set both values to 0.

Note: To ensure that you have a good experience with how this feature works, you may want to setup a separate vCenter Server where you have a controlled environment to capture the required privileges for any given operation. While there may be some operations where this is not possible, it certainly can help while you get familiar with how the vSphere Privilege Recorder works.

Once the vSphere Privilege Recorder is enabled, it will capture all privilege checks for any operation that is performed in your vCenter Server whether that is internal to the system or user generated from the vSphere UI, API or any other CLI or SDK clients. To retrieve the list of vSphere privileges, we need to call the vSphere Privilege Check REST API and include a list of filters that will help us narrow down the results.

The supported filters for the vSphere Privilege Check API includes vSphere Objects (VM, Hosts, Networks, Folder, etc), Principals (users), Session IDs and Operation IDs (OpID). Depending on your goal, you might use one or more filters to narrow down the results that is returned by the API. Since the consumption of the vSphere Privilege Recorder is using the vSphere API, I have spent some time and created an easy to use PowerShell function called Get-VCenterPrivileges which can accept a number of the supported filters and return the list of privileges for the operation(s) that was performed.

I decided to use PowerShell rather than PowerCLI, because I wanted to demonstrate how the REST API functions by adding a -Troubleshoot option, which will provide the JSON payload for those that might want to use another REST Client to interact with this vSphere API.

Step 1 - Login to the vCenter REST API to retrieve the session token (more details can be found HERE) which will then be passed as a header in subsequent API requests

$vcenter_server = "vcsa.primp-industries.local"
$vcenter_username = "administrator[at]vsphere[dot]local"
$vcenter_password = "VMware1!"

$base64 = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("${vcenter_username}:${vcenter_password}"))

$headers = @{
    "Authorization"="Basic $base64";
    "Content-Type"="application/json";
    "Accept"="application/json";
}

$requests = Invoke-WebRequest -Uri "https://${vcenter_server}/api/session" -Method POST -Headers $headers -SkipCertificateCheck
if($requests.StatusCode -eq 201) {
    $sessionToken = $requests.Content | ConvertFrom-Json
}

Step 2 - Download and source the Get-VCenterPrivileges.ps1 file so we have access to the function by running the following command:

. ./Get-VCenterPrivileges.ps1

vSphere Objects & Principals 

Filtering by objects and principals is what I would recommend to start with because you can easily narrow down the operation to a specific user and the resource an action is being performed on. For example, if you want to understand the minimum privileges required to add CPU, Memory and Hard Disk to a VM, then creating a "Dummy" VM and filtering on that object would ensure you do not get privilege recordings for other VMs. The same goes for filtering by user, you can reduce the noise that you might get from other users or system accounts that are performing operations while the privilege recording is running.

Note: You may also want to disable the vSphere Privilege Recorder once you have completed your desired operation(s) as another way to reduce the results that you need to filter through.

The supported objects are all the vSphere Managed Objects such as ComputeResource, Datacenter, Datastore, DistributedVirtualSwitch, Folder, StoragePod, HostSystem, Network, ResourcePool, VirtualApp and VirtualMachine. You will need to provide an array of these objects including their Managed Object (MoRef) Types and Identifier (MoRef ID) as a simple string object, which we will cover in an example shortly.

The principals is pretty straight forward, it is simply the user you would like to only see the privilege results from and again, the input will be an array of these principals that include the identity domain and username as a simple string object.

For this example, we want to identify the minimum privileges required to add CPU and Memory to a VM. We will narrow down the privilege results to a specific VM and the user who will be performing the reconfiguration operation, so we can exclude everything else that might be coming from other resources and users. While we can use the vSphere UI to perform this operation, one of the challenges with the vSphere Privilege Recorder is that it will record privileges for all actions, both explicit and implicit, such as the vSphere UI application doing a refresh or you navigating to particular part of the inventory which might generate additional privilege requests.

If possible, consider using Automation such as PowerCLI or other API clients to perform the operation, this way you can reduce any additional results that might be generated when using the vSphere UI. I would also recommend creating a separate user that has full administrator rights to perform the operation and use your current administrator account to query for the privileges, this way you can avoid privileges being returned from the session that is requesting the list of privileges, hope that makes sense.

To find the MoRef details, we simply use PowerCLI or any API Client to retrieve the object that we are interested in. In PowerCLI, we can simply use the ExtensionData to retrieve the VM MoRef details by running the following:

(Get-VM Dummy).ExtensionData.MoRef

In this example, our VM is called "Dummy" and has type called VirtualMachine and id of vm-121005, go ahead a make a note of this. Next, as you can see from the screenshot above, we are performing the reconfiguration operation using PowerCLI to update the number of vCPU and Memory by running the following command:

Get-VM Dummy | Set-VM -NumCpu 2 -MemoryGB 4 -Confirm:$false

Once the VM reconfiguration has been completed, the required privileges have already been recorded. We can now retrieve the required privileges by using our Get-VCenterPriviledges function with the following -Objects filters:

Get-VCenterPriviledges -SessionToken $sessionToken -Troubleshoot -Objects @(@{"type"="VirtualMachine";"id"="vm-121005"}) -Principals @(@{"domain"="vsphere.local";"name"="william"})

As you can see from the screenshot above, the simply VM reconfiguration shows  four vSphere privileges that is required and it also display where these privileges would need to be defined in case it interacts with other vSphere Inventory objects.

Note: You might have noticed some additional output at the top which is triggered by the -Troubleshooting option which will print out the vCenter REST API endpoint that was used to perform the operation along with the JSON payload that was used to retrieve the following output, which can be useful for anyone who might want to use a different REST client.

Depending on your configured buffer size and clean up interval, you may see the same privilege results if you perform a different operation on the exact same object. This is where the Marker value comes in handy, which is returned from the API on each request and can be used to tell the API, not to show privileges from a prior invocation.

Make a note of the marker value that was returned from this initial request which is "ftunmh2jT2DDRyJnP4v/4b21DcALe367Wt/3c9sUmwRJDcsGneUigWbKjpFkjZSRRdr5LgPIF6hFM9erTl2quw==". Lets now say we want to add a note/annotation and understand the required privileges for this operation on the same VM using the same user?

Well, we can easily do this using PowerCLI with the following command:

Get-VM Dummy | Set-VM -Notes "Adding an important note" -Confirm:$false

Once the operation has completed, we can now retrieve the privileges by using the exact same filter as above but we are now also appending the -Marker option which we provide the value from above with the following:

Get-VCenterPriviledges -SessionToken $sessionToken -Troubleshoot -Objects @(@{"type"="VirtualMachine";"id"="vm-121005"}) -Principals @(@{"domain"="vsphere.local";"name"="william"}) -Marker "ftunmh2jT2DDRyJnP4v/4b21DcALe367Wt/3c9sUmwRJDcsGneUigWbKjpFkjZSRRdr5LgPIF6hFM9erTl2quw=="

If we now look at the results, we only see the results from this new operation, where as if you were to omit the marker, you would see privileges from the previous operation within the results and that could lead to incorrectly capturing the minimum privileges for a given operation.

vSphere Sessions

Filtering by a specific vSphere Session might be useful as it would allow you capture all operations generated by a given user who is logged in and would automatically rule out other users or system accounts. It is important that you are able to identify the right session, especially for large environments where you might have multiple user sessions from different interfaces such as using the vSphere UI, CLI or API clients.

In the vCenter Server Session Manager, it contains a number of useful fields that you can use to identify the session that you wish to capture the privileges from, including client IP Address and User Agent which can tell you where this session is logging in from. For vSphere UI logins, you will notice the client IP Address is 127.0.0.1, because it is the vSphere UI application that is connecting to the vCenter Server and not the actual IP Address of system the user is logging in from along with the User Agent information, if you only have a single session from say an incognito browser.

To retrieve the session ID, which is what we need, you will need to use the vSphere API. Here is a quick PowerCLI snippet that uses the session manager API to list all sessions. In this example, it is simply filtering by a specific user and hopefully you can quickly narrow down the specific session.

$sessionManager = Get-View $global:DefaultVIServer.ExtensionData.Content.SessionManager
$sessionManager.SessionList | where {$_.UserName -eq "VSPHERE.LOCAL\william"}

Once you have identified the specific session ID, you can provide that as a filter by using the -Sessions argument along with ID

Get-VCenterPriviledges -SessionToken $sessionToken -Troubleshoot -Sessions @("52fcf343-ee6a-47b4-b3cf-58bca9f88424")

vSphere Operation IDs

Another really cool filter option by the vSphere Operation ID (OpID) which is a unique ID for a specific workflow that has been initiated. Typically, the OpID is auto-generated by vCenter Server and can be seen in the vpxd.log files. However, for some vSphere API Clients, you can actually set a custom OpID, which would allow us to easily filter all the required privileges for given workflow that might involved multiple operations across different vSphere objects.

PowerCLI currently does not allow you to specify an OpID, but using pyvmomi (vSphere SDK for Python) and other vSphere API Clients, you can configure a custom OpID for a given script invocation. I have updated my create_random_marvel_vms.py example, which now allows you to configure a custom OpID by using the --opid argument or it will automatically default to "create-marvel-vm" as the OpID. If you have a Marvel account with API keys, you can add those to the script but the script has also been designed to work without providing any Marvel API keys. The python script creates a basic (1vCPU/128MB) VM that is randomly named with the following command:

python create_random_marvel_vms.py --host vcsa.primp-industries.local --user william[at]vsphere[at]local --password VMware1! --datastore sm-vsanDatastore --count 1 --datacenter Primp-Datacenter

Once the operation has completed, we can now use the -OpIds filter and provide either the default OpID value from the script or if you decided to use your own custom one, then just simply substitute that in as screenshot below:

Get-VCenterPriviledges -SessionToken $sessionToken -Troubleshoot -OpIds @("create-marvel-vm")

As expected, we get the list of all the privileges that was used by the script, in this case creating a basic VM.