How to configure VMware Compatibility Guide (VCG) database for vSphere Lifecycle Manager (vLCM) in an air-gapped environment?

I was setting up another vSphere 8 environment in my homelab using my handy Automated vSphere & vSAN 8 Lab Deployment Script and I was reminded of another vSphere Lifecycle Manager (vLCM) question that I had received during VMware Explore Barcelona 2022.

The question was about configuring vCenter Server that would include full vLCM functionality in an air-gapped environment, where internet connectivity would not be available directly or in-directly through the use of a network proxy. Today, the VMware Compatibility Guide (VCG) database that is integrated with vLCM can only be synchronized when when your vCenter Server is connected to VMware's online repository when VMware's Customer Experience Improvement Program (CEIP) is enabled.

To synchronize the VCG database using the vSphere UI, navigate to the vLCM administrator page and under Actions, select the Sync HCL operation, which will download the latest VCG database from VMware's online repo. You can also automate this using the vLCM REST API with the Update Compatibility Data Task API.

Here is a quick PowerCLI snippet performing the same operation but using the API instead:

Connect-CiSServer -Server vcsa.primp-industries.local -user administrator[at]vsphere[dot]local -Password VMware1!
$hclCompatDataService = Get-CisService -Name com.vmware.esx.hcl.compatibility_data
$hclLastUpdateOnline = $hclCompatDataService.get().updated_at

$hclCompatDataService.'update$task'()

However, for environments that can not connect to VMware's online repo, this poses a big operational challenge, especially for those looking to transition from vSphere Update Manager (VUM) to vLCM. For the vSAN HCL, we already provide a solution for air-gapped environments by providing an offline copy of the vSAN HCL database which detailed in VMware KB 2145116.

If a connected vCenter Server can download the VCG that vLCM uses, I figure it should also be possible to replicate that behavior for an air-gapped environment? 🤔

I started by just just performing a manual Sync HCL operation using the vSphere UI and then I started to look at various VUM/vLCM logs to see if there were any hints on how it was getting the VCG database. After a few minutes of browsing, I quickly found that it was using the following script /usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py to perform the online update, which then downloads the VCG database from the following URL: https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz

Quickly looking at the hcl_datastore.py script, I found that it included two options: online-update and offline-update for updating the VCG database. This at least gives hope that there is a way to update the local VCG database from an offline source without requiring internet connectivity to VMware's online repository. The next step was to figure out how to actually download the VCG database, since simply opening up the URL in a browser will result in 401, requiring authorization.

Long story short, I found that it requires an OAuth 2.0 client credentials and all the details (login URL, client id, client secrets, etc.) that is needed to login can be found in the following configuration file /usr/lib/vmware-updatemgr/config/vvs-config.json within the vCenter Server Appliance (VCSA). This now makes sense on how the vSphere UI was providing this functionality, by calling into this python script which then uses the client credentials to download the VCG database and then updates the local VCG database that vLCM uses.

Putting everything together, here is how you can download an offline copy of the VCG for vLCM and update the local copy within the VCSA. This solution is applicable for both vSphere 7.0 and vSphere 8.0 from my testing.

Step 0 - You will need access to an existing VCSA to retrieve the required OAuth client id and secret as mentioned above.

Step 1 - Download a copy of the VCG database. Below are two examples using cURL and PowerShell:

cURL

CLIENT_ID="Details in the blog post"
CLIENT_SECRET="Details in the blog post"

JSON=$(curl -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&grant_type=client_credentials" -X POST https://auth.esp.vmware.com/api/auth/v1/tokens)
TOKEN=$(echo $JSON | jq -r .access_token)

curl -L -H 'Content-Type: application/json' -H "X-Vmw-Esp-Client: $TOKEN" -X GET 'https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz' -o vlcm-vcg-offline.gz

PowerShell

$CLIENT_ID="Details in blog post"
$CLIENT_SECRET="Details in blog post"

$JSON = Invoke-WebRequest -Uri "https://auth.esp.vmware.com/api/auth/v1/tokens" -Method POST -Body "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&grant_type=client_credentials"
$TOKEN = ($JSON.Content | ConvertFrom-Json).access_token

Invoke-WebRequest -Uri "https://vvs.esp.vmware.com/v1/compatible/vcg/bundles/all?format=gz" -Method GET -Headers @{"X-Vmw-Esp-Client"="${TOKEN}";"Content-Type"="application/json"} -OutFile vlcm-vcg-offline.gz

If are were successful in obtaining the access token, then you will be able to download VCG database file which is stored as a gzip compressed file.

Step 2 -  SCP the VCG database file vlcm-vcg-offline.gz to the root directory of your desired VCSA.

Step 3 - Extract the contents of the gzipped VCG database file by running the following command:

gzip -d vlcm-vcg-offline.gz

The extracted file, which is a JSON file, will have same filename (e.g. vlcm-vcg-offline).

Step 4 - To create and/or update the local vLCM VCG database from our offline file, run the following command:

/usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py update-offline --filePath /root/vlcm-vcg-offline

This can take about ~30-40 seconds, but once it has completed, you should see a success message as shown in screenshot above.

If you want to compare this local VCG database with another VCSA that is connected to internet, you can run the following command to get the current version of the database:

/usr/lib/vmware-updatemgr/python/hcl/hcl_datastore.py information

For a newly deployed VCSA that has not downloaded a copy of VCG database, you also not see the sqlite3 hcl_cache.db database file, which is stored under /storage/updatemgr/hcl. This will be created automatically once you perform the offline update.

While this approach to update the local vLCM VCG Database is not as easy as the the vSAN HCL offline update method, it is definitely possible for those requiring this capability. Hopefully we can simplify this in the future for customers that have air-gapped requirements and if this is something you would like to see, feel free to drop a comment below.