Creating an offline VMware Cloud Foundation (VCF) Depot for multiple VCF environments

By default, SDDC Manager uses the online VMware Depot for downloading all software updates (security, patch, upgrade) for life-cycling a single VMware Cloud Foundation (VCF) instance. SDDC Manager can also support air-gapped or disconnected environments by leveraging the VCF Offline Bundle Transfer Utility (OBTU), which my colleague Eric Gray recently published a detailed blog post demo'ing OBTU.

While a solution exists for managing an air-gapped VCF environment, the process can be time consuming if you have to support multiple VCF environment as you need to transfer all VCF bundles from where the OBTU is running to each individual SDDC Manager. Ideally, you vi  just host your own offline VCF Depot that all SDDC Managers can simply point to, similar to what is used by vCenter Server with the their Update Manager Download Service (UMDS).

Luckily, an offline depot is not a foreign concept for VCF and while researching this topic, I was even pointed to this old 2020 blog post in creating your own local VCF 3.x depot, but the process was out of date and more importantly, OBTU was not available at the time.

By leveraging a combination of OBTU and the 2020 blog post, I was able to come up with an updated process to create an offline VCF depot that could be used by multiple VCF environments. I was able to successfully validate my offline VCF depot by applying the latest VCF 5.1.1 update to my existing VCF 5.x environment!

Step 1 - You will need to setup and configure an HTTPS web server, which will host your offline VCF depot. In my environment, I already had an Ubuntu Linux VM running, so I simply installed apache and configured it with a self-signed TLS certificate. You can search online for many tutorials on configuring a basic HTTPS web server, which must also include basic authentication (see htaccess), which is required for configuring the VCF depot user within SDDC Manager.

Step 2 - Download and install the VCF OBTU from the Broadcom Support Portal (BSP)

Note: If you run into "[: x86_64: unexpected operator" error when running the lcm-bundle-transfer-util script you need to edit the script at the top and replace it with #!/usr/bin/bash

Step 3 - Create the following directory structure (/PROD2/evo/vmw) under the root directory of your web server. For my setup, the root web server is under /var/www/html but instead of using the root filesystem, I added a second virtual disk to the Ubuntu VM which was then mounted under /depot and I created a symlink so that the directory used to store all VCF bundle would now be under /var/www/html/depot and you would run the following command to create the desired directory structure:

mkdir -p /var/www/html/depot/PROD2/evo/vmw

Step 4 - We are now going to use the OBTU to download the VCF manifest, vSphere VCG and vSAN HCL compatibility files and our desired VCF release(s) to our web server.

lcm-bundle-transfer-util --download --manifestDownload --depotUser 'user[at]company' --outputDirectory /var/www/html/depot/PROD2/evo/vmw
lcm-bundle-transfer-util --download --compatibilityMatrix --depotUser 'user[at]company' --outputDirectory /var/www/html/depot/PROD2/evo/vmw
lcm-bundle-transfer-util --download --vsanHclDownload --depotUser 'user[at]company' --outputDirectory /var/www/html/depot/PROD2/evo/vmw
lcm-bundle-transfer-util --download --sourceVersion 5.1.1.0 --depotUser 'user[at]company' --outputDirectory /var/www/html/depot/PROD2/evo/vmw

Here is what directory structure on my web server looks after downloading all the required files and since I am only interested in VCF 5.1.1, it do not have to waste storage hosting other VCF bundles that I do not require.

Step 5 - Since we are only downloading and hosting a subset of the VCF bundles, we need to create a new index.v3 file in /var/www/html/depot/PROD2/evo/vmw directory that reflects the VCF bundles that we have downloaded. Change to the /vmw directory and run the following command which will search through the downloaded VCF bundles and compare that to original full index.v3 and generate a new index file that contains the VCF bundles and their bundle IDs.

for i in $(ls bundles);
do
    BUNDLE=$(echo ${i%.tar})
    grep ${BUNDLE} tmp/index.v3 >> index.v3
done

Note: To make sure there were no weird permission issues, I changed ownership of all files under /var/www/html/depot/PROD2 to www-data (chown -R www-data:www-data) and set the permissions to 750 (chmod -R 750).

Step 6 - You should should confirm that you can reach /PROD2/evo/vmw using a web browser and that it is prompting for credentials, which should have been setup as part of your web server configuration using something like htaccess.

Step 7 - If you are using a self-signed TLS certificate on your web server, you will also need to add the certificate to the SDDC Manager certificate store or you will get an error when you attempt to add the VCF depot user. Get a copy of your TLS certificate (not the private key) and save that to the filesystem of SDDC Manager, in my example I placed it under /tmp/apache-selfsigned.crt.

Next, run the following command to import our TLS certificate with the desired alias name (e.g. vcf_custom_depot) and then restart SDDC Manager services for the changes to go into effect:

keytool -importcert -alias vcf_custom_depot -file /tmp/apache-selfsigned.crt -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Step 8 - We now need to to update our SDDC Manager configuration file /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties to point to our offline VCF depot.

• lcm.depot.adapter.host - This should be the hostname or IP Address of your web server
• lcm.depot.adapter.remote.rootDir - This may need to get updated depending if you have additional top level directories before /PROD2/evo/vmw (in my environment, I needed to add /depot/PROD2)
• lcm.depot.adapter.enableBundleSignatureValidation - Changing this to false can help speed up VCF bundle validation but is not necessary
• lcm.depot.adapter.certificateCheckEnabled - Not sure if this is still needed due to self-signed TLS certificate but I had set it to false
• lcm.depot.bundleElement.patchFile.checksumValidation - Changing this to false can help speed up VCF bundle validation but is not necessary
• lcm.core.manifest.poll.interval - Changing this to 1000 (ms) can significantly speed up VCF bundle manifest processing

We need to restart the lcm service for the changes to go into effect by running the following command:

systemctl restart lcm

Step 8 - We now need to import the VCF manifest JSON file to SDDC Manager and easiest way to do that is using the embedded API Explorer within SDDC Manager. Search for "manifest" and then copy the contents from /var/www/html/depot/PROD2/evo/vmw/tmp/lcmManifest.json into the payload and then click on the Execute button to send the API request. You should see a 202 response that the JSON payload was accepted without any errors.

Step 9 - Navigate to the Online Depot tab within SDDC Manager and then add the username and password that you had created to access your web server (e.g. htaccess account).

Note: If you are hitting any errors while adding the VCF depot user, it is helpful to take a look at /var/log/vmware/vcf/lcm/lcm-debug.log file and is typically related to either filesystem permission or TLS certificate trust was not setup correctly.

Step 10 - Finally, we can now head over to Lifecycle Management->Bundle Management and we should now see the list of available bundles which we can either download now or schedule later from our centrally managed offline VCF depot!

While setting up an offline VCF depot is not as trivial as I would like, it can certainly aid those that need to manage software for multiple VCF environments and best of all, you can centrally manage the download from a single location and serve that exact same content to multiple environments. I know the VCF team have been working to improve VCF bundle distribution and perhaps this can be simplified and configurable directly within SDDC Manager in a future release of VCF 🙂