VCF Software Depot Structure Deep Dive for Install & Upgrade

Before deploying a VMware vSphere Foundation (VVF) or VMware Cloud Foundation (VCF) 9.0 environment, the VCF Installer must download the required VCF software binaries and metadata from either an online or offline depot, which can also support air-gapped environments.

One of the lesser-known enhancements in VCF 9.0 is the re-architecture of the VCF software depot into a new unified depot, which consolidates all VCF component binaries and metadata into a consistent, centralized structure.

While it is not necessary to fully grasp the structure of the new unified VCF Software Depot to deploy VCF, I’ve seen some confusion among users about which software binaries are required, both for the initial VCF Fleet deployment, VCF Fleet post-deployment installations and VCF Fleet upgrades. To help make these concepts concrete as well as help users create their offline depot, I will be sharing the specific VCF Download Tool (VCFDT) commands along with the files that are automatically downloaded for each scenario.

Perquisites:

• VCFDT installed on a supported operating system, this is especially important when we get to the Update Manager Download Service (UMDS) part of the setup
• Generate a Broadcom Support Portal (BSP) Download Token

Note: In my example, I have the following configured:

1. flt-depot.rainpole.io is the FQDN of my offline depot, which is running on Ubuntu 24.04 (please ensure you are using supported operating system as noted above)
2. BSP Download Token is stored in a file called /root/token which will be used by VCFDT
3. Environmental variable is also configured called TOKEN which will also contain the same value of the BSP Download Token which will be used by UMDS
4. At least 96GB of storage is required for all downloads and will be stored in the /depot directory

Download Initial VCF Fleet Deployment Binaries

The following command will download only the required VCF software binaries (includes metadata) for the initial VCF Fleet deployment:

vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --automated-install --type=INSTALL

After performing the command above, the following directory structure will be setup along with the respective files used by the VCF Installer:

Note: If you are a VMUG Advantage user and have passed your VCP-VCF certification, you can follow the process outlined here before 11/15 to deploy a new VCF Fleet.

You can also download additional VCF components that can be deployed once your VCF Fleet is up and running, you can use the "vcf-download-tool binaries list --depot-download-token-file=/root/token --vcf-version=9.0.1 --type=INSTALL --sku=VCF" to see a list of additional components that you can then download separately.

Download VCF Fleet Upgrade Binaries

The following command will download specific VCF binary components that will be used for upgrades such as going from VCF 9.0.0 to VCF 9.0.1:

vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component SDDC_MANAGER_VCF
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component VCENTER
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component VRSLCM
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component VROPS
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component NSX_T_MANAGER
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component VCF_OPS_CLOUD_PROXY
vcf-download-tool binaries download --depot-download-token-file=/root/token -d /depot --vcf-version=9.0.1 --type UPGRADE --component VRA

Note: While you can omit the --component argument and simply download all upgrade binaries, it will take up more storage for components that you may not need or have installed, so this would be the best option to download only the applicable components you require. Unlike the initial VCF Fleet binaries, which can be downloaded using VCFDT or directly from Broadcom Support Portal (BSP), the upgrade binaries are NOT available for direct download using BSP and must be downloaded using VCFDT with a BSP download token.

After performing the command above, the additional upgrade files will be placed into the existing structure that was created earlier.

Download ESX Upgrade Binaries

You may have noticed the previous command did NOT download any of the ESX updates, that is because another tool is required to pull down those binaries. To preserve backwards compatibility, the ESX software depot structure was kept and is managed by UMDS. Luckily, UMDS is integrated into the VCFDT, but we do need to install it by running the following command:

vcf-download-tool umds install

When you connect VCF Operations or more specifically, SDDC Manager to your VCF offline depot, it will look for a folder called umds-patch-store as the default directory for your ESX software depot. To download the ESX software updates, we need to configure our BSP download token, which we have already setup as an environmental variable called TOKEN and apply a few additional UMDS configuration to specify the local path (e.g. /depot/umds-patch-store) where the ESX updates will be downloaded to and only enable the version we want, thus reducing the need to download additional updates that we do not require.

vcf-download-tool umds run -S --add-entitlement-token $TOKEN
vcf-download-tool umds run -S -P /depot/umds-patch-store
vcf-download-tool umds run -S -e embeddedEsx-9.0-INTL
vcf-download-tool umds run -S -d embeddedEsx-8.0-INTL
vcf-download-tool umds run -S -d embeddedEsx-7.0-INTL
vcf-download-tool umds run -S -d esxio-8.0-INTL
vcf-download-tool umds run -S -d esxio-9.0-INTL
vcf-download-tool umds run vmware-umds -D

After performing the commands above, the following structure and files will be downloaded and placed into the root of /depot directory as shown in the screenshot below.

At this point, you have downloaded all the required binaries to your VCF offline depot that can then be used to deploy a new VCF Fleet and/or upgrade an existing VCF Fleet. To learn more about the VCFDT and UMDS workflow, please refer to the official documentation.

Connecting VCF Installer to VCF Offline Depot using HTTPS

You will need to setup a web server with a TLS certificate and basic authentication, but for those that want something quick and dirty, check out this Python-only solution where you can simply serve the content directly from the local filesystem.

If you are using a self-signed TLS certificate or custom root CA, we need to perform an additional step by importing the root CA into the VCF Installer, so that a trust is established or you will not be able to connect to your offline depot. Copy the root CA certificate (e.g. flt-depot-rootCA.pem) onto the VCF Installer filesystem and then run the following command which will add it to the trust store and then restart the VCF Installer services for the certificate to be trusted.

STOREPASS=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
DEPOT_ALIAS="vcf_offline_depot"
DEPOT_CERT="flt-depot-rootCA.pem"

keytool -importcert -alias $DEPOT_ALIAS -file $DEPOT_CERT -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $STOREPASS -noprompt
echo 'Y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Note: To configure VCF Installer using HTTP, please see this blog post for more details.

If everything was setup correctly, you should be able to connect to your VCF offline depot over HTTPS and we only need to download the initial VCF Fleet installation binaries.

Configuring SDDC Manager to VCF Offline Depot using HTTPS

The default behavior of the VCF Installer as part of the initial VCF Fleet deployment is to change into the SDDC Manager personality, which means root CA certificate that was imported in the prior step has already been performed. However, if you decide to deploy a new SDDC Manager and retain the VCF Installer, then you will need to perform the exact same step outline above to ensure self-signed or custom root CA is trusted. To connect to your VCF Offline Depot using VCF Operations, navigate to Fleet Management->Lifecycle->VCF Instances->(Select VCF Instance)->Depot Settings

For VCF upgrade scenarios such as going from VCF 9.0.0 to 9.0.1, we have three places to initiate the binary downloads:

1. Fleet Management->Lifecycle->VCF Management->Binary Management->Upgrade Binaries which is for various management components such as VCF Automation, VCF Operations, VCF Identity Brokder, VCF Operations for Logs & VCF Operations for Network
2. Fleet Management->Lifecycle->VCF Instances->(Select VCF Instance)->Binary Management which is for the SDDC Core components such as SDDC Manager, vCenter Server and NSX Manager
3. Fleet Management->Lifecycle->VCF Instances->(Select VCF Instance)->Binary Management->ESXi Components which is for the actual ESX upgrade binaries, which are sourced from our UMDS depot setup

When (3) initiates, SDDC Manager will connect to your VCF Offline Depot using https://flt-depot.rainpole.io/umds-patch-store to download ESX upgrade binaries and this happens without basic authentication. This actually took me awhile to figure out as I initially had the entire VCF Offline Depot serving with basic auth and I kept running into errors when looking at SDDC Manager logs.

You will need the ability to add basic authentication only to https://flt-depot.rainpole.io/PROD directory but not for https://flt-depot.rainpole.io/umds-patch-store and my "hacky" way to validate this was to simply use my http_server_auth.py and reserve umds-patch-store without basic authentication which is how I was able to validate the download of ESX upgrade binaries will not work with basic authentication enabled.

Once SDDC Manager has successfully downloaded the ESX upgrade binaries from your VCF Offline Depot, it will add a status message with the last known download date as you can see from the screenshot.

At this point, you can begin your VCF Upgrade (see official documentation for order of operation).

Note: For new VCF users, when you reach the ESX portion of the upgrade, make sure to check out this blog post on how to create and import ESX Image into VCF Operations so that you can perform the upgrade as this is a new workflow for most users who have not used VCF Operations.