As part of the setup for vSphere with Tanzu, a local vSphere Content Library needs be created to store the various Tanzu Kubernetes releases (TKr) which users typically synchronize from VMware's online TKr Content Library repository.
I typically recommend configuring the content library subscription to only download files when needed, rather than the entire library, which is currently over 200GB+.
After standing up another vSphere with Tanzu environment, I needed to download additional TKr images but I could not reuse my existing subscribed content library since it was configured on a different vCenter Server.
With the ability to host a custom vSphere Content Library on my Synology, I realized a better solution would be for me to simply download the full VMware TKr Content Library and host that locally on my network rather than re-downloading the same images each time I have a new deployment.
The VMware TKr Content Library is hosted at https://wp-content.vmware.com/v2/latest/ and there is currently no streamline way of downloading all the files, you need to navigate to each directory and manually download OVF descriptor file, VMDK, manifest and content library metadata file for each TKr.
Luckily, each 3rd party vSphere Content Library contains an items.json file that we use to parse out the TKr's and their respective files and then with some automation, we can easily download each file.
Here is a PowerShell script that will automatically download all required files which you can then place on either your Synology web server or any HTTP(S) endpoint. To ensure that you always have the latest files, you should setup a schedule job to run the script maybe once every couple of weeks as new TKr's are added.
$base_tkg_content_library_uri = "https://wp-content.vmware.com/v2/latest" $tkg_content_library_lib_url = "${base_tkg_content_library_uri}/lib.json" $tkg_content_library_items_url = "${base_tkg_content_library_uri}/items.json" Write-Host -ForegroundColor Cyan "Downloading lib.json" Invoke-WebRequest $tkg_content_library_lib_url -OutFile "lib.json" Write-Host -ForegroundColor Cyan "Downloading items.json" Invoke-WebRequest $tkg_content_library_items_url -OutFile "items.json" $items = (Get-Content -Raw items.json | ConvertFrom-Json).items foreach ($item in $items) { $itemFolderName = $item.name # Create TKr directory if(!(Test-Path -Path $itemFolderName)) { Write-host -ForegroundColor Cyan "Downloading ${itemFolderName} ..." New-Item -ItemType Directory -Path $itemFolderName | Out-Null $files = $item.files.hrefs foreach ($file in $files) { $itemDownloadUrl = "${base_tkg_content_library_uri}/${file}" # Download TKr files Write-host -ForegroundColor Yellow "Downloading ${file} ..." Invoke-WebRequest $itemDownloadUrl -OutFile ${file} } } break }
Here is a screenshot of the script running:
Great script however I needed this to work on linux so I converted it a bit to fit my needs.
Here is the script for linux, pretty basic but it does sync the entire repo.
base_depot_location="/mnt/vcfdepot/tanzu"
base_tkg_content_library_uri="https://wp-content.vmware.com/v2/latest"
tkg_content_library_lib_url="$base_tkg_content_library_uri/lib.json"
tkg_content_library_items_url="$base_tkg_content_library_uri/items.json"
echo "Downloading lib.json"
curl -L -o $base_depot_location/lib.json $tkg_content_library_lib_url
echo "Downloading items.json"
curl -L -o $base_depot_location/items.json $tkg_content_library_items_url
items=$(cat $base_depot_location/items.json)
while read allitems
do
dirname=$(echo $allitems | jq -r '.name')
downloadurls=$(echo $allitems | jq -r '.files[].hrefs')
newdir=$base_depot_location/$dirname
mkdir $newdir
###The --output-dir option is available since curl 7.73.0: So we are using wget who can compare incompletes and specify a directory.
echo $downloadurls | jq -c '.[]' | xargs -I % wget --no-if-modified-sinc -N -P "$newdir/" "$base_tkg_content_library_uri/%"
done < <(echo $items | jq -c '.items[]')
Aii forgot to add the item.json download.
Under "echo $downloadurls..." make a new line with:
wget --no-if-modified-sinc -N -P "$newdir/" "$base_tkg_content_library_uri/$dirname/item.json"
Since there is a item.json in every directory and totally forgot about that one.