As part of the setup for vSphere Kubernetes Service (VKS), a local vSphere Content Library needs be created to store the various vSphere Kubernetes releases (VKr) which users typically synchronize from VMware's online VKr Content Library repository.
I typically recommend configuring the content library subscription to only download files when needed, rather than the entire library, which is currently over 200GB+.
After standing up another VKS environment, I needed to download additional VKr images but I could not reuse my existing subscribed content library since it was configured on a different vCenter Server.
With the ability to host a custom vSphere Content Library on my Synology, I realized a better solution would be for me to simply download the full VMware VKr Content Library and host that locally on my network rather than re-downloading the same images each time I have a new deployment.
The VMware VKr Content Library is hosted at https://wp-content.vmware.com/v2/latest/ and there is currently no streamline way of downloading all the files, you need to navigate to each directory and manually download OVF descriptor file, VMDK, manifest and content library metadata file for each VKr.
Luckily, each 3rd party vSphere Content Library contains an items.json file that we use to parse out the VKr's and their respective files and then with some automation, we can easily download each file.
Here is a PowerShell script that will automatically download all required files which you can then place on either your Synology web server or any HTTP(S) endpoint. To ensure that you always have the latest files, you should setup a schedule job to run the script maybe once every couple of weeks as new VKr's are added.
$base_tkg_content_library_uri = "https://wp-content.vmware.com/v2/latest"
$tkg_content_library_lib_url = "${base_tkg_content_library_uri}/lib.json"
$tkg_content_library_items_url = "${base_tkg_content_library_uri}/items.json"
Write-Host -ForegroundColor Cyan "Downloading lib.json"
Invoke-WebRequest $tkg_content_library_lib_url -OutFile "lib.json"
Write-Host -ForegroundColor Cyan "Downloading items.json"
Invoke-WebRequest $tkg_content_library_items_url -OutFile "items.json"
$items = (Get-Content -Raw items.json | ConvertFrom-Json).items
foreach ($item in $items) {
$itemFolderName = $item.name
# Create VKr directory
if(!(Test-Path -Path $itemFolderName)) {
Write-host -ForegroundColor Cyan "Downloading ${itemFolderName} ..."
New-Item -ItemType Directory -Path $itemFolderName | Out-Null
$files = $item.files.hrefs
foreach ($file in $files) {
$itemDownloadUrl = "${base_tkg_content_library_uri}/${file}"
# Download VKr files
Write-host -ForegroundColor Yellow "Downloading ${file} ..."
Invoke-WebRequest $itemDownloadUrl -OutFile ${file}
}
}
}
Here is a screenshot of the script running:
Great script however I needed this to work on linux so I converted it a bit to fit my needs.
Here is the script for linux, pretty basic but it does sync the entire repo.
base_depot_location="/mnt/vcfdepot/tanzu"
base_tkg_content_library_uri="https://wp-content.vmware.com/v2/latest"
tkg_content_library_lib_url="$base_tkg_content_library_uri/lib.json"
tkg_content_library_items_url="$base_tkg_content_library_uri/items.json"
echo "Downloading lib.json"
curl -L -o $base_depot_location/lib.json $tkg_content_library_lib_url
echo "Downloading items.json"
curl -L -o $base_depot_location/items.json $tkg_content_library_items_url
items=$(cat $base_depot_location/items.json)
while read allitems
do
dirname=$(echo $allitems | jq -r '.name')
downloadurls=$(echo $allitems | jq -r '.files[].hrefs')
newdir=$base_depot_location/$dirname
mkdir $newdir
###The --output-dir option is available since curl 7.73.0: So we are using wget who can compare incompletes and specify a directory.
echo $downloadurls | jq -c '.[]' | xargs -I % wget --no-if-modified-sinc -N -P "$newdir/" "$base_tkg_content_library_uri/%"
done < <(echo $items | jq -c '.items[]')
Aii forgot to add the item.json download.
Under "echo $downloadurls..." make a new line with:
wget --no-if-modified-sinc -N -P "$newdir/" "$base_tkg_content_library_uri/$dirname/item.json"
Since there is a item.json in every directory and totally forgot about that one.