WilliamLam.com

Quick Tip - Correctly naming TKR's in Local Content Library for vSphere with Tanzu in vSphere 8

by // Leave a Comment

Customers can create a Tanzu Kubernetes Releases (TKR) content library by either subscribing to VMware's online repository or by creating a local content library and manually importing the images, which can be useful for air-gapped or non-internet accessible environments.

If you automatically subscribe to VMware's online repository, the TKR images (OVF) will automatically be downloaded and will be stored with a default item name that looks like the following:


However, when creating a local content library, customers must manually import the OVF images after downloading them from VMware's online repository (https://wp-content.vmware.com/v2/latest/). During the OVF import wizard, you will notice that each TKR has the same default name called "photon-ova" and you will most likely rename it to something more useful.


Prior to vSphere 8, you could use any name and vSphere with Tanzu would not care as there is metadata associated within each TKR image that provides version that is needed when creating a Tanzu Kubernetes Grid Cluster (TKC).

I was attempting to deploy a TKC using a new TKR version, which I needed to download and import into my vSphere 8 environment and that is where I ran into a strange error:

[Read more...]

Quick Tip - vSphere with Tanzu fails to sync Content Library with 500 Internal Server Error

by // Leave a Comment

While setting up a new vSphere with Tanzu environment (which can run with just 32GB of memory), I ran into a really strange issue where my vSphere Content Library templates were not being picked up by the VM Service. I was going insane as I have configured this a number of times and I have never ran into this particulare issue before. I thought maybe it was a configuration problem but the enablement of vSphere with Tanzu was 100% successful and everything was showing green.

While looking at the vmware-system-vmop-controller-manager container log, I noticed that the VM Service can see the template but it just fails to extract and process it and throws a 500 Internal Server Error message:

E0917 12:08:23.060929 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="a298369b-7239-4b1c-988f-d361e5a001d6" "itemName"="ubuntu-22.04-custom-image"
E0917 12:08:23.060984 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3" "itemName"="photon-ova"
E0917 12:08:23.060998 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="a298369b-7239-4b1c-988f-d361e5a001d6" "clProviderUUID"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061011 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="1bfd8950-c846-46cf-bd50-55150a16bab3" "clProviderUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061032 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061079 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"
E0917 12:08:23.061123 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061165 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"

I was running out of ideas and things to try and I decided to look at the content library configuration to see if I had missed something.

[Read more...]

Heads Up - Verify the SSL certificate trust for your vSphere with Tanzu Content Library

by // 1 Comment

I just learned that SSL certificate for VMware's vSphere with Tanzu Content Library (https://wp-content.vmware.com/v2/latest/lib.json) had just been updated a few days ago and this will have an impact for anyone who had subscribed to the Content Library prior to the certificate update.

Since I had setup the subscribed Content Library several months back, all the OVAs at the time were already sync'ed and there are no immediate errors when the "Fetch Content Library" task is performed (which will be fixed in a future release). However, I was seeing some strange issues with deploying specific versions of Kubernetes and I did not think much of it and deploying another version was fine, so I figured maybe it was just my setup. I also had another lab, so I ended up using that environment most recently.

It is only until you click on a specific Content Library Item and perform a manual sync will you see the following error, which indicates you are affected:

A general system error occurred: HTTP request error: cannot authenticate SSL certificate for host wp-content.vmware.com.


The quickest way to check whether you are affected is by looking at the configured SSL Thumbprint of your subscribed Content Library and comparing that to the vSphere with Tanzu Content Library endpoint.


Unfortunately, the configured SSL Thumbprint for the subscribed Content Library is not visible in the vSphere UI, but this information is available in the vSphere Content Library API.

I just put together this quick PowerCLI snippet which will retrieve the SSL Thumbprint for your subscribed Content Library and compare to the current thumbprint. If it does not match, you will get an error message printing out the current SSL Thumbprint.

Connect-CisServer -Server vcsa.primp-industries.local -User *protected email* -Password VMware1!

$SubscribedCLName = "TKG-Content-Library"
$TKGCLThumbprint = "01:8D:FD:13:A6:9E:CA:AC:CB:7C:67:18:C1:47:11:8C:64:91:5D:C9"

$contentLibraryService = Get-CisService com.vmware.content.library
$LibraryIDs = $contentLibraryService.list()

foreach($libraryID in $LibraryIDs) {
        $library = $contentLibraryService.get($libraryID)
        if($library.name -eq $SubscribedCLName) {
            $ContentLibrary = $Library
            break
        }
}

if($ContentLibrary.subscription_info.ssl_thumbprint -ne $TKGCLThumbprint.toLower()) {
    Write-Error "SSL Thumbprint $($ContentLibrary.subscription_info.ssl_thumbprint) for $SubscribedCLName does not currently match!`n"
}

The fix is straight forward, simply edit the settings of your subscribed Content Library, do not make any changes and then click on OK. Since the configured SSL Thumbprint no longer matches the hosted Content Library, you will be prompted with an action to confirm the new thumbprint and then you can save the settings.


As of right now, the SSL Certificate for the hosted vSphere with Tanzu Content Library is valid until July 7, 2022 and I expect that VMware will replace the TLS certificate prior to that date and this operation will need to be performed again. Since this issue was initially reported internally, I have also asked to see if an official VMware KB to be published.