While setting up a new vSphere with Tanzu environment (which can run with just 32GB of memory), I ran into a really strange issue where my vSphere Content Library templates were not being picked up by the VM Service. I was going insane as I have configured this a number of times and I have never ran into this particulare issue before. I thought maybe it was a configuration problem but the enablement of vSphere with Tanzu was 100% successful and everything was showing green.
While looking at the vmware-system-vmop-controller-manager container log, I noticed that the VM Service can see the template but it just fails to extract and process it and throws a 500 Internal Server Error message:
E0917 12:08:23.060929 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="a298369b-7239-4b1c-988f-d361e5a001d6" "itemName"="ubuntu-22.04-custom-image"
E0917 12:08:23.060984 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3" "itemName"="photon-ova"
E0917 12:08:23.060998 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="a298369b-7239-4b1c-988f-d361e5a001d6" "clProviderUUID"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061011 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="1bfd8950-c846-46cf-bd50-55150a16bab3" "clProviderUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061032 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061079 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"
E0917 12:08:23.061123 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061165 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"
I was running out of ideas and things to try and I decided to look at the content library configuration to see if I had missed something.
While creating the content library, I remember clicking really fast and I had checked the Apply Security Policy box, which I assumed was a good thing and would not cause any issues?
It turns out, this was the culprit! The new default OVF security policy was introduced back in vSphere 7.0 Update 3 and provides the following functionality:
Starting with vSphere 7.0 Update 3, you can protect the OVF items by applying default OVF security policy to a content library. The OVF security policy enforces strict validation on OVF items when you deploy or update the item, import items, or synchronize OVF and OVA templates. To make sure that the OVF and OVA templates are signed by a trusted certificate, you can add the OVF signing certificate from a trusted CA.
Under normal circumstances, this would not be an issue, especially if you are using VMware's vSphere with Tanzu content library. However, because the content library I had created contained a custom OVF template and it was not signed, it failed the strict validation which resulted in the VM Service not being able to discover and process my OVF template.
Long story short, be careful when enabling default options, even if you think they it can be helpful and make sure you really understand how feature works being doing so. I realized that this was new feature in vSphere 7.0 Update 3 and since I had not come across it before, I took the conservative approach of simply enabling it and wasted several hours of debugging. I think a nice enhancement in the logs would be to state that OVF item does not meet the strict security policy defined for content library, I think that would have really helped and directed me to right place to look.
Thanks for the comment!