WilliamLam.com

Quick Tip - vSphere with Tanzu fails to sync Content Library with 500 Internal Server Error

by // Leave a Comment

While setting up a new vSphere with Tanzu environment (which can run with just 32GB of memory), I ran into a really strange issue where my vSphere Content Library templates were not being picked up by the VM Service. I was going insane as I have configured this a number of times and I have never ran into this particulare issue before. I thought maybe it was a configuration problem but the enablement of vSphere with Tanzu was 100% successful and everything was showing green.

While looking at the vmware-system-vmop-controller-manager container log, I noticed that the VM Service can see the template but it just fails to extract and process it and throws a 500 Internal Server Error message:

E0917 12:08:23.060929 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="a298369b-7239-4b1c-988f-d361e5a001d6" "itemName"="ubuntu-22.04-custom-image"
E0917 12:08:23.060984 1 content_library_provider.go:275] vsphere/contentlibrary "msg"="error extracting the OVF envelope from the library item" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "contentLibraryUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3" "itemName"="photon-ova"
E0917 12:08:23.060998 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="a298369b-7239-4b1c-988f-d361e5a001d6" "clProviderUUID"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061011 1 contentsource_controller.go:203] controllers/ContentSource "msg"="failed to get VirtualMachineImage from content library" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "clProviderName"="1bfd8950-c846-46cf-bd50-55150a16bab3" "clProviderUUID"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061032 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3"
E0917 12:08:23.061079 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="1bfd8950-c846-46cf-bd50-55150a16bab3" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"
E0917 12:08:23.061123 1 contentsource_controller.go:401] controllers/ContentSource "msg"="Error in syncing image from the content provider" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6"
E0917 12:08:23.061165 1 controller.go:317] controller/contentsource "msg"="Reconciler error" "error"="POST https://vcsa.tanzu.local:443/rest/com/vmware/content/library/item/download-session: 500 Internal Server Error" "name"="a298369b-7239-4b1c-988f-d361e5a001d6" "namespace"="" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="ContentSource"

I was running out of ideas and things to try and I decided to look at the content library configuration to see if I had missed something.

[Read more...]

vSphere Event-Driven Automation using VMware Event Router on VMware Cloud on AWS with Knative or AWS EventBridge

by // Leave a Comment

The VMware Event Broker Application (VEBA) is a popular VMware Event-Driven Automation solution that can be consumed using either the open source or commercial offering from VMware. The commercial offering of VEBA is already available to customers today via our Tanzu Application Platform (TAP) offering, which I have previously written about here. The open source offering of VEBA can be consumed in either a pre-packaged Virtual Appliance or a native Kubernetes Application called for those with an existing Kubernetes cluster.

Deploying the VEBA Virtual Appliance is well documented (here and here) and I wanted to spend some time covering the native Kubernetes deployment model, as it there are actually a couple of options and most recently, this came up in a customer discussions as they were interested in forwarding vSphere Events from VEBA to AWS EventBridge.

In the open source version of VEBA, there is a component called the VMware Event Router, which is responsible for connecting to an event source such as vCenter Server and then forwarding those events to a processor which can either be a a function that you have written to react to a specific event using Knative or to AWS EventBridge to integrate with other AWS native services like CloudWatch as an example.

To demonstrate the two different ways to deploy the VMware Event Router, I have created the following Github repo https://github.com/lamw/vsphere-event-driven-automation-vmware-event-router that provides an example to easily deploy the VMware Event Router to an existing Kubernetes cluster. For my environment, I will be using VMware Cloud on AWS and the managed Kubernetes offering called Tanzu services, which is included as part of the base offering and there is no additional cost of running the Kubernetes infrastructure, which is certainly an added bonus 😀

[Read more...]

Using Terraform to activate Tanzu Kubernetes Grid Service on VMware Cloud on AWS

by // 1 Comment

It has been awhile since I have played with Terraform and I was recently investigating on whether I could use Terraform to automate the activation of the Tanzu Kubernetes Grid (TKG) Service on a VMware Cloud on AWS SDDC, which is a part of VMware's new managed Kubernetes offering called VMware Cloud with Tanzu services. Although there is an existing VMware Cloud on AWS (VMC-A) Terraform provider, it currently does not support configuring or managing the TKG Service.

Today, customers can automate VMware Cloud with Tanzu services with a simple REST API and with that in mind, I was curious if calling into a REST API using Terraform was even a thing? While searching online, I not only came to find out that directly calling a REST API using Terraform was a thing but that there were actually a few Terraform providers that enabled this capability. The most popular being Mastercard's Restapi Terraform provider, which was also updated just a couple of weeks ago.

I ended up learning a ton more about Terraform through this exercise and the final solution has been contributed to Ryan Johnson's amazing VMware Terraform Examples repo. I also have to give a huge shoutout to Ryan, who I consider one of the experts in the community for all things VMware and Terraform! I was also able to bounce some ideas and also learn a few new tricks in one of our recent conversations. 

[Read more...]