I was recently helping out fellow colleague Patrick Kremer who was looking into an issue that one of our users had filed on how to configure the VMware Event Broker Appliance (VEBA) so that it can take advantage of a custom container registry for deploying VEBA functions. If you attempt to specify a container image from a private container registry, especially one that has a self-signed certificate, you will see the following error:
Unable to fetch image "harbor.primp-industries.local/library/veba/kn-py-echo:1.0": failed to resolve image to digest: Get "https://harbor.primp-industries.local/v2/": x509: certificate signed by unknown authority; Get "https://harbor.primp-industries.local:443/v2/": x509: certificate signed by unknown authority
I had assumed that this should have been a pretty trivial configuration change to make the underlying Kubernetes container runtime trust the desired container registry and that there would be an easy to follow tutorial that Patrick could search for. The latest release of VEBA has moved away from using the Docker runtime to containerd and this should have helped narrow down the search results, at least that was our assumption.
Not only are there plenty of resources online, but there seem to be multiple methods depending on the version of Kubernetes and containerd which was pretty overwhelming. After several attempts using various blog articles, Patrick found that the trust error has still not gone away. I finally decided to take a closer look and discovered that there are actually two components that must be updated to properly support a private container registry: containerd & Knative Serving Controller. I eventually found this page in the Knative Serving documentation that provided a hint but ultimately, I was not able to fully grok the details until I came across this Github thread that brought clarity on how to create the required secret for the root CA certificate which would allow the Knative Serving controller to trust the root CA certificate.
Below are the instructions for the required changes and I have also attempted to simplify the steps by providing automation snippets that makes it easy for anyone to consume. In my setup, I am using Harbor registry which was built from my Harbor Virtual Appliance but the steps should apply for any other private container registry.
