WilliamLam.com

How to configure Knative and containerd in VMware Event Broker Appliance (VEBA) to use a private registry?

by // 1 Comment

I was recently helping out fellow colleague Patrick Kremer who was looking into an issue that one of our users had filed on how to configure the VMware Event Broker Appliance (VEBA) so that it can take advantage of a custom container registry for deploying VEBA functions. If you attempt to specify a container image from a private container registry, especially one that has a self-signed certificate, you will see the following error:

Unable to fetch image "harbor.primp-industries.local/library/veba/kn-py-echo:1.0": failed to resolve image to digest: Get "https://harbor.primp-industries.local/v2/": x509: certificate signed by unknown authority; Get "https://harbor.primp-industries.local:443/v2/": x509: certificate signed by unknown authority

I had assumed that this should have been a pretty trivial configuration change to make the underlying Kubernetes container runtime trust the desired container registry and that there would be an easy to follow tutorial that Patrick could search for. The latest release of VEBA has moved away from using the Docker runtime to containerd and this should have helped narrow down the search results, at least that was our assumption.

Not only are there plenty of resources online, but there seem to be multiple methods depending on the version of Kubernetes and containerd which was pretty overwhelming. After several attempts using various blog articles, Patrick found that the trust error has still not gone away. I finally decided to take a closer look and discovered that there are actually two components that must be updated to properly support a private container registry: containerd & Knative Serving Controller. I eventually found this page in the Knative Serving documentation that provided a hint but ultimately, I was not able to fully grok the details until I came across this Github thread that brought clarity on how to create the required secret for the root CA certificate which would allow the Knative Serving controller to trust the root CA certificate.

Below are the instructions for the required changes and I have also attempted to simplify the steps by providing automation snippets that makes it easy for anyone to consume. In my setup, I am using Harbor registry which was built from my Harbor Virtual Appliance but the steps should apply for any other private container registry.

[Read more...]

Packer reference for VMware Harbor Virtual Appliance

by // 2 Comments

I recently had a need to setup a container registry for a project that I was working on and Harbor was of course my default choice. Although Harbor is pretty easy to setup, I did not want to manually go through the installation each time I needed Harbor and I figured it was time to build my own Harbor Virtual Appliance (OVA), just like I have shown in the past with these reference implementations here and here.

UPDATE (02/03/23) - VMware has productized and is now shipping an official VMware Harbor Virtual Appliance (OVA) as part of the latest Tanzu Kubernetes Grid (2.1) release.

For those interested, you can find the reference implementation for building a Harbor Virtual Appliance at https://github.com/lamw/harbor-appliance

When deploying the Harbor Appliance, you will find the basic OVF properties that I have encoded including networking, credentials, debugging and advanced settings. Hopefully should be pretty straight forward for anyone who has deployed an OVA before to vSphere.

[Read more...]

Admin account for embedded Harbor registry in vSphere with Kubernetes

by // 3 Comments

After setting up a vSphere with Kubernetes Cluster, customers have the option of enabling a built-in private container registry that can be used with the Supervisor Cluster. This private container registry uses the popular Opensource Harbor solution which is also a Cloud Native Computing Foundation (CNCF) project.


Although this is a convenient capability, one thing to be aware of is that the embedded Harbor registry is limited in functionality compared to a standalone Harbor deployment and this is by design. When logging into Harbor with your vCenter SSO user, you will be able to do perform basic operations such as pushing and pulling images from this registry. For customers that require additional functionality from Harbor, it is recommended that you setup an external Harbor instance which can also be used as a common registry for both the Supervisor Cluster as well any Tanzu Kubernetes Grid (TKG) Clusters that you may provision.

With that said, I have heard from a few folks who were interested in accessing the Harbor UI using the "admin" account, mostly from an exploration standpoint. The admin credentials for Harbor are dynamically generated each time the service is enabled and it is stored as a K8s secret within the Supervisor Cluster. This means the admin password is unique for each environment and the instructions below will show you how to obtain the credentials.

UPDATE (12/16/20) - I was informed by Engineering the ability to read K8s secrets was actually a bug and this has since been fixed in the latest release of vSphere with Tanzu. If you need the harbor credentials, you will need to directly login to the Supervisor Cluster from the VCSA (instructions have been updated below) to retrieve this information.

Disclaimer: This is not officially supported by VMware and the behaviors described below could change in the future without notice.

[Read more...]