Configure non-secure Harbor registry with Tanzu Kubernetes Grid (TKG)

In an earlier blog post, I shared the steps to to configure Harbor with a proper signed SSL certificate that would serve as  private container registry for Tanzu Kubernetes Grid (TKG) CLI running in an air-gapped environment.

Although Harbor can easily be configured to support custom CA signed certificate, self-sign certificate and even just using HTTP, there are several additional steps and dependencies that is required if you wish to use a non-secure container registry with TKG CLI. This definitely was a bunch of trial/error and hopefully this can be made easier in the future to easily enable non-secure registry support with TKG CLI out of the box for development and testing purpose.

I also want to give a huge thanks to Jun Wang from our Modern Application Business Unit (MAPU), he was instrumental in helping me out and ultimately his tip on updating the containerd configuration was the last piece to the puzzle so that the K8s images deployed would use our insecure Harbor registry for pulling container images.

Step 1 - Install Photon OS into a VM that will be used to run your Harbor instance. Internet connectivity is required to initially download the required containers, but it possible to import from another system which has internet access. You can search online for instructions on how to do that. Once Photon OS has been installed, please run the following two commands to update the OS and also install Perl which will be used in a subsequent step.

tdnf -y update
tdnf -y install perl

Step 2 - Enable and start the docker client:

systemctl enable docker
systemctl start docker

Step 3 - To be able to use the Docker client to push containers to our insecure registry, we will need to create the following configuration to allow us to connect and push containers into our insecure registry. In this example, I will be using IP Address of 192.168.2.10 and port 80.

cat > /etc/docker/daemon.json << EOF
{
"insecure-registries": ["http://192.168.2.10:80"]
}
EOF
systemctl restart docker

Step 4 - We need to configure the following firewall rule to allow connectivity from the KinD (Kubernetes in Docker) cluster which we will deploy and will be used as part of the TKG bootstrap process.

iptables -A INPUT -i docker0 -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save

Step 5 - Download and install KinD:

curl -L https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-linux-amd64 -o /usr/local/bin/kind

chmod +x /usr/local/bin/kind

Step 6 - Download and install Docker Compose which is required to run Harbor:

curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Step 7 - Download and install Kubectl which is required by TKG CLI:

curl -L https://storage.googleapis.com/kubernetes-release/release/v1.18.1/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
chmod +x /usr/local/bin/kubectl

Step 8 - Download and extract the Harbor Offline Installer:

curl -L https://github.com/goharbor/harbor/releases/download/v1.10.2/harbor-offline-installer-v1.10.2.tgz -o harbor-offline-installer-v1.10.2.tgz
tar xvzf harbor-offline-installer*.tgz
rm -f harbor-offline-installer-v1.10.2.tgz

Step 9 - Change into harbor directory and edit the harbor.yml configuration file. First, comment out the entire https section as we are just going to be using http. Next, update the following properties with the respective values of your environment and then save the changes and exit.

Step 10 - Run the following command to start the Harbor installation:

./install.sh

Step 11 - Once the installation has successfully completed, we should verify that we can login to our registry by running the following command specifying the admin password you had set during the Harbor installation:

docker login -u admin -p VMware1! 192.168.2.10:80/library

We can also login to the Harbor UI by opening browser to http://192.168.2.10

Step 12 - After we have verified that we can login to our registry, go ahead and download the following mirror_tkg_containers.sh shell script which will automatically download, tag and push the required containers needed by TKG into our Harbor registry. In addition, the script will also update all the respective TKG YAML manifest files to replace the default VMware registry with our Harbor registry.

You will need to edit the script before running it and at the top you will need to update the registry URL which will just be the IP Address and port of the Harbor deployment (e.g. 192.168.2.10:80)

./mirror_tkg_containers.sh

Step 13 - Once the script has completed, all required containers will now be available in Harbor. We can now disconnect or disable internet connectivity and we are now ready to deploy TKG without internet access and using our non-secure Harbor registry.

Step 14 - Unlike the previous article where Harbor uses a valid SSL Certificate and the registry is automatically trusted. Here we need to tell our K8s distribution about our insecure registry and this means we need to "inject" this information prior to the container images being pulled down. To do so, we need to edit the following two TKG plans and append to the containerd configuration starting with "files" section and everything below that. You will need to replace the IP Address and port with your Harbor deployment.

• .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-dev.yaml
• .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-prod.yaml

preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "127.0.0.1   localhost" >>/etc/hosts
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
    files:
      - path: /etc/containerd/config.toml
        content: |
          version = 2
          [plugins]
            [plugins."io.containerd.grpc.v1.cri"]
              sandbox_image = "registry.tkg.vmware.run/pause:3.1"
              [plugins."io.containerd.grpc.v1.cri".containerd]
                default_runtime_name = "runc"
                [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
                    runtime_type = "io.containerd.runc.v2"
                  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
                    runtime_type = "io.containerd.runc.v2"
                [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.2.10:80"]
                    endpoint = ["http://192.168.2.10:80"]

Step 15 - In addition, we also need to tell the KinD cluster about our insecure registry and that means we need to manually stand it up as we can not use the default "tkg init" command as-is. Replace just the IP Address and port with your Harbor instance and then run the following command which will create kind-config.yaml file which we will use in the next step.

cat > kind-config.yaml << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.2.10:80"]
    endpoint = ["http://192.168.2.10:80"]
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  imageRepository: registry.tkg.vmware.run
  etcd:
    local:
      imageRepository: registry.tkg.vmware.run
      imageTag: v3.4.3_vmware.4
  dns:
    type: CoreDNS
    imageRepository: registry.tkg.vmware.run
    imageTag: v1.6.5_vmware.4
EOF

Step 16 - Next, we will manually create the KinD cluster and specify the KinD Image which we have pulled down already and referencing our configuration file:

kind create cluster --image registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2 --config kind-config.yaml

The creation of the KinD cluster should be pretty fast since we the image already on our local system.

Step 17 - Lastly, we are now ready to setup TKG and we will need to specify the -e flag which will use an existing Kubernetes Cluster, this will be our KinD cluster and referencing the path to the K8s configuration as shown below.

tkg init --infrastructure=vsphere:v0.6.3 --name=vghetto-cluster -e --kubeconfig /root/.kube/config -v 6

As you can see from the steps above, this is definitely not straight forward and we are also deviating from the default TKG workflow as we are manually standing up the KinD cluster which is automatically setup for you the background when using the normal workflow. This may help some folks who may not be able to obtain a proper signed SSL certificate but I do not expect many to go down this path and for me it was less about setting up an insecure registry but rather what I had learned while going down this rabbit hole and the various components that work together to make up the TKG offering.