Deploy Harbor in an Air-Gapped environment for Tanzu Kubernetes Grid (TKG)

When using Tanzu Kubernetes Grid (TKG) and the new TKG CLI, outbound internet connectivity is required as part of the initial setup on the machine running TKG CLI but also on the TKG Management Cluster which is automatically stood up as part of the deployment. For demo and testing purposes, this is usually not a problem but for anyone looking to run this in a Production or datacenter environment, direct internet access is generally not available.

TKG does support air-gapped environments today by requiring a private container registry that has been configured with all the required containers. Once your registry has been setup, you will also need to update the TKG YAML manifest files to specify your private registry as by default, it will point to registry.tkg.vmware.run. You can use any container registry that is supported with Kubernetes including the popular Harbor solution. One thing to note is that your private registry must have a proper signed SSL certificate, custom CA certificates or self-signed certificates are not officially supported today with TKG.

Since I recently had to set this up for a project I am working on, which I hope to talk about in a future blog post, I thought it would be useful to share the instructions on how to setup and configure Harbor to be used in-conjunction with TKG as well as any other solution that requires a container registry running in your own environment. In my deployment, I will be using Let's Encrypt for generating the required SSL certificate, but you can use any existing service for performing this operation. I will also be installing Harbor on Photon OS, but you can use any operating system of your choice that Harbor is supported on.

Pre-Requisites 

• Access to a public DNS domain which you have ownership of (e.g. adding new records)
• Access to your internal DNS server to add a custom DNS zone lookup entry (e.g. registry.<yourdomain>.com)

Step 1 - Decide on a hostname (FQDN) that you will use for your registry based off of your DNS domain. For example, I own virtuallyghetto.com and will be using registry.virtuallyghetto.com in this example. Once you have decided on the hostname, you will need to update your internal DNS to add a custom DNS zone lookup to map this entry to the IP Address you will use to deploy the Photon OS running within your datacenter. In my example, registry.virtuallyghetto.com will point to 192.168.2.2.

Step 2 (optional) - If you are using LE, please download and install certbot to your local desktop and you can refer to the Let's Encrypt (LE) instructions here

Step 3 (optional) - You will need access to your DNS domain management site to add a TXT for DNS challenge when requesting a new SSL certificate. Run the following command and replace with your email address, the name of the domain that you wish to register (e.g. registry.<yourdomain>.com) and directory where you want LE to store all its output.

certbot certonly --manual --preferred-challenges=dns --email [EMAIL] --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d "registry.virtuallyghetto.com" --work-dir /Users/lamw --config-dir /Users/lamw --logs-dir /Users/lamw

You will follow the instructions in the console to verify that you own the specific domain and once you have successfully verified the TXT entry, you should find both a fullchain.pem and privkey.pem found in live/<yourdomain.com> directory. In my example, here is the full path:

• /Users/lamw/live/virtuallyghetto.com/fullchain.pem
• /Users/lamw/live/virtuallyghetto.com/privkey.pem

Step 4 - Install Photon OS into a VM that will be used to run your Harbor instance. Internet connectivity is required to initially download the required containers, but it possible to import from another system which has internet access. You can search online for instructions on how to do that. Once Photon OS has been installed, please run the following two commands to update the OS and also install Perl which will be used in a subsequent step.

tdnf -y update
tdnf -y install perl

Step 5 - Enable and start the docker client:

systemctl enable docker
systemctl start docker

Step 6 - We need to configure the following firewall rule to allow connectivity from the KinD (Kubernetes in Docker) cluster which is deployed in the background as part of the initial TKG bootstrap process in setting up our TKG Management Cluster.

iptables -A INPUT -i docker0 -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save

Step 7 - Create the following directory structure and SCP the two LE certificate files that we had generated in Step 2 and place it in the folder:

mkdir -p /etc/letsencrypt/live/registry.<yourdomain>.com

Step 8 - Download and install Docker Compose which is required to run Harbor:

curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Step 9 - Download and install Kubectl which is required by TKG CLI:

curl -L https://storage.googleapis.com/kubernetes-release/release/v1.18.1/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
chmod +x /usr/local/bin/kubectl

Step 10 - Download and extract the Harbor Offline Installer:

curl -L https://github.com/goharbor/harbor/releases/download/v1.10.2/harbor-offline-installer-v1.10.2.tgz -o harbor-offline-installer-v1.10.2.tgz
tar xvzf harbor-offline-installer*.tgz
rm -f harbor-offline-installer-v1.10.2.tgz

Step 11 - Change into harbor directory and edit the harbor.yml configuration file with the respective values of your environment and then save the changes and exit.

Step 12 - Run the following command to start the Harbor installation:

./install.sh

Step 13 - Once the installation has successfully completed, we should verify that we can login to our registry by running the following command specifying the admin password you had set during the Harbor installation:

docker login -u admin -p VMware1! registry.virtuallyghetto.com/library

We can also login to the Harbor UI by opening browser:

Step 14 - After we have verified that we can login to our registry, go ahead and download the following mirror_tkg_containers.sh shell script which will automatically download, tag and push the required containers needed by TKG into our Harbor registry. In addition, the script will also update all the respective TKG YAML manifest files to replace the default VMware registry with our Harbor registry.

You will need to edit the script before running it and at the top you will need to update the registry URL which will just be the hostname of Harbor deployment.

./mirror_tkg_containers.sh

Step 15 - Once the script has completed, all required containers will now be available in Harbor. We can now disconnect or disable internet connectivity and we are now ready to deploy TKG without requiring direct internet access. In addition to having both the TKG Management and Workload Cluster properly standup, we can also see the "Pull" request increment as new deployments reference our registry.

Lastly, I want to give a big shoutout to both Fabio Rapposelli and Timo Sugliani who were instrumental in helping me with various issues while I was banging my head on a variation of this configuration which hopefully I will be able to talk about in the very near future 🙂