WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / Kubernetes / How to deploy Tanzu Kubernetes Grid (TKG) Cluster with Antrea CNI 

How to deploy Tanzu Kubernetes Grid (TKG) Cluster with Antrea CNI 

04.20.2020 by William Lam // 1 Comment

I have been working with Tanzu Kubernetes Grid (TKG) quite a bit lately and using their new slick TKG CLI for deploying standalone Tanzu Kubernetes Clusters (TKC) which can run in both VMware Cloud on AWS as well as your on-premises vSphere 6.7 Update 3 environment. If you have vSphere 7 and the vSphere with Kubernetes capability, it also supports TKG deployments natively as part of that solution but you can also use TKG CLI to deploy TKC's.

Out of the box, TKG includes all the necessary software components to deploy a production grade, upstream and conformant Kubernetes distribution. For most customers, the "batteries included" type of offering is more than sufficient but for some customers who may wish to customize some of these components further when running the standalone distribution. One such example is swapping out the default Container Network Interface (CNI) which uses Calico for a different CNI with more capabilities.


As you may have guess from the title of this post, we will be replacing Calico with Antrea which is another open source CNI. In fact, Antrea was started by VMware last year and uses Open vSwitch (OVS) to provide network and security capabilities to Kubernetes. You can read more about Project Antrea here and more details about its architecture can be found here.

Disclaimer: This is currently not officially supported by VMware. I do know the TKG team is looking at Antrea support in the future.

Today, the process of changing out the CNI in TKG is a manual operation but in the future this could certainly be made simpler. When you install TKG CLI, there are two default plans: dev and prod which are encoded in two yaml files under ~/.tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template*.yaml

The section at the very bottom contains the Calico CNI definition and if we want to deploy another CNI such as Antrea, we need to replace everything after the "---" section with our CNI deployment specification.

---
apiVersion: v1
kind: Secret
metadata:
  name: ${CLUSTER_NAME}-postcreate
  namespace: ${NAMESPACE}
stringData:
  calicoYaml: |
    ---

This is certainly not something you want to be doing by hand and this is where some automation will help. Below are the commands that will automatically create two new plans called dev-antrea and prod-antrea which are clones of the existing plan and append the Antrea YAML so that you will be able to leverage the TKG CLI to deploy TKC running Antrea as the default CNI.

Step 1 - Run the following two command which will create to new YAML files and automatically delete everything after the "---" as mentioned earlier.

sed '/calico-config/,$d' .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-dev.yaml > .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-dev-antrea.yaml
sed '/calico-config/,$d' .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-prod.yaml > .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-prod-antrea.yaml

Step 2 - Run the following command to download the latest Antrea deployment specification. If the system does not have internet access, you can manually download this file

wget https://raw.githubusercontent.com/vmware-tanzu/antrea/master/build/yamls/antrea.yml

Step 3 - Run the following command which will take the contents of the Antrea deployment specification and append that to each of the our new deployment plans.

awk '{print "    ", $0}' antrea.yml >> .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-dev-antrea.yaml
awk '{print "    ", $0}' antrea.yml >> .tkg/providers/infrastructure-vsphere/v0.6.3/cluster-template-prod-antrea.yaml
rm antrea.yaml

We now have our two new plans and we can deploy a new TKC using TKG CLI with the following command:

tkg create cluster --plan=dev-antrea tkg-cluster-02

Once our new TKC is ready, we can list all the pods and we should now see Antrea as the default CNI:


After getting familiar with Antrea, I thought this would also be a good time to revisit an existing enhancement that we have been wanting to make to our vCenter Event Broker Appliance (VEBA) Fling which currently uses Weave for its CNI. As you can see from the screenshot below, I have already been able to get this working with VEBA, I am now just waiting for their official v0.6.0 release before we incorporate that into VEBA.

More from my site

  • Packer reference for VMware Harbor Virtual Appliance
  • How to deploy Knative to a Tanzu Kubernetes Grid (TKG) Cluster on both vSphere with Tanzu and TKG Multi-Cloud?
  • Using Terraform to deploy a Tanzu Kubernetes Grid (TKG) Cluster in vSphere with Tanzu 
  • Customizing Kubernetes cluster template (Dev/Prod) plans in Tanzu Kubernetes Grid 1.2
  • Tanzu Kubernetes Grid (TKG) Demo Appliance 1.1.3

Categories // Kubernetes, VMware Tanzu Tags // antrea, calico, CNI, Kubernetes, Tanzu Kubernetes Grid

Comments

  1. Vincent Chan says

    12/09/2022 at 10:15 pm

    Please provide above yaml file in Github for making the study easier! Thanks!

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • How to disable the Efficiency Cores (E-cores) on an Intel NUC? 03/24/2023
  • Changing the default HTTP(s) Reverse Proxy Ports on ESXi 8.0 03/22/2023
  • NFS Multi-Connections in vSphere 8.0 Update 1 03/20/2023
  • Quick Tip - How to download ESXi ISO image for all releases including patch updates? 03/15/2023
  • SSD with multiple NVMe namespaces for VMware Homelab 03/14/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...