A few months back I saw that HashiCorp had released a new Kubernetes (K8s) Provider for Terraform, currently in Alpha state, which enable users to deploy K8s resources using the popular Infrastructure-as-Code (IaC) tool. I thought this would be pretty cool if it works with our vSphere with Tanzu solution, since the Tanzu Kubernetes Grid (TKG) Service uses ClusterAPI via a custom VM Operator to deploy TKG Guest Clusters which is just a fancy way of saying it uses K8s API to deploy more K8s 🙂
UPDATE (04/27/21) - vSphere 7.0 Update 2a has resolved the admission webhook issue and users can now deploy TKG Guest Cluster using K8s Provider for Terraform
The setting up the new K8s provider was pretty straight forward and after spending a few minutes in figuring out how to convert my existing TKG YAML to the required HCL format for Terraform to understand, I was able to to run a terraform "plan" but quickly ran into the following error:
failed: admission webhook "default.mutating.tanzukubernetescluster.run.tanzu.vmware.com" does not support dry run
It looks like our tanzukubernetescluster admission webhooks does not currently support dry run operations which can be quite useful but also common when using Terraform. I figured this was the end of that idea and I ended up just filing a feature enhancement internally for adding this support in the future as I can see this being quite useful for our customers.
After finishing up recent pet project of getting a fully functional vSphere with Tanzu on a homelab budget and just using 32GB of memory, I decided to take another look at this and discovered the required tweak to get this working was super trivial, literally a single line change.
Disclaimer: This is not officially supported by VMware, use at your own risk.
Step 1 - SSH to the VCSA and then run the following script to retrieve the Supervisor Cluster Control Plane VM credentials:
/usr/lib/vmware-wcp/decryptK8Pwd.py
Step 2 - SSH to the IP Address using root username and the password provided from the previous command
Step 3 - Run the following command to edit the TKG Guest Cluster validation webhook. Locate all sideEffects entries with a value of Unknown and replace that with None. Once you have finished, you can exit the editor and it should state that changes were saved successfully.
kubectl edit validatingwebhookconfigurations vmware-system-tkg-validating-webhook-configuration
Step 4 - Install tfk8s on your local desktop which you will need to have golang installed to build tfk8s from source as a binary is not available
Step 5 - Download the Terraform provider for K8s plugin for your respective operating system. Currently, only v0.1.0 includes pre-built plugin, the latest requires that you manually build from source. The plugin should be copied into your home directory under .terraform.d/plugins (e.g. /Users/lamw/.terraform.d/plugins) If the directory does not exists, go ahead and create it and then set the permission on the plugin to executable
chmod +x terraform-provider-kubernetes-alpha
Step 6 - Next, we need to convert an existing TKG YAML file to Terraform HCL manifest file. In this example, we will use tkc.yaml which you can download to your desktop. Run the following command and replacing the location of your tfk8s binary and the path to the tkc.yaml file:
mkdir tf-tkg-deployment
~/gocode/bin/tfk8s -f tkc.yaml -p kubernetes-alpha > tf-tkg-deployment/tkc.tf
Once you have the converted HCL file, you will need to append the following to the top of the file which tells Terraform where and how to authenticate into our TKG Cluster.
provider "kubernetes-alpha" {
config_path = "~/.kube/config" // path to kubeconfig
}
Here is what the final tkc.tf should look it, in case you just want a working copy:
provider "kubernetes-alpha" { config_path = "~/.kube/config" // path to kubeconfig } resource "kubernetes_manifest" "tanzukubernetescluster_william_tkc_01" { provider = kubernetes-alpha manifest = { "apiVersion" = "run.tanzu.vmware.com/v1alpha1" "kind" = "TanzuKubernetesCluster" "metadata" = { "name" = "william-tkc-01" "namespace" = "primp-industries" } "spec" = { "distribution" = { "version" = "v1.17.8+vmware.1-tkg.1.5417466" } "settings" = { "network" = { "cni" = { "name" = "antrea" } "pods" = { "cidrBlocks" = [ "193.0.2.0/16", ] } "serviceDomain" = "managedcluster.local" "services" = { "cidrBlocks" = [ "195.51.100.0/12", ] } } } "topology" = { "controlPlane" = { "class" = "best-effort-xsmall" "count" = 1 "storageClass" = "vsan-default-storage-policy" } "workers" = { "class" = "best-effort-xsmall" "count" = 1 "storageClass" = "vsan-default-storage-policy" } } } } }
Step 7 - Login to Supervisor Control Plane using the kubectl-vsphere plugin and switch context to your vSphere Namespace which will update the .kube/config file for Terraform to authenticate in and create a new TKG Guest Cluster.
./kubectl-vsphere login --server=10.10.0.64 -u *protected email* --insecure-skip-tls-verify
./kubectl config use-context primp-industries
Step 8 - Change into our tf-tkg-deployment folder containing our Terraform manifest file and run the following commands to init terraform and verify that you can connect by using the "plan" operation.
cd tf-tkg-deployment
terraform init
terrform plan
Step 9 - After the validation passes with the plan operation, you are now ready to deploy a TKG Guest Cluster using Terraform by running the following command:
terraform apply
You should start seeing the TKG Guest Cluster deploy shortly in the vSphere UI and you can now use Terraform to also manage your TKG Guest Cluster deployment which is quite nice for those who have standardized and/or operationalized against using Terraform. Although the K8s Provider for Terraform is still currently in Alpha development, the possibilities are certainly endless and curious to see if others would like to see such support from VMware?
Dennis Faucher says
Magic. Thanks for all the hard work and documentation.
Eric D. says
Thanks for the insights William. Went through most of the setup and found that I am still getting the dry run error despite changing the values you mentioned.
Cyruslab says
Hi, just want to know what is the minimum edition of Vmware vsphere in order for terraform to orchestrate vSphere? would Essential kit, essential plus kit, standard edition or enterprise plus edition be able to work with Terraform?
Ayush says
Hi, Is there a way to automate tkgm workload creation in aws and azure with terraform. Thanks