Automation

Automating kubectl-vsphere login for vSphere with Tanzu

11/12/2020 by William Lam 5 Comments

Before you can start deploying workloads to your vSphere with Tanzu Cluster, you need to first download the vSphere Plugin for Kubectl and then use that to login to your Supervisor Cluster which will generate a Kubernetes (K8s) context file that is stored in .kube/config

Here is an example of using the vSphere Plugin for Kubectl:

./kubectl-vsphere login --server=10.10.0.64 -u *protected email* --insecure-skip-tls-verify

For interactive sessions this is fine and upon successfully entering your password when prompted, you can switch to the correct K8s context to begin your workload deployment. For folks interested in automation, the one downside today is that the plugin does not provide a way to specify your password using either a command-line argument or reading from a configuration file.

I have actually seen this topic come up a few times both internally and externally for those wanting to automate the end to end deployment of a Tanzu Kubernetes Grid (TKG) Cluster and have gotten stuck on trying to figure a way around having to perform this required manual step.

Using Terraform to deploy a Tanzu Kubernetes Grid (TKG) Cluster in vSphere with Tanzu

11/10/2020 by William Lam 2 Comments

A few months back I saw that HashiCorp had released a new Kubernetes (K8s) Provider for Terraform, currently in Alpha state, which enable users to deploy K8s resources using the popular Infrastructure-as-Code (IaC) tool. I thought this would be pretty cool if it works with our vSphere with Tanzu solution, since the Tanzu Kubernetes Grid (TKG) Service uses ClusterAPI via a custom VM Operator to deploy TKG Guest Clusters which is just a fancy way of saying it uses K8s API to deploy more K8s 🙂

UPDATE (04/27/21) - vSphere 7.0 Update 2a has resolved the admission webhook issue and users can now deploy TKG Guest Cluster using K8s Provider for Terraform

The setting up the new K8s provider was pretty straight forward and after spending a few minutes in figuring out how to convert my existing TKG YAML to the required HCL format for Terraform to understand, I was able to to run a terraform "plan" but quickly ran into the following error:

failed: admission webhook "default.mutating.tanzukubernetescluster.run.tanzu.vmware.com" does not support dry run

It looks like our tanzukubernetescluster admission webhooks does not currently support dry run operations which can be quite useful but also common when using Terraform. I figured this was the end of that idea and I ended up just filing a feature enhancement internally for adding this support in the future as I can see this being quite useful for our customers.

After finishing up recent pet project of getting a fully functional vSphere with Tanzu on a homelab budget and just using 32GB of memory, I decided to take another look at this and discovered the required tweak to get this working was super trivial, literally a single line change.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Mapping between vSphere Container Volume to Persistent Volume Claim (PVC) in vSphere 7.0 Update 1 using PowerCLI

11/04/2020 by William Lam Leave a Comment

With the introduction of the vSphere Container Storage Interface (CSI) 2.1, it looks like the previous method outlined by Cormac Hogan no longer applies when looking to map between a vSphere Container Volume (CV), which is a vSphere construct to the underlying Persistent Volume Claim (PVC), which is a Kubernetes construct.

William Arroyo, a K8s Solution Engineer recently noticed this behavior change and was asking if there was a way to use PowerCLI to still perform this look up. Given I had provided the original PowerCLI snippet on Cormac's blog, I was curious myself and since I had just rebuilt my vSphere with Tanzu environment, I figured I take a quick look to see where this new information might now be placed.

I did also want to mention that you can easily find this information using the vSphere UI by just clicking on the "Details" box next to the PVC

With the latest PowerCLI 12.1 release, we have a number of Cloud Native Storage (CNS) cmdlets that we can leverage and after a quick minute of poking around, this new information can be found using the Get-CnsVolume cmdlet and using the ExtensionData property to get more detailed properties.

Stateless ESXi-Arm with Raspberry Pi

11/03/2020 by William Lam 18 Comments

I am super excited to be able to finally share, what I think, is a really cool ESXi-Arm solution which has been an evolution of this and this. This solution also incorporates a number of automation techniques I have shared over the years when it comes to ESXi scripted installation aka Kickstart, so it was really neat to all those things get pulled into a single solution. Lastly, I also want to give huge thanks to Cyprien Laplace who threw the initial challenge my way after I had shared how to perform an ESXi-Arm scripted installation without using SD Card.

ESXi-x86 can be deployed using either a stateful or stateless installation. In the latter case, ESXi is booted over the network using the vSphere Auto Deploy feature in vCenter Server which does not require any local media for ESXi. Upon attaching itself to vCenter Server, Auto Deploy then leverages vSphere Host Profiles and its rules engine to determine which configurations or profiles should be applied to ensure the ESXi hosts are configured per their desired stated. Here is a quick video overview of how Auto Deploy and Host Profiles work.

Fundamentally, vSphere Auto Deploy and Host Profiles can also work with ESXi-Arm but today, vCenter Server would require some code modification for this to actually work.

OK, so am I teasing you with something that does not exists? Nope, but I just wanted to help set the context 🙂

The solution that I have created boots ESXi-Arm over the network in a "stateless" manner, so there is no need for an SD Card or USB device plugged into the Raspberry Pi (rPI). In addition to the ESXi-Arm files, it also includes a custom payload which runs to retrieve additional configurations which can automatically join a desired vCenter Server as well as apply further customizations of an ESXi-Arm host. As you can see, this solution behaves similar to that of vSphere Auto Deploy and Host Profiles but does not use either of these vSphere features and works with the ESXi-Arm Fling right now.

Technically speaking, these techniques can also be applied to ESXi-x86 but I will leave that to the reader for further exploration.

Automating HAProxy VM deployment with 3-NIC configuration using PowerCLI

11/02/2020 by William Lam Leave a Comment

When deploying the HAProxy VM as part of vSphere with Tanzu, customers have the option of deploying the HAProxy VM using either a 2-NIC or 3-NIC configuration. The default OVF Deployment Option is the 2-NIC design called "Default" and the 3-NIC design is called "Frontend".

From an Automation point of view, you can use either OVFTool or PowerCLI to automate the deployment. For a 2-NIC example, you can refer to my Automated vSphere with Tanzu Lab Deployment Script. However, for the 3-NIC example, a few folks were running into some issues when using PowerCLI for the automation.

The main issue is that because the default OVF Deployment Option is the 2-NIC design (Default), the two additional OVF properties frontend_ip and frontend_gateway is basically hidden when processing the OVF properties when PowerCLI.

Note: You can view these optional properties by running the following OVFTool command: ovftool --X:enableHiddenProperties vmware-haproxy-v0.1.8.ova

Even if you specified the "Frontend" OVF Deployment Option, PowerCLI does not seem to have the logic to retrieve the other optional parameters and hence can not be set as part of the initial deployment.

Automated Nested Lab Deployment on SDDC Part 4: Oracle Cloud VMware Solution

10/30/2020 by William Lam Leave a Comment

In this final blog series, we are now going to take a look at deploying our Nested Lab in an Oracle Cloud VMware Solution environment. Similiar to GCVE, we also will have the ability to run Nested vSAN on top of the physical vSAN within OCVS and by default, customers are provided with the full administrator credentials. For networking, customers also have full access to the NSX-T Manager instance and this means that MAC Learning can be enabled and will allow inner-guest workloads will be able to communicate properly within and outside of the Nested Lab deployment.

Disclaimer: Nested ESXi is not officially supported on Oracle Cloud VMware Solution or by VMware.