WilliamLam.com

Quick Tip - How to deploy NSX Advanced Load Balancer (NSX-ALB) with a single Service Engine

by // 1 Comment

I saw an interesting question today from Robert Kloosterhuis in the private vExpert App Modernization Slack Channel who working with vSphere with Tanzu using NSX Advanced Load Balancer (NSX-ALB) and wanted to know if it was possible to deploy NSX-ALB with just a single Service Engine (SE)?

The default behavior of NSX-ALB is to deploy two SE for availability purpose but for testing and/or homelab usage, it could certainly help with resources and time to spin up an environment using NSX-ALB. I was also curious if this was possible and reached out to NSX-ALB Engineering team and within a few minutes, I got a response that not only was this possible to do but pretty easy to configure.

To modify this default behavior, we need to update the Service Engine group prior to SE VMs being deployed. To do so, login to NSX-ALB UI and under Infrastructure->Service Engine Group and then click on the Advanced tab and change the default Buffer Service Engines value of 1 to 0 which will will have NSX-ALB deploy just a single SE VM rather than the default two.


To confirm that our NSX-ALB have been configured correctly, I have enabled vSphere with Tanzu using NSX-ALB and as you can see from the screenshot below, only a single SE VM has been deployed rather than the default behavior of two SE.

Quick Tip - Setting up Kubernetes using Containerd on Photon OS

by // 1 Comment

As part of the VMware Event Broker Appliance (VEBA) project, I was recently evaluating a newer version of Kubernetes (v1.21.3) and also switching the container runtime from Docker to Containerd. I figured this probably should not be that difficult, especially since we are already use Containerd within Tanzu Kubernetes Grid (TKG) which is our commercial Kubernetes (k8s) offering and that base OS is VMware Photon OS. How hard could this be, right!? (famous last words) 😂

We use kubeadm to setup K8s and read in a very basic configuration file and after following the official K8s instructions for prepping the environment to use containerd, I was surprised when I ran into the following error:

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.

Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

Unfortunately, this lead me down a huge rat hole of troubleshooting and trying various configurations and suggestions from the Internet. Ultimately, none of the suggested solutions solved my problem. After exhausting all my options and spending more time that I would like to admit, I decided to ask in the Kubernetes Slack community to see if anyone might have an idea. There were not any specific suggestions that helped me understand the issue further but there was a question about how Containerd came to be on the system and that gave me one more thing to try.

Both Photon OS 3.0 and 4.0 ships with Containerd and after installing the desired kubeadm, kubectl and kubelet, I had wrongfully assumed that the version of Containerd would simply work.

[Read more...]

Exploration of Tanzu Kubernetes Grid (TKG) multi-vCenter Server templating using YTT

by // Leave a Comment

The motivation behind this blog post originates from a really cool blog post by Mike Brown who shared an interesting Telco use case for wanting to running Tanzu Kubernetes Grid (TKG) on VMware Cloud on AWS (VMConAWS) and centrally managing TKG Workload Clusters, which would run at each individual Edge/Cell Site location.

While reading through Mike's blog post, I noticed one of the steps was to edit the generated YAML from the TKG Management Cluster which would then be used to deploy the individual TKG Workload Clusters. This would need to happen for each new deployment 😮 and of course, this could be very error prone and frustrating for end users. Here is an example of what the YAML file looks like which is over 1K+ lines!

This screams for automation and I had been looking for a reason to try out YTT again, which is a YAML templating tool that is part of the open source project Carvel. Although I had played with YTT before, it did not feel intuitive, especially for a new user who was trying to solve a quick problem. I figured this was my opportunity to take another look at YTT.

After a couple of hours and a lot of trial/error, I ended up with a partial solution and realized that I would not be able to figure this out given there were even more complicated sections within the YAML. I felt the bar to getting started with YTT was still too high and it may not be the right tool for this particular situation. I opted for a quicker solution using sed, which I had experience with before, but I also know that depending on the problem, sed can be just as complex and I also dislike regular expressions  🙂

[Read more...]