WilliamLam.com

Quick Tip - Preserving FQDN hostname on Photon OS

by // 1 Comment

Over the weekend, I was troubleshooting an issue that was reported by one of our VMware Event Broker Appliance (VEBA) users who was helping with testing one of our upcoming features. The user found that after rebooting the VEBA appliance, the Antrea interfaces were no longer being re-created and pod networking seems to have been broken.

We initially thought it was related to switching to the latest Photon OS version or updating to the latest Antrea CNI release, since everything else was pretty much the same. Even after reverting both versions back to what we initially had, the reboot issue continued to persist. What was even more strange was that the current shipping version of the VEBA (v0.6.1) OVA was not experiencing this issue and had no problems with an OS reboot, which is something I have done many times.

The only logical conclusion that I could come up with to explain this problem is that a behavior change must have occurred within Photon OS from the time we built the previous appliance to what we are seeing now. While troubleshooting Antrea, it was pointed out that Kubernetes (K8s) node is probably unhealth and if so, I may want to look at the kubelet logs to see if it provided any hints. I initially did not both looking at the K8s layer, thinking this was related to change in Antrea since it handled pod networking. Looking at the kubelet logs, I found a ton of entries with the following:

396 kubelet.go:2243] node "veba" not found

I thought this was a bit strange, especially as our appliance has its hostname configurred with a Fully Qualified Domain Name (FQDN) which is veba.primp-industries.local and we had proper entries in both /etc/hostname and /etc/hosts.

Sure enough, when I ran hostname, they all returned the short hostname instead of the FQDN (which it returned properly prior to the reboot)

[Read more...]

ESXi on Intel NUC 11 Extreme (Beast Canyon)

by // 17 Comments


The NUC 11 Extreme (codenamed Beast Canyon) is the latest in Intel's Tiger Lake based NUC lineup which includes the NUC 11 Performance (Panther Canyon), NUC 11 Enthusiast (Phantom Canyon) and NUC 11 Pro (Tiger Canyon). As you can see from the picture above, the codename for the NUC 11 Extreme is quite fitting as this is currently the largest "NUC" that Intel has built to date, coming in at 8L. Yes, this is definitely "stretching" the NUC label in terms of what folks historically expect but I believe Intel is simply expanding on their well known NUC brand, especially as there are also NUC laptops.

However, this is also not the first time Intel has explored a larger NUC design. In 2020, Intel introduced the NUC 9 Pro (Quartz Canyon) and Extreme (Ghost Canyon) which took advantage of the new NUC Compute Element and enabled a new modular design and form factor adding support for discrete GPU and PCIe expandability. As a successor to the NUC 9, the NUC 11 Extreme extends this concept further by adding support for a full length discrete GPU, which is the primary driver for the larger form factor.

The NUC 11 Extreme can also enhance your homelab experience with LED lights which are located underneath the chassis and on the front with the classic Intel NUC Skull. Even cooler, the design in the front is customizable and can be user replaced with a different graphic. For an example of what this could look like, jump down to the customizable logo section of this blog post 🙂

I know there are a number of folks in the VMware community who are currently using the NUC 9 for their VMware Homelab, especially for those with GPU and/or additional network and storage requirements. Let's now take a closer look at what the NUC 11 Extreme has to offer the VMware Community.

[Read more...]

Quick Tip - Setting up Kubernetes using Containerd on Photon OS

by // 1 Comment

As part of the VMware Event Broker Appliance (VEBA) project, I was recently evaluating a newer version of Kubernetes (v1.21.3) and also switching the container runtime from Docker to Containerd. I figured this probably should not be that difficult, especially since we are already use Containerd within Tanzu Kubernetes Grid (TKG) which is our commercial Kubernetes (k8s) offering and that base OS is VMware Photon OS. How hard could this be, right!? (famous last words) 😂

We use kubeadm to setup K8s and read in a very basic configuration file and after following the official K8s instructions for prepping the environment to use containerd, I was surprised when I ran into the following error:

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.

Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint /run/containerd/containerd.sock logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

Unfortunately, this lead me down a huge rat hole of troubleshooting and trying various configurations and suggestions from the Internet. Ultimately, none of the suggested solutions solved my problem. After exhausting all my options and spending more time that I would like to admit, I decided to ask in the Kubernetes Slack community to see if anyone might have an idea. There were not any specific suggestions that helped me understand the issue further but there was a question about how Containerd came to be on the system and that gave me one more thing to try.

Both Photon OS 3.0 and 4.0 ships with Containerd and after installing the desired kubeadm, kubectl and kubelet, I had wrongfully assumed that the version of Containerd would simply work.

[Read more...]