The tiny form factor of a Raspberry Pi (rPI) is a fantastic hardware platform to start playing with the ESXi-Arm Fling. You can already do a bunch of fun VMware things like running a lightweight vSAN Witness Node to setting up basic automation environment for PowerCLI, Terraform and Packer to running rPI OS as VM, enabling some neat use cases like consolidating your physical rPI assets which might be running RetroPi and Pi-Hole which many home labbers are doing.
In addition to VMware solutions, its is also a great platform to learn and tinker with new technologies like Kubernetes (K8s) which I am sure many of you have been hearing about 🙂 Although our vSphere with Tanzu and Tanzu Kubernetes Grid (TKG) does not currently work with the ESXi-Arm Fling, I have actually been meaning to try out a super lightweight K8s distribution designed for IoT/Edge called k3s (pronounced k-3-s) which also recently joined Cloud Native Computing Foundation (CNCF) Sandbox level.
k3s is supported on rPI and you normally would have multiple rPI devices to represent the number of nodes, for example if you want a basic 3-Node cluster, you would need three physical rPI devices. With ESXi-Arm, you can now create these nodes as VM, using just a single rPI. This opens up the door for all sorts of explorations, you can create HA cluster or try out more advanced features which might be more difficult if you needed several physical devices. If you mess up, you can simply re-deploy the VM without much pain or simply clone the VM.
In my setup, I am using 3 x Photon OS VMs. One for the primary node and two for k3s worker nodes. You can certainly install k3s on any other Arm-based OS including rPI OS (which can now run as a VM as mentioned earlier).
Step 1 - Download Photon OS 3.0 Arm ISO and create three Photon OS Virtual Machines on your ESXi-Arm host. If you need more details, please refer to the official ESXi-Arm PDF documentation on how to install GuestOS. In addition, I also disabled the default firewall, mainly as I always trip over it when debugging since ICMP is blocked by default. It looks like if you do not do this, there are issues setting up k3s.
systemctl stop iptables
systemctl disable iptables
Step 2 - On the first node, run the following command to install k3s as a primary (master) node:
curl -sfL https://get.k3s.io | sh -
Step 3 - After installing k3s, we need to retrieve the auth token so that when we deploy additional k3s worker nodes, it will automatically join our k3s cluster:
cat /var/lib/rancher/k3s/server/node-token
Make a note of the string which will be used in the next step.
Step 4 - Now login via SSH to second and third Photon OS node and replace token and the IP Address/Hostname of your first Photon OS and run the following command which will install k3s but instead of creating a standalone deployment, it will automatically join our existing k3s cluster.
NODE_TOKEN=[YOUR-TOKEN]
curl -sfL https://get.k3s.io | K3S_URL=https://photon-aarch64-01:6443 K3S_TOKEN=${NODE_TOKEN} sh -
Once the new k3s worker nodes have joined the cluster, from any node including the primary, you can run the following to verify k3s cluster is ready to start deploying workloads:
kubectl get nodes
I've tried the steps a few times but receive the following error from the first worker node after executing kubectl get nodes: "The connection to the server localhost:8080 was refused - did you specify the right host or port?" Do you have any quick ideas on what I may have screwed up? Thanks!
I'm getting the same error. Haven't had time to look into it yet.
While not the best fix, this will get it working:
iptables --list (Note: not really required, just wanted to see the rules)
systemctl stop iptables
systemctl disable iptables
Lab, it's fine. Will need more work if you want to keep the firewall on and create the right rules.
Thanks for sharing Steven. I actually recalled disabling iptables as ICMP is blocked by default and always gets me during troubleshooting. Let me update the blog post but haven't dug too deep on what rules would need to be opened up for k3s
Thanks William. Glad it helped. It's always the "network", "dns" or the "firewall" 😉 I'm totally new to k3s, but looking for to using it to learn some more Kubernetes.
Thank you Steve, took me 1-2 hours a day for 4-5 days just to get the VMs setup and running and when I got happy to have them all set up, I ran into the same issue you and Paul got, so I was glad to see your work around.
And William thanks for another great article
@Paul. I hear you. I went through quite a few setups of the VMs, with the standard tweaks I like to make before getting into the app load. After I did this a few times before I remember snapshots! d'oh! 🙂 Got my VMs to the setup, before starting the k3s install and snapshotted them as Vanilla.
Have fun with k3s. ESXi on ARM, Pi and K3s is such a great setup for learning.
It just occurred to me that the firewall might not have helped with the error we both saw:
"The connection to the server localhost:8080 was refused - did you specify the right host or port?"
when running "kubectl get nodes", or any kubectl cmd from a worker node.
This helped me solve this. Basically, looks like need to login to the cluster from the worker before running kubectl cmd:
https://opensource.com/article/20/3/kubernetes-raspberry-pi-k3s
Sorry, the section under "Access the cluster from our PC". Just do this on worker node. Ultimately you'll be wanting to do this stuff from a client anyway, so that helps with that too 🙂
Hi Steven / Paul - I walked through the other video noted above - I'm on a Mac just using terminal and SSH into each of the nodes - I am still getting the localhost:8080 was refused - just looking for a little direction - I too am learning, and I am not a linux guy at all - but my master still can not see the workers and vice versa
Hi William. I also ran into the error “The connection to the server localhost:8080 was refused – did you specify the right host or port?”. It almost drove me nuts. Why does it say "localhost"? After a while I found the root of the issue: I installed PhotoOS (v4beta) once and copied the vmdk into blank VM directories of node 2 and node 3. Then I adjusted IP address, machine-ID and hostname. BUT I forgot to adjust /etc/hosts There I still had 127.0.0.1 pointing to the original hostname of my master-node VM.Douh! 🙂
Thanks for the great post!