WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / ESXi-Arm / Kubernetes on ESXi-Arm using k3s

Kubernetes on ESXi-Arm using k3s

10.16.2020 by William Lam // 11 Comments

The tiny form factor of a Raspberry Pi (rPI) is a fantastic hardware platform to start playing with the ESXi-Arm Fling. You can already do a bunch of fun VMware things like running a lightweight vSAN Witness Node to setting up basic automation environment for PowerCLI, Terraform and Packer to running rPI OS as VM, enabling some neat use cases like consolidating your physical rPI assets which might be running RetroPi and Pi-Hole which many home labbers are doing.

In addition to VMware solutions, its is also a great platform to learn and tinker with new technologies like Kubernetes (K8s) which I am sure many of you have been hearing about πŸ™‚ Although our vSphere with Tanzu and Tanzu Kubernetes Grid (TKG) does not currently work with the ESXi-Arm Fling, I have actually been meaning to try out a super lightweight K8s distribution designed for IoT/Edge called k3s (pronounced k-3-s) which also recently joined Cloud Native Computing Foundation (CNCF) Sandbox level.

k3s is supported on rPI and you normally would have multiple rPI devices to represent the number of nodes, for example if you want a basic 3-Node cluster, you would need three physical rPI devices. With ESXi-Arm, you can now create these nodes as VM, using just a single rPI. This opens up the door for all sorts of explorations, you can create HA cluster or try out more advanced features which might be more difficult if you needed several physical devices. If you mess up, you can simply re-deploy the VM without much pain or simply clone the VM.

In my setup, I am using 3 x Photon OS VMs. One for the primary node and two for k3s worker nodes. You can certainly install k3s on any other Arm-based OS including rPI OS (which can now run as a VM as mentioned earlier).



Step 1 - Download Photon OS 3.0 Arm ISO and create three Photon OS Virtual Machines on your ESXi-Arm host. If you need more details, please refer to the official ESXi-Arm PDF documentation on how to install GuestOS. In addition, I also disabled the default firewall, mainly as I always trip over it when debugging since ICMP is blocked by default. It looks like if you do not do this, there are issues setting up k3s.

systemctl stop iptables
systemctl disable iptables

Step 2 - On the first node, run the following command to install k3s as a primary (master) node:

curl -sfL https://get.k3s.io | sh -


Step 3 - After installing k3s, we need to retrieve the auth token so that when we deploy additional k3s worker nodes, it will automatically join our k3s cluster:

cat /var/lib/rancher/k3s/server/node-token

Make a note of the string which will be used in the next step.

Step 4 - Now login via SSH to second and third Photon OS node and replace token and the IP Address/Hostname of your first Photon OS and run the following command which will install k3s but instead of creating a standalone deployment, it will automatically join our existing k3s cluster.

NODE_TOKEN=[YOUR-TOKEN]
curl -sfL https://get.k3s.io | K3S_URL=https://photon-aarch64-01:6443 K3S_TOKEN=${NODE_TOKEN} sh -


Once the new k3s worker nodes have joined the cluster, from any node including the primary, you can run the following to verify k3s cluster is ready to start deploying workloads:

kubectl get nodes

More from my site

  • VEBA + Knative + k3s on ESXi-Arm
  • ESXi running in unexpected places ...
  • Cluster API BYOH Provider on Photon OS (Arm) with Tanzu Community Edition (TCE) and ESXi-Arm
  • Hybrid (x86 and Arm) Kubernetes clusters using Tanzu Community Edition (TCE) and ESXi-Arm
  • Stateless ESXi-Arm with Raspberry Pi

Categories // ESXi-Arm, Kubernetes Tags // Arm, esxi, k3s, Kubernetes

Comments

  1. Paul Sundquist says

    10/21/2020 at 9:24 am

    I've tried the steps a few times but receive the following error from the first worker node after executing kubectl get nodes: "The connection to the server localhost:8080 was refused - did you specify the right host or port?" Do you have any quick ideas on what I may have screwed up? Thanks!

    Reply
    • Steven Soave says

      10/22/2020 at 4:44 pm

      I'm getting the same error. Haven't had time to look into it yet.

      Reply
    • Steven Soave says

      10/23/2020 at 3:54 am

      While not the best fix, this will get it working:

      iptables --list (Note: not really required, just wanted to see the rules)
      systemctl stop iptables
      systemctl disable iptables

      Lab, it's fine. Will need more work if you want to keep the firewall on and create the right rules.

      Reply
      • William Lam says

        10/23/2020 at 4:31 am

        Thanks for sharing Steven. I actually recalled disabling iptables as ICMP is blocked by default and always gets me during troubleshooting. Let me update the blog post but haven't dug too deep on what rules would need to be opened up for k3s

        Reply
        • Steven Soave says

          10/26/2020 at 4:33 pm

          Thanks William. Glad it helped. It's always the "network", "dns" or the "firewall" πŸ˜‰ I'm totally new to k3s, but looking for to using it to learn some more Kubernetes.

          Reply
      • PAUL says

        10/23/2020 at 10:37 pm

        Thank you Steve, took me 1-2 hours a day for 4-5 days just to get the VMs setup and running and when I got happy to have them all set up, I ran into the same issue you and Paul got, so I was glad to see your work around.
        And William thanks for another great article

        Reply
        • Steven Soave says

          10/26/2020 at 4:38 pm

          @Paul. I hear you. I went through quite a few setups of the VMs, with the standard tweaks I like to make before getting into the app load. After I did this a few times before I remember snapshots! d'oh! πŸ™‚ Got my VMs to the setup, before starting the k3s install and snapshotted them as Vanilla.

          Have fun with k3s. ESXi on ARM, Pi and K3s is such a great setup for learning.

          Reply
    • Steven Soave says

      10/26/2020 at 4:45 pm

      It just occurred to me that the firewall might not have helped with the error we both saw:
      "The connection to the server localhost:8080 was refused - did you specify the right host or port?"
      when running "kubectl get nodes", or any kubectl cmd from a worker node.

      This helped me solve this. Basically, looks like need to login to the cluster from the worker before running kubectl cmd:

      https://opensource.com/article/20/3/kubernetes-raspberry-pi-k3s

      Reply
      • Steven Soave says

        10/26/2020 at 4:48 pm

        Sorry, the section under "Access the cluster from our PC". Just do this on worker node. Ultimately you'll be wanting to do this stuff from a client anyway, so that helps with that too πŸ™‚

        Reply
  2. Van says

    11/09/2020 at 8:42 am

    Hi Steven / Paul - I walked through the other video noted above - I'm on a Mac just using terminal and SSH into each of the nodes - I am still getting the localhost:8080 was refused - just looking for a little direction - I too am learning, and I am not a linux guy at all - but my master still can not see the workers and vice versa

    Reply
  3. Michael says

    11/27/2020 at 11:52 am

    Hi William. I also ran into the error β€œThe connection to the server localhost:8080 was refused – did you specify the right host or port?”. It almost drove me nuts. Why does it say "localhost"? After a while I found the root of the issue: I installed PhotoOS (v4beta) once and copied the vmdk into blank VM directories of node 2 and node 3. Then I adjusted IP address, machine-ID and hostname. BUT I forgot to adjust /etc/hosts There I still had 127.0.0.1 pointing to the original hostname of my master-node VM.Douh! πŸ™‚
    Thanks for the great post!

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Changing the default HTTP(s) Reverse Proxy Ports on ESXi 8.0 03/22/2023
  • Quick Tip - How to download ESXi ISO image for all releases including patch updates? 03/15/2023
  • SSD with multiple NVMe namespaces for VMware Homelab 03/14/2023
  • Is my vSphere Cluster managed by vSphere Lifecycle Manager (vLCM) as a Desired Image or Baseline? 03/10/2023
  • Interesting VMware Homelab Kits for 2023 03/08/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...