WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / Automation / Quick Tip - Preserving FQDN hostname on Photon OS

Quick Tip - Preserving FQDN hostname on Photon OS

08.02.2021 by William Lam // Leave a Comment

Over the weekend, I was troubleshooting an issue that was reported by one of our VMware Event Broker Appliance (VEBA) users who was helping with testing one of our upcoming features. The user found that after rebooting the VEBA appliance, the Antrea interfaces were no longer being re-created and pod networking seems to have been broken.

We initially thought it was related to switching to the latest Photon OS version or updating to the latest Antrea CNI release, since everything else was pretty much the same. Even after reverting both versions back to what we initially had, the reboot issue continued to persist. What was even more strange was that the current shipping version of the VEBA (v0.6.1) OVA was not experiencing this issue and had no problems with an OS reboot, which is something I have done many times.

The only logical conclusion that I could come up with to explain this problem is that a behavior change must have occurred within Photon OS from the time we built the previous appliance to what we are seeing now. While troubleshooting Antrea, it was pointed out that Kubernetes (K8s) node is probably unhealth and if so, I may want to look at the kubelet logs to see if it provided any hints. I initially did not both looking at the K8s layer, thinking this was related to change in Antrea since it handled pod networking. Looking at the kubelet logs, I found a ton of entries with the following:

396 kubelet.go:2243] node "veba" not found

I thought this was a bit strange, especially as our appliance has its hostname configurred with a Fully Qualified Domain Name (FQDN) which is veba.primp-industries.local and we had proper entries in both /etc/hostname and /etc/hosts.

Sure enough, when I ran hostname, they all returned the short hostname instead of the FQDN (which it returned properly prior to the reboot)

# hostname
veba 

# hostname -s 
veba 

# hostname -f 
veba

We use hostnamectl to change the hostname, so I manually ran that and then restarted kubelet and of course, everything started to function properly again!

While doing an online search to figure out why this behavior was happening, I came across this thread and it turns out this is some how related to cloud-init resetting the hostname settings. There were a couple of solutions in the post, but I went with the quickest one which was simply touching a file called /etc/cloud/cloud-init.disabled and after rebooting, the hostname was preserved. What was even more baffling is that we did not use cloud-init to setup Photon OS, it is built using Packer and leveraging its Kickstart automation and its possible, that maybe cloud-init is used under the hood or maybe the service was some how enabled in newer updates.

In any case, after configuring your hostname, make sure to create this file and you will not run into this problem.

More from my site

  • Quick Tip - Prepare VMware Photon OS for use with vSphere Guest OS Customization and cloud-init
  • Quick Tip - Setting up Kubernetes using Containerd on Photon OS
  • CIM monitoring caveat with ESXi
  • Thunderbolt 3 enclosures with (Single, Dual & Quad) M.2 NVMe SSDs for ESXi
  • Maximum number of vCenter Servers per Single Sign-On (SSO) Domain

Categories // Automation Tags // hostnamectl, Photon OS

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • vSphere with Tanzu using Intel Arc GPU 01/26/2023
  • Quick Tip - Automating allowed and not allowed Datastores for use with vSphere Cluster Services (vCLS) 01/25/2023
  • ESXi with Intel Arc 750 / 770 GPU 01/24/2023
  • How to bootstrap vSAN Express Storage Architecture (ESA) on unsupported hardware? 01/19/2023
  • Automating Virtual Machine screenshots in vSphere 01/18/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023