WilliamLam.com

Unable to power on vSphere Cluster Services (vCLS) VM in Nested ESXi with no host is compatible with the virtual machine

by // 9 Comments

After deploying a new VMware Cloud Foundation (VCF) Workload Domain using the VCF Holodeck Toolkit, which leverages Nested ESXi, I noticed the vSphere Cluster Services (vCLS) VMs kept failing to power on and threw the following error message:

No host is compatible with the virtual machine


I thought this was quite strange, especially since the vCLS VMs ran fine when the VCF Management Domain was setup.

UPDATE (07/03/2024) - The reason for the vCLS error is actually due to the miss-configuration of the Nested ESXi VM created by VCF Holodeck Toolkit, please see this blog post for an easier fix.

Looking at the vmware.log for the vCLS VM, I quickly found the issue where the VM expects to have the MWAIT CPU instruction exposed:

2024-03-19T16:35:35.736Z In(05)+ vmx - Power on failure messages: Feature 'cpuid.mwait' was 0, but must be 0x1.
2024-03-19T16:35:35.736Z In(05)+ vmx - Module 'FeatureCompatLate' power on failed.
2024-03-19T16:35:35.736Z In(05)+ vmx - Failed to start the virtual machine.

I figure I was probably not the first person to run into this and asked Ben Sier, who works on Holodeck and indeed he ran into this before. It looks like with newer vSphere releases, it expects to configure Per-VM EVC but the vCLS VM may not function properly within a Nested ESXI environment. Luckily, Ben has a workaround that we can quickly use.

[Read more...]

Dynamic ESXi firewall rulset for non-standard syslog ports in vSphere 8.0 Update 2b

by // 5 Comments

For most users who configure syslog for their ESXi hosts (hopefully everyone is doing that for audit, compliance and troubleshooting purposes), they typically stick with the default syslog ports 514 for UDP/TCP or 1514 for TLS.

A huge benefit of using the default syslog ports is that the ESXi firewall is already configured with these rulesets configured for outbound access.


If you require to use a non-standard syslog port for ESXi, the current solution was not ideal. While you can open up a custom port using the ESXi firewall, the issue is persisting that customization, which either requires a custom VIB or messing around with local.sh startup script.

A nice enhancement that is included with the recent release of vSphere 8.0 Update 2b is the support for a dynamic ESXi ruleset when non-standard syslog ports is configured.

As you can see in the example below when I configure my ESXi host to use a syslog server with a custom port 12345, the ESXi will automatically create a dynamic firewall ruleset that will open up that port for outbound connectivity. If you change the port or disable the syslog configuration, then the dynamic ruleset will be updated and/or removed.

Automated VMware Cloud Foundation (VCF) host commission using ESXi Kickstart

by // 1 Comment

ESXi Scripted Installation (Kickstart) has been my go-to method for achieving zero-touch provisioning of ESXi hosts at scale, which I had started using back in the ESX 2.5 days when I was a customer! Having worked at some very larger enterprises, I got the opportunity to experience and manage a variety of environments for automated ESXi provisioning.

For more than a decade, I have written hundreds of articles about ESXi kickstart and how it can help solve a variety of use cases stemming from my own background to some of the unique requirements that have come up from some of our largest VMware customers. To date, some of my favorite ESXi kickstart solutions includes my 2014 blog post in automating VM deployments using a USB device which became the basis for my USB to SDDC project in 2017.

While playing with the latest VMware Cloud Foundation (VCF) 5.1 Holodeck release (currently in Beta), I was thinking about the current VCF host commissioning workflow, which is a multi-step process after an ESXi host has been provisioned where you need to manually (or using automation) to add the hosts to SDDC Manager before they can be consumed for either expanding and/or deploying a new workload domain.

I thought, why could we not just skip this step all together and that was when I had the idea of just incorporating the VCF host commissioning workflow automatically as part of an ESXi Kickstart installation! 😀

[Read more...]