WilliamLam.com

Unable to power on vSphere Cluster Services (vCLS) VM in Nested ESXi with no host is compatible with the virtual machine

by // 8 Comments

After deploying a new VMware Cloud Foundation (VCF) Workload Domain using the VCF Holodeck Toolkit, which leverages Nested ESXi, I noticed the vSphere Cluster Services (vCLS) VMs kept failing to power on and threw the following error message:

No host is compatible with the virtual machine


I thought this was quite strange, especially since the vCLS VMs ran fine when the VCF Management Domain was setup.

UPDATE (07/03/2024) - The reason for the vCLS error is actually due to the miss-configuration of the Nested ESXi VM created by VCF Holodeck Toolkit, please see this blog post for an easier fix.

Looking at the vmware.log for the vCLS VM, I quickly found the issue where the VM expects to have the MWAIT CPU instruction exposed:

2024-03-19T16:35:35.736Z In(05)+ vmx - Power on failure messages: Feature 'cpuid.mwait' was 0, but must be 0x1.
2024-03-19T16:35:35.736Z In(05)+ vmx - Module 'FeatureCompatLate' power on failed.
2024-03-19T16:35:35.736Z In(05)+ vmx - Failed to start the virtual machine.

I figure I was probably not the first person to run into this and asked Ben Sier, who works on Holodeck and indeed he ran into this before. It looks like with newer vSphere releases, it expects to configure Per-VM EVC but the vCLS VM may not function properly within a Nested ESXI environment. Luckily, Ben has a workaround that we can quickly use.

[Read more...]

Dynamic ESXi firewall rulset for non-standard syslog ports in vSphere 8.0 Update 2b

by // 5 Comments

For most users who configure syslog for their ESXi hosts (hopefully everyone is doing that for audit, compliance and troubleshooting purposes), they typically stick with the default syslog ports 514 for UDP/TCP or 1514 for TLS.

A huge benefit of using the default syslog ports is that the ESXi firewall is already configured with these rulesets configured for outbound access.


If you require to use a non-standard syslog port for ESXi, the current solution was not ideal. While you can open up a custom port using the ESXi firewall, the issue is persisting that customization, which either requires a custom VIB or messing around with local.sh startup script.

A nice enhancement that is included with the recent release of vSphere 8.0 Update 2b is the support for a dynamic ESXi ruleset when non-standard syslog ports is configured.

As you can see in the example below when I configure my ESXi host to use a syslog server with a custom port 12345, the ESXi will automatically create a dynamic firewall ruleset that will open up that port for outbound connectivity. If you change the port or disable the syslog configuration, then the dynamic ruleset will be updated and/or removed.

Custom ESXi "Dummy" Reboot VIB for vSphere Lifecycle Manager (vLCM)

by // 2 Comments

A few weeks back, I had a request from one of our Technical Adoption Managers (TAM) that their customer wanted to create a custom ESXi VIB that could be used with vSphere Lifecycle Manager (vLCM) and would only require the ESXi host to reboot as part of the remediation.

This might sound like a strange request but I suspect the customer was either building out some automation for vLCM or simply getting more hands on with vLCM without applying any changes, which is great because its predecessor, vSphere Update Manager (VUM) will be removed in a future major release of vSphere.

While the customer was able to create a custom VIB by following the instructions in my recent blog post for building a custom VIB for ESXi 8.x, I did noticed that their descriptor.xml did not properly set the live-install-allowed and live-remove-allowed options which controls whether an ESXi host should reboot after installing and removing a VIB from the host respectively.


Since vLCM only works with offline bundles, we actually need to create an offline bundle with our custom ESXi VIB that vLCM can import. To further complicate things, starting with vSphere 7.x, a proper offline bundle that can be imported into vLCM requires the use of components rather than bulletins, which is what VUM previously had used.

With the assistance of the vLCM Engineering team, I was able to create my own "Dummy" ESXi VIB/Offline Bundle that is compatible with both vSphere 7.x and 8.x, which can be used directly by a standalone ESXi host via ESXCLI or imported and lifecycle using vLCM.

[Read more...]