WilliamLam.com

How to recover ESXi installed on USB device after disabling vmkusb module?

by // Leave a Comment

I have to say, this is one of the more interesting challenges that I have come across in quite some time. A user was looking for assistance after they accidentally disabled the vmkusb module, which is the USB driver for ESXi and allows it to communicate with USB devices that are connected to the system.

The vmkusb module also plays a very critical role if you have ESXi installed on a USB device, as the driver is required for proper functionality such as being able to save the ESXi state and configurations to the USB device. So what happens when you disable the vmkusb module and you reboot the ESXi host, which is also installed on a USB device?

Well, everything continues to work including VMs since ESXi by design runs in memory after the initial boot from the USB device. However, any configuration changes made after that is lost after a system reboot including the attempt to re-enable the vmkusb module since ESXi is unable save any of the settings to the USB device. Fortunately, I was able to help the user out as I had a few ideas on how we could fully recover from this type of scenario and hence the blog post.

Hopefully a lesson can be learned here, do not make changes or disable things that you are not familiar with 🙂

[Read more...]

Applying additional security hardening enhancements in ESXi 8.0

by // 16 Comments

While responding to a few ESXi security configuration questions, I was referencing our ESXi Security documentation, which includes a lot of useful information and latest best practices. It is definitely worth re-reviewing this section from time to time to take advantage of all the ESXi security enhancements to help protect and secure your vSphere environment.

In certain areas of the ESXi security documentation, I noticed that it mentions CLI and API, but it does not always provide an example that customers can then reference and use in their Automation, which is really the only guaranteed method to ensure configurations are consistent across your vSphere environment. After answering some of the security related questions, especially on the Automation examples, I figure it would be useful to share this information more broadly so that folks are aware of some of the new and existing security enhancements along with some of their implications if you are not implementing them.

Speaking of new ESXi security enhancements, one of the new features that was introduced in ESXi 8.0 is the ability to disable ESXi Shell access for non-root users. While this might sound like a pretty basic feature, applying this towards the vCenter Server service account vpxuser can help add another layer of protection for your ESXi hosts against attackers. It turns out that users with ESXi Shell access can also modify other local users password on ESXi host including the root user. By restricting ESXi Shell access for the vpxuser, you prevent attackers, which can also be insiders who have access to vCenter Server the ability to just change the ESXi root password without knowing the original password. As a result, this can lock you out of your ESXi hosts or worse, enable an attacker to encrypt your workloads, especially as the rise ransomeware attacks has been increasing.

[Read more...]

Video of ESXi install workaround for Fatal CPU mismatch on feature for Intel 12th Gen CPUs and newer

by // 56 Comments

I have been noticing more and more users that have acquired hardware that includes the latest Intel 12th Generation CPU (Alder Lake) and even the newest Intel 13th Generation CPU (Raptor Lake) for use with ESXi. Starting with the Intel 12th Generation CPU, Intel has introduced a new hybrid "big.LITTLE" CPU architecture that integrates two types of CPU cores: Performance-cores (P-cores) and Efficiency-cores (E-cores) into the same physical CPU die.

ESXi is currently not aware of this new consumer architecture and it currently expects all cores within a CPU package to have uniform characteristics. If you boot the ESXi installer, it will PSOD (Purple Screen of Death) by default and you will see a message about "Fatal CPU mismatch on feature" which is due to the different CPU properties across both the P-Cores and E-Cores. However, there is a way to workaround the issue by disabling the CPU uniformity check that ESXi performs as part of its boot up.

UPDATE (01/16/24) - See this blog post on some updated experiments using CPU affinity when both E-Cores and P-Cores are enabled when using ESXi.

UPDATE (04/22/23) - If you decide NOT to disable either E-Cores or P-Cores, you may also run into an additional PSOD when powering on a VM with GP exception in world message. To workaround this problem, please see this blog post HERE.

UPDATE (03/24/23) - It is possible and recommended to actually disable the E-cores within the Intel NUC BIOs following the instructions HERE to prevent ESXi from PSOD'ing due to non-uniform CPU cores rather than applying the ESXi boot option workaround as described in the video below.

I initially wrote about the solution back in Feb of 2022 where this new CPU was first introduced in the Intel NUC line with the Intel NUC 12 Extreme (Dragon Canyon) and subsequently, I had also wrote about the solution reviewing both the Intel NUC 12 Pro (Wall Street Canyon) and the Intel NUC 12 Enthusiast (Serpent Canyon).

While the majority of folks have not had any issues applying the workaround, I have started seeing some folks running into challenges, perhaps its familiarity with ESXi or applying kernel options. In any case, I figured it might help to record a video demonstrating the workaround for those that rather visualize the solution along with the written instructions (included below).

[Read more...]