I have been noticing more and more users that have acquired hardware that includes the latest Intel 12th Generation CPU (Alder Lake) and even the newest Intel 13th Generation CPU (Raptor Lake) for use with ESXi. Starting with the Intel 12th Generation CPU, Intel has introduced a new hybrid "big.LITTLE" CPU architecture that integrates two types of CPU cores: Performance-cores (P-cores) and Efficiency-cores (E-cores) into the same physical CPU die.
ESXi is currently not aware of this new consumer architecture and it currently expects all cores within a CPU package to have uniform characteristics. If you boot the ESXi installer, it will PSOD (Purple Screen of Death) by default and you will see a message about "Fatal CPU mismatch on feature" which is due to the different CPU properties across both the P-Cores and E-Cores. However, there is a way to workaround the issue by disabling the CPU uniformity check that ESXi performs as part of its boot up.
UPDATE (01/16/24) - See this blog post on some updated experiments using CPU affinity when both E-Cores and P-Cores are enabled when using ESXi.
UPDATE (04/22/23) - If you decide NOT to disable either E-Cores or P-Cores, you may also run into an additional PSOD when powering on a VM with GP exception in world message. To workaround this problem, please see this blog post HERE.
UPDATE (03/24/23) - It is possible and recommended to actually disable the E-cores within the Intel NUC BIOs following the instructions HERE to prevent ESXi from PSOD'ing due to non-uniform CPU cores rather than applying the ESXi boot option workaround as described in the video below.
I initially wrote about the solution back in Feb of 2022 where this new CPU was first introduced in the Intel NUC line with the Intel NUC 12 Extreme (Dragon Canyon) and subsequently, I had also wrote about the solution reviewing both the Intel NUC 12 Pro (Wall Street Canyon) and the Intel NUC 12 Enthusiast (Serpent Canyon).
While the majority of folks have not had any issues applying the workaround, I have started seeing some folks running into challenges, perhaps its familiarity with ESXi or applying kernel options. In any case, I figured it might help to record a video demonstrating the workaround for those that rather visualize the solution along with the written instructions (included below).
Video Instructions
Manual Instructions
Step 1 - During the bootup of the ESXi installer, you will see an option to to append additional ESXi kernel boot settings. Press SHIFT+O and append the following kernel option cpuUniformityHardCheckPanic=FALSE to the command line and the press enter to continue the boot process.
Step 2 - Install ESXi by following the install wizard and once you are prompted to reboot, do not reboot yet. We need to add the kernel option again so that ESXi can successfully boot after the installation. To do so, switch into the ESXi shell by pressing ALT+F1 and login using root and blank password as ESXi has not gone through full reboot and is not using the configured password.
Step 3 - Edit /vmfs/volumes/BOOTBANK1/boot.cfg and append following kernel option cpuUniformityHardCheckPanic=FALSE to existing kernelopt entry, which should look like the following:
kernelopt=weaselInstalled autoPartition=FALSE cpuUniformityHardCheckPanic=FALSE
Save your changes and then change back to the reboot prompt by pressing ALT+F2 and then reboot.
Instead of having to manually append the ESXi kernel option, you will notice that it has been appended due to Step2 and applying the setting in the boot.cfg file.
Step 3 - To permanently configure the ESXi kernel boot setting, in case of updates/upgrades in the future, we can set the kernel setting using ESXCLI. We first need to login to DCUI by pressing F2 and then navigating to Troubleshooting Option and enable ESXi Shell. Switch into the ESXi shell by pressing ALT+F1 and login with root and the password you had configured during installation. Now run the following ESXCLI command to configure the kernel option:
esxcli system settings kernel set -s cpuUniformityHardCheckPanic -v FALSE
Excellent post man! I noticed a lot of people having this problem and truth be told if you haven’t worked for Vmware color or you haven’t done a bunch of automation for Vmware… You might not be familiar with the OS options. You
I just bought Intel NUC 13 Pro I5 and with the steps in this article it was possible to install and stop the psod. Thanks.
Does this give you access to the e-cores as well and p-cores?
Yes
Very helpful. When dealing with keyboard shortcuts at different prompts, a video is a so much easier to follow. 🙂 Thank you!
How about the scheduler? Because esxi doesn’t aware of the cores, did you see any scheduler related performance issues or power consumption issues like assigning the tasks to wrong core type? In my opinion AMD is golden standard for me since 12th gen intels. Or does it see the core types as different numa nodes? Or are you able to choose core type to assign to the vm. For example little cores for light tasks like dns-dc-automation services etc. and p cores for personal vm , dbs, and more power hunger vms?
When I say ESXi does not understand the architecture, I'm referring to scheduler.
In my limited testing in deploying vCenter and other infrastructure including Tanzu Kubernetes Grid (TKG), I've not seen any noticeable difference in behavior. It runs snappier than previous generations (as I'd expect) but again, it may vary based on workloads. Most VMware-based homelabs, the resource that is usually stressed is memory and CPU is generally idle ... so may not make a difference but again, all workload dependent and there's no expected behavior given the ESXi scheduler doesn't see a difference and you could be schedule on either E or P cores. On some Intel systems, you can disable the E-Cores all together, but not something I've seen on the Intel NUCs
Thank you, thank you, thank you, thank you....it works 100%
Great video. It helped a lot.
Thanks
Thanks for your guide.
I think the P core hyper threading is not well supported yet on 12th gen.
My hardware is 1240p, which has 4P cores (total 8 threads in theory) and 8E cores (total 8 threads).
I can only see 12 cpus in ESXI8, and hyper threading can't be enabled.
HT is indeed disabled when the cores are non-uniform
Hi William thx for the blog it helped me a lot, i used the command esxcli system settings kernel set -s cpuUniformityHardCheckPanic -v FALSE with SSH and that works better for me after the reboot.
This workaround no longer works after Dell BIOS update!
Anybody have any idea when will VMware get serious and FIX this bug? (along with the 100s things that break so easily, and as a result require an actual developer to troubleshoot - no wonder professionals prefer to pay more for AWS.)
Probably it is not going to happen in the near future. Because VMware is driving the ESXi server and certified hardware exclusive more and more each day. Which is not so bad in my opinion. because intel's big little architecture is designed directly for consumer electronics not for Servers yet. The best and robust way to resolve this issue is disabling the e cores from bios and this will give you avx support back too or if you are building a system just choose the cpu which is not using the mixed core design. I am wondering if new amd 7900x3d and 7950x3d chips are going to be effected from this issue because their only one chiplet has 3dvcache. I belive 7800x3d will not be effected because it has one chiplet and all cores have same design.
same here, i applied the command but stills gives me the pink screen before it finishes, any other work around?
You need to re-watch the video and ensure you've followed the instructions carefully.
In this way, does it allow to use both E core and P cores at full load / efficiency?
You'll be able to use all the E and P Cores (no HT)
If you cannot enable HT on these new CPU’s, is the new CPU the best choice then? Doesn’t an older generation i5 or i7 with HT give you more performance?
You _can_ enable HT if you disable the E-Cores (assuming your BIOS allows for it). Here's how you do it for Intel NUC https://williamlam.com/2023/03/how-to-disable-the-efficiency-cores-e-cores-on-an-intel-nuc.html and ultimately it'll depend on your needs. While you can certainly look at older platforms, some may want to invest in modern systems that'll last for the next several years and with older systems, there's always a risk and also some of the consumer Intel NUC have a shorter support lifecycle. I know the latest Intel NUC 13 Pro now have 5yrs of support where it is typically 2-3 at best. If you want to use the last generation w/o hybrid architecture, look at 11th Gen NUCs
Glad I came back to read old posts and comments. I was wondering why I have HT disabled on my new little box. I found I am better served with 8 more e-cores, even if slower, than 4 HT p-cores. I'm really surprised and excited with the punch these little NUCs pack - jumping from an i5-3400 to an i7-1360p with double the memory, the difference is night and day. Glad I spent years exercising patience.
What I also found out is that the Watchdog setting, if enabled on the BIOS, makes the installation to fail. I was puzzled at first why it was stopping at TPM initialization (even though you said it was irrelevant to ESXi). Disabled it, then the installation began failing on the network interfaces prep-work. Ended getting the BIOS set to the very basic settings - one P core, absolutely everything disabled and went turning things on until they passed. With the time set on the watchdog, I got crashes again, which were indeed timely: from moving from the boot screen to loading something, the BIOS reset the system as per watchdog's request - most likely because it cannot interpret (or ESXi does not publish) the OS as alive.
William,
My CPU is a i9-13900k. I watched the video and got my ESXI 8.0 host up and running, thank you!. I've applied each patch as they've been released. Today I applied the ESXI 8.0 Update 1. The host still boots up fine, however now, when I try to power on a guest VM it crashes. I'd love to send you a screenshot if your interested. I'll give you the top two lines of the crash dump output on the screen. line1 = "#GP Exception 13 in world 2099894:vmx-vcpu-2:S @ 0x42000df74ec9" line2 = "Module(s) involved in panic: [vmkernel Version Releasebuild-21495797]
I open to any and all of your suggestions 🙂
I reviewed your updated article around these CPUs. I went ahead and disabled all the efficiency cores in the BIOS as suggested. Now ESXI shows 8 cores with 16 threads (HT enabled). I know I'm losing 16 efficiency cores, but now I can power on my VM guests after the updated. This is a prime example of why a XEON processor of the same performance costs $3000+ LOL.
Mark,
I just recalled there was a workaround after debugging issue w/Engr, so if you still want to enable all cores, then you will need one additional workaround which I have just published https://williamlam.com/2023/04/esxi-psod-due-to-gp-exception-13-in-world-with-intel-13th-generation-cpu.html
As mentioned already, this workaround shouldn't be needed in a future ESXi update and only the ESXi kernel option for ignoring the CPU uniformity in the future.
I was actually going to suggest disabling the E-Cores (https://williamlam.com/2023/03/how-to-disable-the-efficiency-cores-e-cores-on-an-intel-nuc.html) as this behavior looks to only occur with the 13th Gen CPUs. I've already reported this issue internally and a fix will be available in a future update of ESXi, but for now, disabling E-Core will be your best option.
so far i tested enable P core and Ecore in my ESXi 8 with my 13Gen i13900 CPU and install VM with windows 2022 and Centos 9 without issue. however seem like there is no way to enable hyperthreading, the Host summary show Hyperthreading = Yes, Disabled. i try to check the advance setting and seem Hyperthreading is enable.
Please see https://williamlam.com/2023/03/how-to-disable-the-efficiency-cores-e-cores-on-an-intel-nuc.html for your answer
Hi William, seem like disabled E-Core is the only way to be able to use HT, I m currently running disabled it. 8 P core with HT is 16. if with e-Core enable total will be 24, wonder which option will be better performance
any latest update on esxi supporting 12th Intel CPU P & E Cores
still need to disable E core wasted the E core ?
There are no plans as these CPU are only found in consumer grade platforms which VMware does not support. You can either disable as mentioned in the article or apply the workaround, pick the one that fits your needs
Quick question, performance wise, which way is better? disable E cores, or cpuUniformityHardCheckPanic ? Thanks,
Actually there is no single correct answer for this question.
- in total, keeping the ecores enabled gives you more threads and more performance points. but Unfortunately vmwarecpu scheduler doesn't aware oh the asymmetric design of the cores. if your CPU performance depenend application will be assigned to e core you will see the performance drop.
- On the other hand if you will sacrifice the ecore performance, then disable them. And you will have more consistent performance without having crush risks
hey,
I currently have an Intel NUC 13 ANHi7 and am currently trying to install VMware ESXi 8.0.1 21813344.
The problem with cpuUniformityHardCheckPanic=FALSE or disabling the efficient cores is not a problem but for me the installation gets stuck at the following point.
cndi_igc loaded successfully
activating: vmkdevmgr
I can't find a post here or on other sites to fix this and I keep reading that others have this problem too.
Is there already a solution for this?
I will try it now with version 7.x.
Hi, I have a HP elite mini G9 with an intel i7-12700.
I set cpuUniformityHardCheckPanic=FALSE and ignoreMsrFaults=TRUE.
Unfortunately I get a PSOD after done hours with following message:
Machine Check Exception on PCPU9 in world 2101451:vmast .210144
System has encountered a Hardware Error - Please contact the hardware vendor
Has anyone an idea?
Best regards
MCE typically point to hardware issue ... you could run diagnostic using the vendor utilities and update all the firmware/BIOS if you've done so
Hi William,
Thanks for your answer.
I can run the diagnostic tool.
But actually the machine is running since 2 days. I disabled the complete energy management for the cpu.
Is it possible, that the esx can’t handle the c-stats? Or is that a reason of the a hardware fault?
Best regards
ESXi can handle C-States fine https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-resource-management/GUID-4D1A6F4A-8C99-47C1-A8E6-EF3865603F5B.html, so I don't think its related to that ... its consumer hardware after all, so this shouldn't come as a surprise if something else is having issues, but MCE is typically accurate that its actual HW issue.
Hi, Unfortunately the problem occurred again. CPU stress test was successful. I have an old photo of the PSOD. Can you see more clearly what the problem is?
https://cloud.danigl.de/s/za8SgxaDmczF28y
Many thanks and best regards
Thank you so much for your tutorial, it helped me a lot. I am so glad that all works fine now.
The NUC 12 Pro (Intel® Core i7-1260P, NUC12WSHi7 ) is a great product and the Vmware ESXI8 is a great software.
I have a very small issue with the sound.
When i use the virtual machine installed on the esxi8, it don't have any sound on the client side.
The virtual machine seem okay but no sound is heard on the real PC.
(on client side)
It seems that there is no compatible sound driver NUC 12 Pro for the esxi8.
It's very frustrating.
Can you please help me ?
Where can we find a sound driver for ESXI8 for the NUC 12 Pro ?
Thanks for your great tutorial.
Hi William,
Thanks for your guides.
I have an HP Elite Mini 800 G9 i9 12th and when I try to install ESXi 7.x I get the PSOD (Purple Screen of Death) error. I tried entering the command cpuUniformityHardCheckPanic=FALSE while booting ESXi but without success.
It is not possible to disable the E cores in the PC BIOS and I cannot currently install the ESXi environment. Do you have any suggestions on this problem?
A thousand thanks
Best regards
Anthony
Are you sure the PSOD is due to non-uniform CPU cores ... often times, it can be other things like memory, storage, etc.
I also don't know what you mean by entering command and its not successful. This also leads me to believe, this may not be related and its something else causing PSOD 🙂
Hello William,
I certainly explained myself badly, when booting ESXi 7.0u3 from the USB stick, I get the error message: HW features incompatiblity detected..., restarting the boot and inserting the cpuUniformity.... string, once the loading is complete the PC reboots again and only after the second reboot by always adding the cpuUnifornity.... string the PC complete the loading.
Is it normal for the PC to reboot a second time after completing the first load?
Many thanks
Best regards
Antonio
Hello William,
Please tell me which keys will save the changes on edite boot option.
Thanks
Thank you so much you are a life saver
Dear William Lam
After installing VM8, I do not have access to the disks. Do you have any idea how I can resolve this issue? Thanks.
Thanks for preparing such document.
I have another problem, can you pls support if you faced similar issues?
I recently got a Lenovo m90q and has the SSD as "t10.NVMe____SKHynix_HFS001TEJ9X162N".
I installed ESXi 7.0.3 (many options Lenovo Custom and other subversions) and tried many diferrent Lenovo drivers. But still the I/O data rate doesnt exceed 1MB/s, I cant upload a file to datastore or create a VM.
I also changed many CPU features (enabled/disabled) on BIOS but nothing. Any idea what the reason can be?