If you are using an Intel 13th Generation (Raptor Lake) CPU and you have not disabled either the E-Cores or P-Cores, you may run into a PSOD when powering on a VM with the following exception:
GP Exception 13 in world
Since ESXi does not support Intel's hybrid consumer CPU architecture, Option 1 below is the current recommended approach that provides the best customer experience as ESXi will see one consistent set of cores, ideally the Performance Cores (P-Cores), as they include Hyper-threading (HT). If you can not or prefer not to disable the E-Cores or P-Cores, then Option 2 can be used with two additional workarounds.
Option 1 (Recommended) -
- Disable Efficiency Cores (E-Cores) within system BIOS
Option 2 -
- Add ESXi kernel boot option to disable the CPU uniformity check, please see this video HERE for the detailed instructions
- To workaround the PSOD mentioned in this blog post, you will need add the additional ESXi kernel setting after installing ESXi which ignores the MSR Faults and prevents the PSOD from happening by running the following:
esxcli system settings kernel set -s ignoreMsrFaults -v TRUE
Hi William, do you have insights if the ESXi scheduler would eventually support it in the future? I've noticed the latest 4th Generation Intel Xeon Scalable Processors have something called High Priority Cores and Low Priority Cores. It's not the same as little-big concept, but still there's some difference in the Cores and it might be nice to have the option eg on the VM config (prefer which type of core).
No, there are no plans. The upcoming Xeon will support either all P-Cores or E-Cores with same socket, so no big.LITTLE which has different CPU features which makes them non-uniform such as HT as example. For now, outside of Arm which uses big.LITTLE, this type of architecture doesn’t seem like it’ll make it into Datacenter which would influence the type of CPU we’d support since these hybrid CPUs are found only in consumer systems like Intel NUC which are not officially supported
Sad to hear as newer Intel consumers CPUs are becoming less suitable for home labs. And from the AMD world there are lot of relatively cheap NUC-sized Ryzen based systems, but their downside is unsupported NIC.
Hi William, seems (I cannot remember exactly if it was GP Exception 13) the mentioned PSOD happens on 12th gen CPUs as well with ESXi8.0u1(was OK on ESXi8.0b, maybe 8.0u1 adds this new checking?) during VM startup (with CPU uniformity check disabled)
I have a system with i7-12650H, hits into this problem, the option 2 workaround resolved the issue.
Hi William, it seems that when adding a host to a cluster, the clusteragent generates the GP#E13 PSOD. I will have a closer look later, but for now it makes my 3 node cluster a two node team :).
Niels
Hi William, is there a permanent solution to option 2? When I restart esxi and turn on a vm it shows the PSOD. I would then need to turn off esxi and turn it back on, press shift O and type ‘esxcli system settings kernel set -s ignoreMsrFaults -v TRUE‘ which fixes the problem temporarily. But this goes away again after a restart
Not sure I follow ... using ESXCLI does permanently set the value ... unless your system somehow isn't persisting the setting. Are you booting off of USB or HD/SSD? Alternatively, if you're using USB, you can manually add ignoreMsrFaults=True to the kernel boot line, which accomplishes the same behavior but again, using ESXCLI command will persist the setting, which is the whole point
Hi William, thanks for responding. Just troubleshooted further into this and it looks like the PSOD only happens when I start vcenter on the host. All other vms work as normal. The ESXCLI command above does solve the problem but if I was to reboot the host the settings would not persist. I am booting off SSD.
And just to confirm if I am doing this correctly, you are supposed to input the esxcli command (esxcli system settings kernel set -s ignoreMsrFaults -v TRUE) when the host first starts by pressing Shift + O or do you need to input this into some file?
Thanks for all you do!
Hi William, I picked up a very cheap i5 13500H Minis Forum mini pc to be a new ESXi host. It has not arrived yet but I just realised after looking at reviews that the BIOS is severely locked down with no options to disabled cores 🙁 Just curious if Option 2 will suffice going forward or will it be a struggle/ unstable mess? Thanks
That's strange they've locked down the BIOS ... that should be something end users should be able to control based on their needs. I'd ask if they can enable at least P/E-Cores and provide a BIOS update, I know other OEMs have done this for me in the past. If not, then you can try Option 2 but TBH, I've seen enough quirks when leaving P/E-Cores running, that I don't bother with Option 2, so I really don't know if its viable longer term
Thanks. I did ask them but they said their engineers would not be doing that. 🙁
Is your mini PC using Aptio BIOS? I bought a mini PC from SimplyNUC and I couldn't find any option to disable E-Cores in BIOS. I asked support and waiting for their engineering team to confirm if there is a way of changing this.
hi
i have a problem. when i add promt in ssh to kernel and check it with list -d everythings is ok but when i reset esxi all order were reset and i must use SHIFT+O to boot.
what should I do?
I'm still getting GP Exception 13 on 13th with all E cores disabled....
Getting MAD
What hardware platform is this? Have you tried Option 2 in the blog post for ignoring the MSR faults?
Hi William,
Sorry for jumping in - My new Asus NUC 14 (Intel Gen 14 CPU - Meteor lake) has just arrived. Even though I have disabled E cores in UEFI - ESXi 8.0U2 still refuses to install. its showing the PSOD message as in the top of this blog post. Do you have any clue why ?
I might found the problem myself! Gen14 Ultra 1255H has three types of cores!
# of Performance-cores
6
# of Efficient-cores
8
# of Low Power Efficient-cores
2
The last two i'm not able to disable !
Hello,
Just wanted to let you know the panic occurs sometimes even if E-cores are disabled and ignoreMsrFaults is set to TRUE on 13th gen intel and ESXi-8.0U3-24022510:
#GP Exception 13 in world XXXXX:swapob jd
Module(s) involved in panic: lumkernel Version Releasebui 1d-24022510]
cr0=0x80010031 cr2=0x6446b1e5e0 cr3=0x110928000 cr4=0x142768
FMS=06/ba/2 uCode=0x410c
This happened unexpectedly over the weekend, without touching the server. No stop/start VM activity.
I am digging around this but I don't have much hope of finding a stable solution.
Hello,
Great news! I've managed to overcome the errors and currently have an impressive 46 days of uptime.
I've disabled SpeedStep, SpeedShift, RTH, and C-states.
So far, everything's running smoothly!
Detailed instructions here: