coredump

I was recently working with Engineering to reproduce an issue which causes an ESXi PSOD (Purple Screen of Death) and I wanted the generated ESXi coredump to simply write to the USB device, which I could easily grab.

As of ESXi 7.x, I know we had removed a few of the old ESXi kernel boot options for allowing ESXi to store coredumps on a USB device and the using the ESXi kernel boot option allowCoreDumpOnUsb=TRUE should now be used, however I was struggling to get it to work.

Since I was using a debug version of ESXi, I needed to install ESXi from scratch and I thought I could simpply add the required kernel option, as shown in the screenshot below, and I had assumed it would automatically configure the ESXi coredump file to be stored on the VMFS-L volume residing on the USB device.

After a couple of attempts, I finally realized that this particular ESXi kernel boot option, is literally that, a boot option that is only applicable after the initial ESXi installation. 🤦 Unlike other ESXi kernel boot options which can be used during the initial installation which would apply certain configuration changes, this setting applies after ESXi has been installed. Once I appended the setting, the ESXi coredump file was created in the VMFS-L volume and I was then able to reproduce the issue and generate vm-support bundle that included the coredump!

A couple of days back I had to re-install ESXi on a physical host for some troubleshooting purposes and while looking at the partitions on the disks using ESXCLI, I noticed the fresh ESXi installation had created two coredump partitions.

I was quite surprised to see two, since normally you would just have one configured. I even asked a colleague if he had ever see this before and he had not, so I wanted to double check that there was in fact two coredump partitions being created which I verified by using partedUtil.

As you can see from the screenshot above, there are definitely two coredump partitions. I took a look at our vSphere documentation, but did not find any mention of this. I decided to look internally and found that this is actually a new behavior that was introduced in ESXi 5.5. From what I can tell, the second coredump partition which is 2.5GB was created to ensure that there was sufficient space to handle ESXi hosts configured with a huge amount of memory (up to 4TB) if a coredump were to occur. This new coredump partition is only created on a fresh ESXi install, for upgrade scenarios the original partition structure is preserved. I suspect even on the fresh install, the original coredump partition was kept for potential backwards compatibility.

This definitely made sense given the reason. I guess this actually raises another interesting point from an operational point of view that though upgrades may be preferred, there are also good reasons to perform a fresh install over an upgrade. In this case, to ensure we do not break past requirements/assumptions, we could not just automatically expand or create a larger coredump partition to adhere to new requirements. This is actually not the first instance of this, here are two additional examples in which a fresh installation would have potentially yielded a more optimal environment:

Quick Tip - Enabling ESXi Coredumps to be stored on USB

Two coredump partitions in ESXi 5.5?