ESXi on Lenovo ThinkStation P3 Ultra

After getting hands on with the Lenovo P3 Tiny, which was my first time experiencing a Lenovo kit, I was recommended by the Lenovo team to also check out its larger and more powerful sibling the Lenovo P3 Ultra.

While many of the smaller form factor systems have gotten more capable over the years, especially the classic 4x4 kits, they do have their limits in terms resources and expandability.

It has been some time since I have looked at larger kits but after glancing at the P3 Ultra specs, I can see why the Lenovo team thought this might be of interests to the VMware Community, especially with all the additional capabilities, packed into a larger but still pretty compact form factor design.

Compute

There are up to four CPU options (Intel Core i9, i7 and i5) that span two different Intel CPU generations (Raptor Lake and Alder Lake), with a total of 13 configurations with varying base CPU frequencies:

Intel 13th Generation:

• Core i9-13900T - 24 Cores 32 Threads (8P + 16E)
• Core i9-13900K - 24 Cores 32 Threads (8P + 16E)
• Core i9-13900 - 24 Cores 32 Threads (8P + 16E)
• Core i7-13700T - 16 Cores 24 Threads (8P + 8E)
• Core i7-13700K - 16 Cores 24 Threads (8P + 8E)
• Core i7-13700 - 16 Cores 24 Threads (8P + 8E)
• Core i5-13600K - 14 Cores 20 Threads (6P + 8E)
• Core i5-13600 - 14 Cores 20 Threads (6P + 8E)
• Core i5-13400T - 10 Cores 16 Threads (6P + 4E)
• Core i3-13100 - 4 Cores 8 Threads (4P + 0E)

Intel 12th Generation:

• Core i9-12900 - 16 Cores 24 Threads (8P + 8E)
• Core i7-12700 - 12 Cores 20 Threads (8P + 4E)
• Core i5-12500 - 6 Cores 12 Threads (6P + 0E)

While there are some overlap of CPU options with the recently reviewed Lenovo P3 Tiny, there are some additional CPU SKUs that are only available with the P3 Ultra. Either way, great to see more Intel Core i9 options for those interested in having the maximum number of cores.

UPDATE (11/05/24) - Thanks to reader Bijan, who shared the P3 Ultra now officially supports up to 192GB of memory using 4 x 48GB DDR5 SODIMM!

The memory capacity is what makes the P3 Ultra stands out from all other systems in the market, it can support a whopping 128GB memory (ECC and non-ECC) using 4 x DDR5 4800 SO-DIMM modules! When you take out the internal P3 Ultra chassis, you will find 2 slots located on top, next to the NVMe storage (see the Storage section for a close up picture) and 2 more slots located on the underside.

One question that I have received from many readers is whether the P3 Ultra can support the new non-binary DDR5 48GB memory modules? In theory, this could bring the system from 128GB to 192GB of memory by using the larger 48GB DDR5 SODIMM modules! Unfourntately, I have to report that the P3 Ultra can NOT support the new non-binary DDR5 modules. When attempting to boot the system with just the 2 x 48GB DDR5 SODIMM and combining it with the 2 x 32GB DDR5 SODIMM, the system does not post and the screen was blank without any error sounds that would have indicated an issue.

Perhaps in a future revision of this platform or maybe even via firmware update, Lenovo can support the higher memory capacity as that would make this a pretty killer setup for a VMware environment!

Form Factor

While the P3 Ultra is definitely larger than most of the small form factor systems (pictured above top to bottom: Intel NUC 13 Pro, Lenovo P3 Ultra & Supermicro E200-8D), it was not much bigger (87 x 223 x 202 mm) in person than the E200-8D and it provides a lot more expandability. Inside of the P3 Ultra, you will find two expansion slots: 1 x PCIe 4.0 x16 and 1 x PCIe 3.0 x8 that can support a number of different add-on devices such as graphics and/or storage. The overall design is very robust and I also love the simple latch on the back that allows you to simply slide out the internal chassis without requiring any tools, which is a plus for serviceability.

Network

The P3 Ultra comes with two onboard network adaptors: Intel I225-LM (2.5GbE) and Intel I210-AT (1GbE), both of which are recognized by ESXi. Lenovo also provides a few additional network add-on adaptors that can provide up to four additional 1GbE ports using either an Intel I350-T2 (2 x 1GbE), Broadcom BCM5719 (4 x 1GbE) or Broadcom BCM5720 (2 x 1GbE), all of which are supported by ESXi and are even listed on the official VMware HCL!

If you need to add faster networking, there are two Thunderbolt 4 ports on the P3 Ultra which would allow you to add these Thunderbolt 10GbE solutions for ESXi. You can also add some USB-based networking by using the popular USB Network Native Driver for ESXi Fling.

Storage

With a larger chassis, the P3 Ultra can support up to three storage devices comprised of 2 x M.2 PCIe x4 Gen 4 (2280) and 1 x 2.5" SATA HDD, providing an optimal setup for installing ESXi on a reliable medium while enabling the use of vSAN for those interested including the new vSAN Max capability that is now included in the latest release of vSphere 8.0 Update 2. Both NVMe slots are easily accessible after lifting up the fan as shown in the screenshot above and no screws are required to service the NVMe devices as it uses a simple plastic push pin to secure the devices.

Security

For those interested in ESXi attestation capabilities, you will be pleased to know that the P3 Ultra includes a discrete TPM which is fully compatible with vSphere as you can see from the screenshot above. As some of you may know, not all systems includes a proper TPM device and for consumer systems, they typically prefer the cheaper fTPM which is not supported with ESXi.

Graphics

The P3 Ultra includes a standard built-in Intel integrated graphics (iGPU) that can be used with ESXi, but with the additional space, it can also support a more powerful NVIDIA discrete GPU (dGPU) ranging from a T400 (4GB) or T1000 (4GB), which are same options as the P3 Tiny or the more capable RTX2000 (12GB) or the RTX A5500 Laptop GPU (16GB), both of which are only available on the P3 Ultra.

Note: One thing to be aware of is that if decide to add the extra NVIDIA GPU, make sure your monitor is connected to the dGPU rather than iGPU if you need to enter the system BIOS. It took me awhile to figure out the default behavior for entering the BIOS as that is only available when the monitor is connected to the dGPU. Furthermore, the dGPU uses a Mini DisplayPort (mini DP), so if you only have HDMI, you will also need a mini DP to HDMI converter. Once you are in the BIOS, you can change this default behavior (automatic) to prefer the iGPU, but for the initial setup, you will need to connect and use the mini DP.

As expected, the Intel iGPU (UHD 770), shown above can be successfully passthrough and consumed by an Ubuntu VM by following these instructions HERE.

One of the reasons that I was excited for the P3 Ultra is that you can add an NVIDIA RTX A5500, which is one of the GPUs that supports NVIDIA vGPU or at least I thought it did. After installing the required NVIDIA vGPU ESXi drivers, I noticed it was not able to detect the GPU. I pinged one of our Engineers to debug and came to learn that the RTX A5500 Laptop is NOT the same as an RTX A5500, which is listed as as a supported device for NVIDIA vGPU. I was certainly disappointed to see that NVIDIA treats these two models differently even though it probably uses the exact same chipset and should just work with the vGPU drivers but probably from a testing perspective, they decided to not support it.

Since vGPU is not possible with the RTX A5500 Laptop, then the only other option is VM passthrough which I used an Ubuntu 23.04 VM and was able to install the required drivers as shown in the screenshot above. For successful VM passthrough of the NVIDIA GPU, you will also need to add the following VM Advanced Setting to properly power on the VM:

pciPassthru.use64bitMMIO = TRUE

Once the NVIDIA graphics drivers are installed, we can see that it is properly claimed by the Ubuntu VM and I can also use the nvidia-smi utility (pictured above) demonstrating that everything is functional.

The RTX5500 Laptop can also be used with Windows based system, unlike the Intel iGPU which can not be passthrough'ed to a to a Windows VM (known issue with Intel graphics drivers for Windows). As shown in the screenshot above, I was able to successfully passthrough the RTX5500 Laptop to a Windows 11 VM and all drivers installed without any issue.

While it would have been great to be able to use NVIDIA vGPU with the RTX5500 Laptop GPU, the P3 Ultra does provide some more capable NVIDIA GPU options, especially for those interested in exploring AI/ML.

ESXi

The latest release of ESXi 8.0 Update 1 installs fine on the P3 Tiny without any issues, no additional drivers are required as the Community Networking Driver for ESXi has been productized as part of the ESXi 8.0 release. With vSphere 8.0 Update 2 now available, I am happy to share that it also runs perfectly fine on the P3 Ultra! If you want to install ESXi 7.x, you will need to use the Community Networking Driver for ESXi Fling to have it recognize the onboard network devices.

It is recommended to disable the E-cores within the BIOs to prevent ESXi from PSOD'ing due to the non-uniform Intel CPU cores, which will result in the following error "Fatal CPU mismatch on feature". To update the CPU cores settings, enter the Lenovo BIOS under Advanced->CPU Setup, adjust either the Core Multi-Processing (P-Cores) or Efficient (E-Cores) settings.

If for some reason you prefer not to disable either the P-cores or E-Cores, then you can add the following ESXi kernel option cpuUniformityHardCheckPanic=FALSE to workaround the issue which needs to be appended to the existing kernel line by pressing SHIFT+O during the boot up. Please see this video HERE for the detailed instructions for applying the workaround.

Note: If you decide NOT to disable either E-Cores or P-Cores, you may also run into an additional PSOD when powering on a VM with GP exception in world message. To workaround this problem, please see this blog post HERE.

VMware Cloud Foundation

With 128GB of memory, I knew I had to deploy something special on the P3 Ultra and VMware Cloud Foundation (VCF) immediately came to mind as I know this is something many users are interested in exploring and deploying within their environment. The quickest and easiest way to deploy VCF is by using the VMware Cloud Foundation Lab Constructor (VLC) Holodeck Toolkit and as you can see above, I still had plenty of resources to spare!

I say this partially joking with a bit of seriousness because this was deployed using Holodeck as-is without applying any of the the VCF optimizations that I have shared in the past, which can help further reduce the amount of resources that is actually required and can enable you to deploy additional workloads. Just imagine what you can deploy running a few of these systems in your lab, from exploring vSphere with Tanzu leveraging the NVIDIA GPU to some basic AI/ML training or inferencing? 😀

Project Keswick

For completeness sake, I have also deployed the recently announced Project Keswick on the P3 Ultra and with a ton more resources, you can run quite a few container or VM-based workloads using the new Keswick Cloud Service, all deployed using a GitOps model!