After getting hands on with the Lenovo P3 Tiny, which was my first time experiencing a Lenovo kit, I was recommended by the Lenovo team to also check out its larger and more powerful sibling the Lenovo P3 Ultra.
While many of the smaller form factor systems have gotten more capable over the years, especially the classic 4x4 kits, they do have their limits in terms resources and expandability.
It has been some time since I have looked at larger kits but after glancing at the P3 Ultra specs, I can see why the Lenovo team thought this might be of interests to the VMware Community, especially with all the additional capabilities, packed into a larger but still pretty compact form factor design.
Compute
There are up to four CPU options (Intel Core i9, i7 and i5) that span two different Intel CPU generations (Raptor Lake and Alder Lake), with a total of 13 configurations with varying base CPU frequencies:
Intel 13th Generation:
- Core i9-13900T - 24 Cores 32 Threads (8P + 16E)
- Core i9-13900K - 24 Cores 32 Threads (8P + 16E)
- Core i9-13900 - 24 Cores 32 Threads (8P + 16E)
- Core i7-13700T - 16 Cores 24 Threads (8P + 8E)
- Core i7-13700K - 16 Cores 24 Threads (8P + 8E)
- Core i7-13700 - 16 Cores 24 Threads (8P + 8E)
- Core i5-13600K - 14 Cores 20 Threads (6P + 8E)
- Core i5-13600 - 14 Cores 20 Threads (6P + 8E)
- Core i5-13400T - 10 Cores 16 Threads (6P + 4E)
- Core i3-13100 - 4 Cores 8 Threads (4P + 0E)
Intel 12th Generation:
- Core i9-12900 - 16 Cores 24 Threads (8P + 8E)
- Core i7-12700 - 12 Cores 20 Threads (8P + 4E)
- Core i5-12500 - 6 Cores 12 Threads (6P + 0E)
While there are some overlap of CPU options with the recently reviewed Lenovo P3 Tiny, there are some additional CPU SKUs that are only available with the P3 Ultra. Either way, great to see more Intel Core i9 options for those interested in having the maximum number of cores.
UPDATE (11/05/24) - Thanks to reader Bijan, who shared the P3 Ultra now officially supports up to 192GB of memory using 4 x 48GB DDR5 SODIMM!
The memory capacity is what makes the P3 Ultra stands out from all other systems in the market, it can support a whopping 128GB memory (ECC and non-ECC) using 4 x DDR5 4800 SO-DIMM modules! When you take out the internal P3 Ultra chassis, you will find 2 slots located on top, next to the NVMe storage (see the Storage section for a close up picture) and 2 more slots located on the underside.
One question that I have received from many readers is whether the P3 Ultra can support the new non-binary DDR5 48GB memory modules? In theory, this could bring the system from 128GB to 192GB of memory by using the larger 48GB DDR5 SODIMM modules! Unfourntately, I have to report that the P3 Ultra can NOT support the new non-binary DDR5 modules. When attempting to boot the system with just the 2 x 48GB DDR5 SODIMM and combining it with the 2 x 32GB DDR5 SODIMM, the system does not post and the screen was blank without any error sounds that would have indicated an issue.
Perhaps in a future revision of this platform or maybe even via firmware update, Lenovo can support the higher memory capacity as that would make this a pretty killer setup for a VMware environment!
Form Factor
While the P3 Ultra is definitely larger than most of the small form factor systems (pictured above top to bottom: Intel NUC 13 Pro, Lenovo P3 Ultra & Supermicro E200-8D), it was not much bigger (87 x 223 x 202 mm) in person than the E200-8D and it provides a lot more expandability. Inside of the P3 Ultra, you will find two expansion slots: 1 x PCIe 4.0 x16 and 1 x PCIe 3.0 x8 that can support a number of different add-on devices such as graphics and/or storage. The overall design is very robust and I also love the simple latch on the back that allows you to simply slide out the internal chassis without requiring any tools, which is a plus for serviceability.
Network
The P3 Ultra comes with two onboard network adaptors: Intel I225-LM (2.5GbE) and Intel I210-AT (1GbE), both of which are recognized by ESXi. Lenovo also provides a few additional network add-on adaptors that can provide up to four additional 1GbE ports using either an Intel I350-T2 (2 x 1GbE), Broadcom BCM5719 (4 x 1GbE) or Broadcom BCM5720 (2 x 1GbE), all of which are supported by ESXi and are even listed on the official VMware HCL!
If you need to add faster networking, there are two Thunderbolt 4 ports on the P3 Ultra which would allow you to add these Thunderbolt 10GbE solutions for ESXi. You can also add some USB-based networking by using the popular USB Network Native Driver for ESXi Fling.
Storage
With a larger chassis, the P3 Ultra can support up to three storage devices comprised of 2 x M.2 PCIe x4 Gen 4 (2280) and 1 x 2.5" SATA HDD, providing an optimal setup for installing ESXi on a reliable medium while enabling the use of vSAN for those interested including the new vSAN Max capability that is now included in the latest release of vSphere 8.0 Update 2. Both NVMe slots are easily accessible after lifting up the fan as shown in the screenshot above and no screws are required to service the NVMe devices as it uses a simple plastic push pin to secure the devices.
Security
For those interested in ESXi attestation capabilities, you will be pleased to know that the P3 Ultra includes a discrete TPM which is fully compatible with vSphere as you can see from the screenshot above. As some of you may know, not all systems includes a proper TPM device and for consumer systems, they typically prefer the cheaper fTPM which is not supported with ESXi.
Graphics
The P3 Ultra includes a standard built-in Intel integrated graphics (iGPU) that can be used with ESXi, but with the additional space, it can also support a more powerful NVIDIA discrete GPU (dGPU) ranging from a T400 (4GB) or T1000 (4GB), which are same options as the P3 Tiny or the more capable RTX2000 (12GB) or the RTX A5500 Laptop GPU (16GB), both of which are only available on the P3 Ultra.
Note: One thing to be aware of is that if decide to add the extra NVIDIA GPU, make sure your monitor is connected to the dGPU rather than iGPU if you need to enter the system BIOS. It took me awhile to figure out the default behavior for entering the BIOS as that is only available when the monitor is connected to the dGPU. Furthermore, the dGPU uses a Mini DisplayPort (mini DP), so if you only have HDMI, you will also need a mini DP to HDMI converter. Once you are in the BIOS, you can change this default behavior (automatic) to prefer the iGPU, but for the initial setup, you will need to connect and use the mini DP.
As expected, the Intel iGPU (UHD 770), shown above can be successfully passthrough and consumed by an Ubuntu VM by following these instructions HERE.
One of the reasons that I was excited for the P3 Ultra is that you can add an NVIDIA RTX A5500, which is one of the GPUs that supports NVIDIA vGPU or at least I thought it did. After installing the required NVIDIA vGPU ESXi drivers, I noticed it was not able to detect the GPU. I pinged one of our Engineers to debug and came to learn that the RTX A5500 Laptop is NOT the same as an RTX A5500, which is listed as as a supported device for NVIDIA vGPU. I was certainly disappointed to see that NVIDIA treats these two models differently even though it probably uses the exact same chipset and should just work with the vGPU drivers but probably from a testing perspective, they decided to not support it.
Since vGPU is not possible with the RTX A5500 Laptop, then the only other option is VM passthrough which I used an Ubuntu 23.04 VM and was able to install the required drivers as shown in the screenshot above. For successful VM passthrough of the NVIDIA GPU, you will also need to add the following VM Advanced Setting to properly power on the VM:
pciPassthru.use64bitMMIO = TRUE
Once the NVIDIA graphics drivers are installed, we can see that it is properly claimed by the Ubuntu VM and I can also use the nvidia-smi utility (pictured above) demonstrating that everything is functional.
The RTX5500 Laptop can also be used with Windows based system, unlike the Intel iGPU which can not be passthrough'ed to a to a Windows VM (known issue with Intel graphics drivers for Windows). As shown in the screenshot above, I was able to successfully passthrough the RTX5500 Laptop to a Windows 11 VM and all drivers installed without any issue.
While it would have been great to be able to use NVIDIA vGPU with the RTX5500 Laptop GPU, the P3 Ultra does provide some more capable NVIDIA GPU options, especially for those interested in exploring AI/ML.
ESXi
The latest release of ESXi 8.0 Update 1 installs fine on the P3 Tiny without any issues, no additional drivers are required as the Community Networking Driver for ESXi has been productized as part of the ESXi 8.0 release. With vSphere 8.0 Update 2 now available, I am happy to share that it also runs perfectly fine on the P3 Ultra! If you want to install ESXi 7.x, you will need to use the Community Networking Driver for ESXi Fling to have it recognize the onboard network devices.
It is recommended to disable the E-cores within the BIOs to prevent ESXi from PSOD'ing due to the non-uniform Intel CPU cores, which will result in the following error "Fatal CPU mismatch on feature". To update the CPU cores settings, enter the Lenovo BIOS under Advanced->CPU Setup, adjust either the Core Multi-Processing (P-Cores) or Efficient (E-Cores) settings.
If for some reason you prefer not to disable either the P-cores or E-Cores, then you can add the following ESXi kernel option cpuUniformityHardCheckPanic=FALSE to workaround the issue which needs to be appended to the existing kernel line by pressing SHIFT+O during the boot up. Please see this video HERE for the detailed instructions for applying the workaround.
Note: If you decide NOT to disable either E-Cores or P-Cores, you may also run into an additional PSOD when powering on a VM with GP exception in world message. To workaround this problem, please see this blog post HERE.
VMware Cloud Foundation
With 128GB of memory, I knew I had to deploy something special on the P3 Ultra and VMware Cloud Foundation (VCF) immediately came to mind as I know this is something many users are interested in exploring and deploying within their environment. The quickest and easiest way to deploy VCF is by using the VMware Cloud Foundation Lab Constructor (VLC) Holodeck Toolkit and as you can see above, I still had plenty of resources to spare!
I say this partially joking with a bit of seriousness because this was deployed using Holodeck as-is without applying any of the the VCF optimizations that I have shared in the past, which can help further reduce the amount of resources that is actually required and can enable you to deploy additional workloads. Just imagine what you can deploy running a few of these systems in your lab, from exploring vSphere with Tanzu leveraging the NVIDIA GPU to some basic AI/ML training or inferencing? 😀
Project Keswick
For completeness sake, I have also deployed the recently announced Project Keswick on the P3 Ultra and with a ton more resources, you can run quite a few container or VM-based workloads using the new Keswick Cloud Service, all deployed using a GitOps model!
How loud is the machine?
Edit: Looks like I wasn't pushing the GPU hard enough, I can now hear the fan but its not like Supermicro fan which has a pretty distinct (jet engine type sound), its definitely audible but nothing unbearable if its next to you, which my system is. Even under decent load earlier, it didn't generate much noise that bothered me but I wanted to update this response as I was jus doing some processing and could hear it kick he fans over my E200-8D. I suspect it'll vary on your usage ...
Thoughts on physical space for a half height dual 100GbE PCIe adapter? Like a mlnx connect-x 5?
Can you provide a direct link to the adaptor? I can ping the Lenovo PM and see if they can confirm device could be use
Thank you for doing this, does the bios support sriov? Looking to use this with a GPU that does support it. Also, were you able to test the Intel vpro/ame for out of band remote console and power management?
Yes, the CPU is SR-IOV capable and can be toggled within BIOS. No, I've not played with any of the remote/power settings.
the intel AMT works (and only works) with the 2.5GbE onboard nic
vGpus on esxi(standalone) will be welcome.
intel AMT wow i love it on the 2 tinys i have it on one Dell does not have the UI poor inplementation by Dell.
Hi, I couldn’t able to config Intel AMT on P3. Can you please share me how to set it up
Hello William
For the TPM, did you config anything from bios or esxi? I still have "TPM 2.0 device detected but a connection cannot be established" warning for my P360 ultra.
From hardware spec, I can see P360 ultra also has "Discrete TPM 2.0, TCG certified, FIPS 140-2 certified"
Nothing special, TPM was enabled by default.
Its important to note that P360 Ultra != P3 Ultra and I suspect the earlier models probably used fTM (VERY common in consumer systems and while it'll show TPM 2.0, that doesn't mean much as there are dTPM and fTPM that both show up as TPM 2.0. See https://williamlam.com/2022/10/quick-tip-tpm-2-0-connection-cannot-be-established-after-upgrading-to-esxi-8-0.html for reasoning between the two types)
Very cool! Thanks for the great write up! It kind of reminds me of the late '90s Compaq Deskpro EN SFF which really started the trend of small desktops.
Were you able to get the hardware sensors working with your P3 Ultra in ESXi?
Hi, do you know how are they cooling the 13900k?
I have bought a P3 Ultra because of your informations here in the article 🙂
But I have a problem and hope that you can help me:
Using the built-in NVMe, I have "Hardware Acceleration = unsupported" which means that the automatic storage reclaiming doesn't work 🙁
But that would be very important for me, do you know of a driver or so that I need to install to get that working?
Thanks!
Awesome!
re: NVMe, this is typically dictated by the underlying storage device and for remote storage, this would be VAAI capabilities. Are you sure that your NVMe device is _actually_ capable as thats where the functionality is being provided
Hello William,
thanks for your reply 🙂
Well I am using the onboard "main" storage device, do you know if your P3 Ultra can do space reclamation with the internal M.2 storage? Not that I have forgotten to configure s.th. ...
Thanks 🙂
Found out that, It supports 48x4 gb ddr5 now
I will purchase & give a try
Now? Was there a refresh of HW or new firmware update?
been told by reseller & they have tested it.
That’s good to know! When I tested it, it didn’t work, so wondering if there was new FW that enables it
send you a separate email also regards that to communication
could you please test 48gb modules on your own p3 ultra with new bios update? i have got p3 ultra with 2x16gb dimms and want to upgrade to 4x48gb for 192gb total. returning modules is not an option so want to make sure it could work before i make a purchase ;(
Hooray, they officially say 192GB is supported.
See section "Memory": https://psref.lenovo.com/Product/ThinkStation/ThinkStation_P3_Ultra_SFF?tab=spec - Last updated: 2024-09-27
Excerpt from the specs:
Max Memory[1]
Up to 192GB (4x 48GB DDR5 SO-DIMM)
Memory Type**[2]
* DDR5-4800 SO-DIMM ECC, maximum transfer speeds of up to 4000 MT/s
* DDR5-4800 SO-DIMM non-ECC, maximum transfer speeds of up to 4000 MT/s
Memory Slots
Four DDR5 SO-DIMM slots, dual-channel capable
Memory Protection
ECC on models with ECC DIMMs and ECC capable processor
Notes:
1. The max memory is based on the test results with current Lenovo® memory offerings.
2. System comes with DDR5-4800 memory and will run at lower speed due to platform limitations:
4x 32GB configuration runs at 3600 MT/s;
other configurations run at 4000 MT/s.
Best regards,
Bijan
Thank you for sharing Bijan! I’ve updated the article w/new info
Hi William, came across your post on the P3 Ultra, interesting build out.
One question I couldn't find an answer to when looking through forums or the Lenovo hardware manual...
The GPU in this situation (A5500 I think), would normally come with a single 8-pin power connector. I don't see that cable, nor a spot on the motherboard to connect GPU power. Was that true in this build?
Thanks so much
Let me ping the Lenovo team and see if I can get you an answer
Thanks to William's great post, I also bought two P3 Ultra recently and would like to contribute some information.
## My Machines
- CPU: Intel Core i7-12700 w/ vPro
- Model: 30HB
- BIOS version: S0JKT1BA (Factory default)
- Manufactured on 2024/06
## Notes
- Non-binary RAM is working flawlessly by default, I installed 2 x Micron Crucial DDR5-5600 48GB (Total 96GB), but as the whitepaper states, they are limited to 4000MHz.
- If 4 slots populated fully, memory speed will be locked to 3600MHz.
- Discrete TPM is NOT working on my 2 machines, only Firmware TPM is available in BIOS. So you will receive "TPM 2.0 device detected but a connection cannot be established" in ESXi 8 (update3).
- However, I've opened the chassis to confirm that the physical TPM (Infineon SLB9672) is installed on my machines (near wireless LAN slot on top side), just don't know why I cannot switch to discrete TPM.
I am wondering if your BIOS is different from mine, could you check what options you have under "TCG Feature Setup"? Thank you.