ESXi on Lenovo ThinkStation P3 Tiny

Earlier this year, I came to learn about an interesting system from Lenovo called the ThinkStation P3 Tiny, which packs a ton of potential for its tiny form factor.

Funny enough, I have never actually worked with a Lenovo kit before, so I was definitely interested in getting hands on with this system. In fact, with the recent announcement of VMware Edge Cloud Orchestrator (VECO), the P3 Tiny could also make for an interesting candidate for deploying at the Edge using Project Keswick! 😄

Thanks to the Lenovo ThinkStation team, I got the opportunity to explore the P3 Tiny and see what it can offer to the VMware community.

Compute

There are four primary Intel CPU options (Core i9, i7 and i5) to select from across 12 diffrent configurations with varying base CPU frequencies:

• Intel Core i9-13900T 24 Cores 32 Threads (8P+16E)
• Intel Core i9-13900 24 Cores 32 Threads (8P+16E)
• Intel Core i7-13700T 16 Cores 24 Threads (8P+16E)
• Intel Core i7-13700 16 Cores 24 Threads (8P+16E)
• Intel Core i5-13600T 14 Cores 20 Threads(6P+8E)
• Intel Core i5-13600 14 Cores 20 Threads(6P+8E)
• Intel Core i5-13500T 14 Cores 20 Threads(6P+8E)
• Intel Core i5-13500 14 Cores 20 Threads(6P+8E)
• Intel Core i5-13400T 10 Cores 16 Threads (6P+4E)
• Intel Core i5-13400 10 Cores 16 Threads (6P+4E)
• Intel Core i3-13100T 4 Cores 8 Threads (4P + 0E)
• Intel Core i3-13100 4 Cores 8 Threads (4P + 0E)

You typically do not see an Intel Core i9 being offered in these smaller form factor systems, so this is certainly a welcome addition for those interested in having the maximum number of cores.

While the P3 Tiny supports DDR5 4800 SO-DIMM memory, it is not compatible with the new non-binary DDR5 48GB memory modules, which in theory would have enabled a total of 96GB of memory. When I attempted to boot the system with 96GB of memory, the system would not post and the screen was blank without any error sounds that would have indicated an issue. It definitely would have been awesome if the P3 Tiny could support 96GB of memory, which would have made it that much more powerful!

Form Factor

The P3 Tiny is pretty compact with the following dimensions: 179 x 182.9 x 37mm and for relative size comparison, it is smaller than a Supermicro E200-8D but slight larger (~1.5x) than the classic 4x4 Intel NUC. For a more detailed mechanical design break down, see the Specifications section.

Network

The networking out of the box on the P3 Tiny is a single Intel I219-LM (1GbE), which ESXi fully recognizes thanks to the productization of the Community Networking Driver for ESXi. Lenovo does also provide a few additional ethernet-based add-ons that utilizes the the low-profile expansion slot and can support either an Intel I350-T2 (2 x 1GbE), Intel I350-T4 (4 x 1GbE), Broadcom BCM5720 (2 x 1GbE) or Broadcom BCM5719 (4 x 1GbE), all of which are supported by ESXi. This means you can potentially have up to 5 network interfaces depending on the configuration you choose.

If you need to add faster networking, you could opt for the single Thunderbolt 4 port via the low-profile expansion slot and use one of these Thunderbolt 10GbE solutions for ESXi and/or look at USB-based networking by using the popular USB Network Native Driver for ESXi Fling.

Storage

For storage, you can install up to 2 x M.2 PCIe x4 Gen 4 (2280) which are easily accessible underneath the P3 Tiny after removing a single screw from the back, which is quite nice from a serviceability standpoint. You can setup vSAN or standard VMFS, but for the former you will need to install ESXi on either a USB device or add additional storage using the lower-profile expansion slot by using the the single Thunderbolt 4 port. Here are some Thunderbolt M.2 NVMe solutions for ESXi which can provide you with additional storage capacity.

Security

For those interested in ESXi attestation capabilities, you will be pleased to know that the P3 Tiny includes a discrete TPM which is fully compatible with vSphere as you can see from the screenshot above. As some of you may know, not all systems include a proper TPM device and for consumer systems, they typically prefer the cheaper fTPM which is not supported with ESXi.

Graphics

The graphics is what makes the P3 Tiny really shine as you not only get the built-in Intel integrated graphics (iGPU) which you can use, but you also can get add an additional NVIDIA discrete GPU (dGPU): T400 (4GB), T1000 (4GB) or T1000 (8GB) using the low-profile expansion slot.

Note: One thing to be aware of is that if decide to add the extra NVIDIA GPU, make sure your monitor is connected to the dGPU rather than iGPU if you need to enter the system BIOS. It took me awhile to figure out the default behavior for entering the BIOS as that is only available when the monitor is connected to the dGPU. Furthermore, the dGPU uses a Mini DisplayPort (mini DP), so if you only have HDMI, you will also need a mini DP to HDMI converter. Once you are in the BIOS, you can change this default behavior (automatic) to prefer the iGPU, but for the initial setup, you will need to connect and use the mini DP.

As expected, the Intel iGPU (UHD 770), shown above can be successfully passthrough and consumed by an Ubuntu VM by following these instructions HERE.

For my P3 Tiny, I had it configured with the higher-end T1000 (8GB) model and as you can see from the screenshot above, I was able to pass that through to an Ubuntu VM and install the required drivers to have it fully recognized.

Once the NVIDIA graphics drivers are installed, we can see that it is properly claimed by the Ubuntu VM (pictured above) and I can also use the nvidia-smi utility (pictured below) demonstrating that everything is functional.

One thing to note is that both the T400 & T1000 do not support NVIDIA vGPU, meaning you can not share the physical GPU across multiple VMs and passthrough is the only way to consume the GPU.

While the Intel iGPU can not be passthrough to a Windows VM (known issue with Intel graphics drivers for Windows), the NVIDIA dGPU can indeed be used with a Windows based system. As demonstrated in the screenshot above, I was able to successfully passthrough the T1000 to a Windows 11 VM and all drivers installed without any issue.

With all the interests in AI/ML these days, having an NVIDIA GPU option, while not the most powerful can still be beneficial for folks interested in exploring this space and you can use dGPU for training scenarios while using the iGPU for inferencing.

ESXi

The latest release of ESXi 8.0 Update 1 installs fine on the P3 Tiny without any issues, no additional drivers are required as the Community Networking Driver for ESXi has been productized as part of the ESXi 8.0 release. Since vSphere 8.0 Update 2 was also recently announcement, I figured I would also put that to the test and I am pleased to share that it also installs flawlessly on the P3 Tiny! If you want to install ESXi 7.x, you will need to use the Community Networking Driver for ESXi Fling to have it recognize the onboard network devices.

It is recommended to disable the E-cores within the BIOs to prevent ESXi from PSOD'ing due to the non-uniform Intel CPU cores, which will result in the following error "Fatal CPU mismatch on feature". To update the CPU cores settings, enter the Lenovo BIOS under Advanced->CPU Setup, adjust either the Core Multi-Processing (P-Cores) or Efficient (E-Cores) settings.

If for some reason you prefer not to disable either the P-cores or E-Cores, then you can add the following ESXi kernel option cpuUniformityHardCheckPanic=FALSE to workaround the issue which needs to be appended to the existing kernel line by pressing SHIFT+O during the boot up. Please see this video HERE for the detailed instructions for applying the workaround.

Note: If you decide NOT to disable either E-Cores or P-Cores, you may also run into an additional PSOD when powering on a VM with GP exception in world message. To workaround this problem, please see this blog post HERE.

Project Keswick

Finally, with the small form factor of the P3 Tiny, it can also provide a nice platform for an Edge deployment and what better solution to try than the recently announced Project Keswick! As you can see from the screenshot below, I was able to successfully deploy Keswick on the P3 Tiny and have it auto-register with the Keswick Cloud Service, ready for container or VM-based workloads!