Dual Intel M.2 Optane 4801x in Supermicro E300-9D

For those that have been following along, I have been testing the Supermicro E300-9D (SYS-E300-9D-8CN8TP) kit in my home lab which I have written about here. One of the last things I was looking into was storage expandability with this platform, especially since Supermicro has a PCIe Add-On Card (AOC-SLG3-2M2) which can add two additional M.2 devices supporting 2260 (60mm), 2280 (80mm) & 22110 (110mm) form factors.

At the same time, our friends over at Intel had just given me access to a couple of their pre-release Intel M.2 Optane 4801x devices which were then released right before the holiday last year. For those wondering why Optane is so interesting, especially from a vSAN perspective, check out this blog post here from my good buddy Pete Flecha over in our Storage and Availability Business Unit (SABU).

This was actually perfect timing as I was also interested to see if there were any high performance and supported M.2 devices that could be used for for vSAN Caching and this device would definitely fit the bill! When I had initially received all the components, I was scratching my head as the AOC was too big to fit horizontally in the E300-9D.

I then came to learn that the AOC is supposed to be installed vertically and this was made possible with an additional Supermicro vertical PCIe riser card that I was actually missing.

Once I had the vertical riser, the AOC plugged right in and then I could connect everything into the E300-9D as shown in the picture below.

The next issue I had ran into was after booting up the system I could NOT see both of the Optane devices. I thought perhaps they were faulty or the AOC or riser were having problems and manually verified that each device was visible independently on the system. I was still not able to figure it out and went on Twitter to see if any of my followers had worked with the AOC before as this was my first time using it. With the help from Paul Braren over at TinkerTry, I learned that I needed to enable bifurcation which would allow the system to see both M.2 devices and it looks like the instructions to do this may vary from system to system. Paul linked to me to the specific E300-9D motherboard documentation and I found what I needed on Page 76.

As a reference for myself and others, you will need to enable this in the BIOS by navigating to Advanced->Chipset Configuration->North Bridge->IIO Configuration->CPU Configuration and then under the IOUx, change the value from default to x4x4x4x4. Save your changes and then reboot

Once this configuration change was made, I immediately saw both Optane devices, whew! I was then able to setup vSAN using both of these devices as shown in the screenshot below:

Note: You may have noticed I have an additional Samsung M.2 device showing up in the same vSAN Diskgroup as my Intel Optane devices, this was purely from a testing standpoint and is not recommended to mix capacity devices with different IO and performance characteristics together.

I know someone of you might be wondering if this device is on both the vSphere and VSAN HCL, it looks like it just maps to the existing Intel SSD DC 4800x series Optane as the M.2 Optane has the exact same values for VID, DID & SVID as the existing 4800x series device.

For basic home lab usage, you easily add an additional M.2 (2280) using a cheap PCIe to NVMe M.2 (also possible on E200-8D) as outlined in this post here but if you are looking to get the best performance or getting up to two additional M.2 devices, you may want to consider adding the Supermicro AOC and Intel Optane devices. The cost of the two cards were $49 USD for the AOC-SLG3-2M2 and $15 USD for the PCIe Vertical Riser, both sold from Supermicro or any of their resellers.