Supermicro kits such as the E200-8D is a very popular platform amongst the VMware community and with powerful Xeon-based CPUs and support for up to 128GB of memory, it is perfect for running a killer vSphere/vSAN setup!
Earlier this Fall, Supermicro released a "big daddy" version to the E200-8D, dubbed E300-9D and specifically, I want to focus on the 8-Core model (SYS-E300-9D-8CNTP) as this system actually listed on the VMware HCL for ESXi! The E300-9D can support up to half a terabyte of memory and with the 8-Core model, you have access to 16 threads. The E200-8D is also a supported platform by VMware, you can find the VMware HCL listing here.
I was very fortunate to get my hands on a loaner E300-9D (8-Core) unit, thanks to Eric and his team at MITXPC, a local bay area shop specializing in embedded solutions. In fact, they even provided a nice vGhetto promo discount code for my readers awhile back, so definitely check it out if you are in the market for a new lab. As an aside, when doing a quick search online, they also seem to be the only ones actually selling the E300-9D (8-Core) system which you can find here and in general, they seem to be priced fairly competitively. This is not an endorsement for MITXPC, but recommend folks to compare all prices when shopping online, especially as today is Black Friday in the US and Cyber Monday is just a few days away.
When I first heard about the E300-9D platform, I was excited for the platform as it listed support for up to two M.2 devices, which would match what the Intel NUC Hades Canyon supported. The other reason is that there are actually a number of supported M.2 NVMe devices listed on VMware's HCL which you can find here. This is actually greats news for those in the market for a small form factor systems that is officially supported by VMware, especially useful for ROBO or Edge-based deployment where space and cost is usually a constraint.
However, upon closer inspection, it looks like the system only supports a single M.2 (M-Key) which is the traditional M.2 SSD devices you would purchase for a platform like the Intel NUC (lower left hand corner of the picture). Support for the other M.2 device is actually using the B-Key form factor (immediately to the right of the M-Key in the picture) and this would be SATA based rather than PCIe.
Having said that, thanks to David Chung who was the first to share the news, you can easily add an M.2 to PCIe add-on card enabling a secondary M.2 NVMe (M-Key) device for just $13 bucks. Here is the Amazon listing, in case folks are interested. This add-on card fits nicely in both the E200-8D and E300-9D (pictured above) and this will give you another M.2 NVMe using PCIe!
With this, you can add 2nd #NVMe drive on #Supermicro E200-8D or any other low profile 1U server. You can run all-NVMe high performance #vSAN #HomeLab in a very small form factor. Picked up few from Amazon and it's running great in my #SDDCinaBox #vmware #HCI #SDDC pic.twitter.com/9pUAC3VK37
— David Chung 🇺🇸 (@dchung615) September 28, 2018
Several folks have also shared that Supermicro makes a PCIe add-on card that supports dual M.2 that can also be used for these 1U low profile systems, which is nice from a support standpoint with just having a single vendor for the system itself.
Once ESXi is installed, both of my NVMe M.2 devices were automatically detected as shown in the output below. I also used a SATADOM for installing ESXi (pictured upper right hand corner) which not only gives me a higher endurance device, but it also provides a place for me to store vSAN tracefiles and coredumps.
You can run standard VMFS, but with two of these devices, it would be a waste not to run vSAN 🙂 which of course I had setup as you can see from the screenshot below.
The networking on the E300-9D has also been significantly upgraded compared to the E200-8D which comes with 1 x 1GbE for the BMC, 2 x 1GbE and 2 x 10GbE (RJ45 or SFP+). The E300-9D includes the standard 1 x GbE for BMC, but now includes 4 x 1GbE, 2 x 10GbE (RJ45) and 2 x 10GbE (SFP+), this should be plenty of ports for anyone wanting to do some serious NSX networking!
- 4 x 1GbE is based on Intel I350 and uses the igbn driver
- Both 2 x 10GbE (RJ45) and 2 x 10GbE (SFP+) is based on Intel X722 and uses the i40en driver
All network adapters were also automatically detected by ESXi, no additional drivers or tricks are required.
Compared to the E200-8D, it is definitely a bit more louder when the system boots up but then idles out with a normal hum. Paul over at TinkerTry did a review of the 4-Core system which includes noise analysis which you can check out here. Zack Widing who also works for VMware has been spending quite a bit of time working with the E200-8D, he recently shared on Twitter that you can purchase Noctura silent fans that work nicely with both E200/E300 systems.
Compared to its little brother E200-8D, the E300-9D is definitely bigger but trying to get a sense of how large the unit is not always an easy thing if you do not have it in front of you. I thought it would be nice to compare the new E300-9D to a number of other systems the community is already familiar with.
Top to bottom:
- Intel NUC 6th Gen
- Intel NUC Hades Canyon
- Apple Mac Mini 5,3
- Supermicro E200-8D
- Supermicro E300-9D
- 1 x Supermicro E300-9D (SYS-E300-9D-8CN8TP) which MITXPC sells here. (vGhetto Discount Code)
- 1 x M.2 to PCIe Add-On Card
- Any supported LRDIMM memory (I had 128GB from a previous setup using Samsung 32GB DIMM)
- Any supported M.2 NVMe storage (I had 2 x WD which I had pulled from my Hades Canyon setup)
The E300-9D is definitely a powerful platform that packs a ton of resources in a relatively small amount of space. Depending on your goals, it may be a nice home lab setup (but I suspect it might be out of most folks budget) but where a system like this really shines would be for the Enterprise, especially for ROBO/Edge-based deployments. The other huge benefit is that this system is on VMware's HCL and is fully supported including networking and storage! If you are in the market for a small form factor system that packs a punch, definitely take a look at this system and who knows, this might even make a nice vSAN Ready Node if you ask me 😀
UPDATE (01/14/19) - Using the Supermicro AOC, you can get two additional M.2 devices, including Intel's latest M.2 4801x Optane devices. For more details, please take a look at this blog post here.
The E300-9D also comes with two U.2 connectors (located at the very back of the board pictured above) which in theory can then be connected to supported SSD devices for use with vSAN. However, looking at the space within the chassis, it might be difficult to get two standard devices inside, at least with no proper mounting which maybe a concern for Production usage. I know some folks were discussing the potential of mounting the devices outside via an external enclosure but I think that is probably more for home labs usage than having something that can just be shipped out to remote locations and just simply work. In any case, we have reached out to a partner to see if we can get access to supported SSD devices and using the U.2 slots. I will update this blog post if I make any progress in this area.
Thanks to our friends over at Intel, I was able to get my hands on a couple of the Intel Optane P4800X Series drives (2.5"). You will need to install the Intel NVMe driver, which you can find the latest version for ESXi 6.7 Update 1 here.
Came home to some new toys courtesy from our good friends over at @IntelStorage 😃 /cc @DuncanYB #Optane #P4800X pic.twitter.com/KnmhsIndvf
— William Lam (@lamw) December 1, 2018
Although I was able to get vSAN up and running using the Optane drive in the E300-9D, as you can see from picture below, there is simply no room to fit both the drive + cable (which also requires an additional power cable), yet alone trying to close up the case.
Oh yeaaaa, vSAN running on Intel Optane!
Good news is the #E300-9D can use either U.2 or M.2 cable (uses up M.2 slot)
Bad news, there’s no room in E300-9D to fit all these cables, drive + close case 🙁 pic.twitter.com/SaVHnq03ZB
— William Lam (@lamw) December 1, 2018
For both E200-8D and E300-9D, the recommendation is definitely to stick with the PCIe AOC riser card and using two standard M.2 (M-Key) which you can verify using the VMware HCL.
Brad Lay says
Why not use the AOC-SLG3-2M2 which can also support 22110 sized cards (which you pointed out are even on the HCL).
That would also mean you could use the Optane M.2 version of the 905P too!
I'm building a version of the same thing David Chung built, using the 900P 58GB M.2 sticks in my E200-8D's and using a 24 port 10G switch - you can follow along as I build it out at https://sddc.info
William Lam says
Thanks Brad, I've updated the article to mention SM's AOC, I didn't have it on hand so I didn't want to assume it would work 🙂
Jerry Huang says
The one you bought from MITXPC is called "SYS-E300-9D-8CN8TP". It's not E300-9D.
In order to let people to buy the correct one, please help to fine tune the name from your web-page.
Thanks a lot!
Jerry Huang (*protected email*)
FYI for anyone thinking... it looks like the Supermicro AOC-SLG3-2M2 doesn't fit the E200-8D. There doesn't seem to be a riser that would fit the card with the onboard x16 slot in the position it is 🙁
Hi Brad. Could you provide information on that case? Where did you buy it and what are the components?
These puppies are sweet. Why would people still be trying to run esxi on Mac minis?
For Home Labs, I would mostly agree ... but for Enterprise/Production usage, anyone who needs to develop for iOS will have a need to Virtualize MacOS and the only way to easily do so at scale and legally is using Mac HW, so Mini is still a very viable option and with 2018 models, the performance even outpaces the higher end Mac Pro systems.
You can virtualize Mac os x on esxi? When esxi runs on Apple hardware
.. I didn't know this
William Lam says
Yup! It’s been possible since vSphere 5.0, I still remember attending on-site beta when I was a customer when this was announced and how happy I was since we had number of iOS/MacOS development apps and it was not manageable. Check out “Apple” section in my blog to see various customer Production stories on how they consume this feature
The "top to bottom" list says "Supermicro E900-9D" as the bottom item, think that should be E300-9D (photo has correct description)
William Lam says
Thanks for the catch. Fixed
Francois Corfdir says
I am looking to use this host for a customer as a esxi to host some witness VM for a strech cluster vsan.
does this host have a controler that can match the vsan HCL with all flash capacity and all flasch caching ssd device.
can we find some M2 drive that can match the vsan hcl "all flash capacity tier" and the "all flash caching tier" that can go in this host
thanks a lot for your help. and have a gread day
Ty Morton says
If you are using m.2 no need for HBA controller as they are attached directly and supported.
Hey there William,
I'm currently considering either the newer SYS-E300-9D-4CN8TP or to go with SYS-E300-8D. My main concern currently is noise. Although, if the performance boost is really huge, i'd go for the E300-9D-4CN8TP.
Thanks for your advise.
I cannot seem to get my NVMe devices to be recognized by ESXi (version 6.7 U1) in this identical E300-9D system. I have a Crucial P1 1TB 3D NAND NVMe PCIe M.2 SSD (CT1000P1SSD8). When I run the 'esxcli nvme device list' command it has a Status of 'offline' and I cannot figure out what I need to do to get the SSD to appear to the ESXi host. I also have a second NVMe drive, Patriot SCORCH M.2 2280 PCIe 3.0 x2 (PS256GPM280SSDR) that I purchased simply because it was cheap and perfect for a lab server. If you can shed some light on what I can do to fix this I would greatly appreciate it.
I have replaced 3 Noctura silent fans on my e200-8d, but CPU temperature is much high(99~100) than before(55~65). Do you experience the same issue? It seems it's trade off that less noise higher temperature.
I consider to change one of previous fan back....Too sad!!
William Lam says
Sorry, I can't comment on replacing the fans as I've just got the stock ones. I'm sure others who have can comment
Hi William, I just got another high speed fan(6500~7000 rpm )but less noise than previous one,right now it keeps under 70 degrees which I am comfortable to run any testing I want. Thank you.
William Lam says
Great to hear! Do let us know how your testing goes and also share the brand/model you used as I'm sure others are interested as well
@Rocky Which model of "high speed fan(6500~7000 rpm )" did you buy?
I hope this is not late. I bought the SUNON fan from a computer shop in Taiwan as below link.
So,basically I installed 2 Noctura silent fans and SUNON fan. it's pretty quiet and CPU temperature is under 70 degrees when CPU workload is under 50%, in my actual case, when CPU usage is 70~ 85% , the degree is around 85 ~ 90 which is acceptable for me.
Have you been having a look at Supermicro X11SDV-8C-TP8F
Do you know how this compares to the E300-9D for a homelab setup? I think this will match my needs better as it is possible to put more "stuff" in it. Would like to have internal disks.
brent c douglas says
Hi Lars, the E300-9D comes in 4 (SYS-E300-9D-4CN8TP) core and 8 (E300-9D-8CN8TP) core versions. The Supermicro X11SDV-8C-TP8F is the motherboard in the E300-9D-8CN8TP. The E300-9D-8CN8TP has loads of more I/O!!!
Tommy Braes says
Well, I love my E200-8D but it was too loud for me, so I made it virtually silent with a 140mm fan
Great article. Couple of comments. Supermicro also over a few different chassis. For example 5019D (or 1019D dial power) is a rack mount version with space for 4 2.5" form factor drives. I didn't provide the full part number but one uses the same motherboard that is discussed here. Same is true of the 5019A (Atom C3000).
Regarding noise and CPU temp, I've done quite a bit of testing on this. I started by ordering an air shroud (ref 5019D accessories). I then modified for the E300 chassis. The increased air focus helped lower fan RPM.
However, with hot air dumping onto the NIC components it also had the effect of increasing temp in that area.
I also tested with Noctua 40x40x20 fans. They simply can't push enough CFM in my opinion and a system under heavy load would create a problem.
So the Frankenstein mod to truly solve is a top mounted larger fan which removed the heat from the chassis at the point of origin.
I selected a high quality 90mm fan 4 pin from an old HPE desktop. I also borrowed the RPM reduction adapters that ship with Noctua fans.
The result is as follows:
40x40x28 fans running at 1300 RPM
90mm fan running at 900 RPM
CPU temp at 60 degrees C
The next mod I plan is to update the air shroud to seal the heat sink with the top of the case. This will have the effect of forcing the top mounted fan to pull air only from the heat sink instead of also pulling air from the gap between the heat sink and top of the case.
Back to the air shroud, which is the OEM optional accessory for the 5019D chassis, that was a fun party of the experiment. The two 40x40x28 fans behind the heat sink, they are not perfectly centered on the heat sink So, I cut and refabricated to create the shape needed. Thanks to some Scotch tape the result was effective and relatively clean looking.
Marcelo Costa says
Do you have any plans to test AS-E301-9D-8CN4? It uses AMD EPYC, and it looks like promising.
I too would like to see a comparison of the AMD AS-E301-9D-8CN4 vs the Intel SYS-E300-9D-8CN8TP. I'm currently trying to make a decision on which on to buy.
Marcelo Costa says
I decided to take a risk and asked my contractor to acquire one. No regrets, it works smoothly. It's almost the same size as E200 but if the same processing capabilities of E300.
John H. says
I'm sure this new unit will be appealing as it now has 4 x 2.5 " SSD Bays.
Im trying to decide with the following two servers here, Supermicro E200-8D vs SUPERMICRO SUPERSERVER 5018D-FN8T XEON D.
ESXi 7/vSphere, vCenter, vSAN, vCloud, etc for home lab/SOHO setup.
Richard Bejtlich says
Thanks for your articles. The SYS-E300-9D-8CN8TP looks like it is still a great system for ESXi 7. In a home lab with 120 v AC power sockets, how do you handle the DC power cable?
John Luce says
The external power supply will take AC and convert to DC.
James S says
In late 2019 I bought a SYS-E300-9D-8CN8TP from MITXPC and took it with me overseas. The system stopped posting shortly after it went out of warranty, so I sent it off to SuperMicro TW. CPU is bad - they're charging me 645 USD to swap it. :/ Think I'll stick with swappable systems or better warranties in the future.