I have been running the full VMware Cloud Foundation (VCF) 9.0 stack using the Minisforum MS-A2 in a three-node configuration for almost a year now. While the MS-A2 is not hardware that you would find on the Broadcom Compatibility Guide (BCG), these units have been solid for my lab environment.
Over the past couple of weeks, I have been putting my lab through its paces while testing various VCF.Next deployment and upgrade scenarios, and I began to notice the environment was sluggish, which was not something I had experienced before.
Long story short, after hours of debugging, including over the weekend with the amazing Nicholas Wilson, we were able to pinpoint one of the issues (more details at the end for the other issue) related to the onboard 10GbE SFP+ adapters. The MS-A2 uses an Intel x710, which apparently have been quite notorious when it comes to hardware offload. 😵💫
Some of the sluggishness that I had observed was due to Intel's implementation of LRO (Large Receive Offload) where jumbo packets were split up across RX queues, resulting in packets arriving out of order and forcing retransmits. Since this was an vSAN Express Storage Architecture (ESA) deployment, we believe the adaptive network traffic shaping had also kicked in, flagging these issue as hardware errors and further reducing I/O performance. This would also explain why I was only seeing at most 1-2Gbps over the vSAN network even though the setup was 10GbE end-to-end.
In parallel, Nicholas also had access to an MS-A2 environment and he was able to reproduce the symptoms that I had observed even with the current shipping version of VCF 9.0.2, which was a surprise. This meant that the performance degradation was always there when using Intel X710 with vSAN ESA, but I never noticed it until recently when I really started to push my environment harder.
With the issue now mostly understood, the primary fix was to disable hardware offload of LRO via an ESX advanced setting, which will require a system reboot for the changes to go into effect. After rebooting all ESX hosts within my vSAN Cluster, I immediately saw an increase in bandwidth exceeding ~1-2Gbps, which was the maximum I had noticed when watching my MikroTik switch.
Disable Hardware LRO (Large Receive Offload)
Run the following ESXCLI command to disable LRO offload:
esxcli system settings advanced set -i 0 -o /Net/TcpipDefLROEnabled
To verify, you can run the following command:
esxcli system settings advanced list -o /Net/TcpipDefLROEnabled
A system reboot is required for the change to go into effect.
Disable Hardware TSO (TCP Segmentation Offload)
Since there are known LRO/TSO issues with the Intel X710, I have also disabled TSO offload to be safe.
Run the following ESXCLI command to disable TSO offload:
esxcli system settings advanced set -o /Net/UseHwTSO -i 0
To verify, you can run the following command:
esxcli system settings advanced list -o /Net/UseHwTSO
A system reboot is required for the change to go into effect.
Use Enhanced Data Path (EDP) switch mode
While debugging my setup, I also realized that I had been using the Standard switch mode, since I had initially started with a hardware system that did NOT support Enhanced Data Path (EDP).
The Intel X710 on MS-A2 does support EDP, which you can check for EDP supportability by running the following command on your ESX host:
esxcfg-nics -e
When deploying VCF through the VCF Installer, you can specify several switch mode options, which I had been using STANDARD. I since switched to using the ENS switch mode, which provides vSAN ESA with a total of 8 CPU helper threads instead of the default 4.
Increase ENS RX Ring Size
Another tidbit that I had learned from Nicholas was that the Intel X710 or any network adaptor for that matter may only be using the minimum RX/TX ring size, so you may want to increase it from the default which is 1024 for both RX/TX and maximum supported for these Intel X710 is 4096 for both RX/TX. For homelab usage, especially with consumer switches, you should only increase RX ring and leave TX ring to default or you may potentially over run your physical switch.
Since we are using ENS, it has its own network stack and to increase the RX ring to 4K, you need to run the following command:
nsxdp-cli ens uplink ring set -r 4096 -n vmnic1
net-dvs --persist
To verify, you can run the following command:
nsxdp-cli ens uplink ring get -n vmnic1
At the beginning of this blog post, I mentioned that networking was just one of the issues I ran into ... After resolving the network problem, I still observed sluggishness as the environment became busier. It turned out that my consumer NVMe drives had either degraded or, more likely, were simply unable to keep up. I tried several different NVMe models, including some with onboard DRAM, but write latencies were too high and negatively impacted my workloads.
I eventually bit the bullet and purchased more performant NVMe drives, specifically three Samsung 990 Pro 1TB NVMe M.2 2280 devices for the vSAN ESA devices. Once the new NVMe drives were in, they performed exactly as I had hoped!
As shared on social media, there is a lesson to be learned here and while consumer hardware has improved greatly over the years ... investing slightly more into performant hardware can pay dividends in future, especially with the current pricing of storage and memory!
2 out the 3 990 Pro's have arrived ... I'm really regretting not spending a bit more 6 months back to get these or even Enterprise grade NVMe drives at 1/3 to 1/2 the cost it is today! 😩 pic.twitter.com/KnKmC1Iw5m
— William Lam (@lamw) March 30, 2026
Nice find William! Have the same units except 7940HX and have noticed vMotions seemed slower than my older vSphere 8 lab which is still alive running other core infrastructure VM. I'll give it a try.
By the way your "fix" for running NSX Edge's on AMD was spot-on and those VM with memory tiering off has produced a solid NSX lab.
Would it be usefull for VSAN OSA to disable the LRO/TSO offload and increase the ring buffers if X710 nics are in use?
LRO/TSO offload is simply broken on these NICs, best to disable
How did you know LRO was the issue ? What logs did you see and error messages did you see to confirm this ?
I was working with colleague from within Broadcom who's helped many customers tune their network stack for vSAN. Once I had mentioned that the NICs were X710, he already knew these would be problematic and could impact vSAN ESA issues that I had been observing. The next step would've been tcpdump, which we would see retransmit but luckily, his hunch was already spot on given these X710 have historically had issues and most Enterprise aren't using them at least with vSAN
Out of curiosity, are you going to try the 9.1 beta on your 3-node MS-A2 cluster?
This fixed the CPU softlock error messages I received on my NSX Edges when redeploying, probably some vSAN performance issue I am guessing.
Interesting thought, I've experienced the same. Great find
Well fine, I guess ill create and account. Basically x710 nics have gone by the wayside in favor of newer generations, while they still receive updates they've been known for regression issues. Additionally they have major issues reporting issues. Many of the errors they see they do not rollup through the drivers, additionally they really don't even populate the tcp transmission dashboard under the support->performance for support dashboards. I went through the known previous bugs with intel nics and saw a bunch of issues with LRO. I also was 4 days in on helping out a customer with x710s in production. I always max out ring buffer RX on vsan for performance. I had access to another colleagues MS-A2 lab and I HCI benched several settings changes over the same test profile. via the LRO and RX buffer changes i was able to 7.6x the iops perf. I additionally tried other settings but they didn't change perf more than a few % points. FYI ALL IF YOU HAVE HIGH PNIC ERROR RATE DETECTED ERRORS, they are not something that should be ignored ever, get to the bottom of it and get your network team involved.