For customers who had SATA controllers that consumed the VMware Advanced Host Controller Interface (AHCI) driver found that after upgrading to ESXi 6.5, the disk performance for those devices were significantly impacted. Basic operations such as cloning or uploading an OVF/OVA would literally double if not triple in time. In fact, I too had observed this same behavior when I had upgraded my Intel NUC (not an officially supported platform) to ESXi 6.5. One thing I had noticed at the time when others were reporting simliar issues was that their HW platforms were also not on the VMware HCL, so I was not sure if this was limited to only home-lab environments?
In any case, I and others eventually stumbled onto this blog article by Sebastian Foss who I believe may have been the first to identify a workaround which was to simply disable the new AHCI Native Driver which loads by default and forcing it fall back to using the legacy AHCI driver which made the issue go away after a reboot. Although the folks who had reported seeing simliar issue were all using hardware platforms that were not officially on the VMware HCL, I decided to still file an internal bug and hoped someone could take a look to see what was going on.
With the release of ESXi 6.5 Update 1, I am happy to report the observed performance issues with the Native AHCI driver have now been resolved! I have been running on earlier release of ESXi 6.5 Update 1 build for couple of weeks now and have not seen any of the problems I had before. For those interested, the official fix went is in version 1.0.0-37vmw or greater of the vmw_ahci driver.
You can easily verify for this by running the following ESXCLI command to retrieve the version of your vmw_ahci driver:
If you had disabled the Native AHCI driver, you will definitely want to re-enable it. You can check if its been disabled by running the following ESXCLI command and checking the second column to see if it shows "false":
esxcli system module list | grep vmw_ahci
If the Native AHCI driver is disabled as shown in the previous command, then you can re-enable it by running the following ESXCLI command:
esxcli system module set --enabled=true --module=vmw_ahci
Once you have re-enabled the driver, you will need to reboot for the changes to go into effect.
Any reason to move back to the native driver if the legacy AHCI driver is working fine?
Please see http://www.virtuallyghetto.com/2013/10/esxi-55-introduces-new-native-device.html 🙂 Native Drivers is future driver framework and much of that has been getting ported over in the last couple of releases. There's a reason I've prefixed the existing AHCI driver as "legacy"
Afraid this did not work for me. While getting 500 MBs + read, write speeds are still at ~60MBs. ( Samsung SSD PRO 480 GB )
Double checked the driver was updated & enabled. Re-booted the host as well.
Same for me.
I've update the host to 6.5 UD1 and the module info reports. I did disable the driver before upgrading, though. Does this affect anything?
[root@vm:~] esxcli system module get -m vmw_ahci
Module: vmw_ahci
Module File: /usr/lib/vmware/vmkmod/vmw_ahci
License: BSD
Version: 1.0.0-32vmw.650.0.0.4564106
Build Type: release
Provided Namespaces:
Required Namespaces: com.vmware.vmkapi@v2_4_0_0
Containing VIB: vmw-ahci
VIB Acceptance Level: certified
Michael
Please take a look at the article above that shows you how to check whether its disabled and re-enable if it is 🙂
Silly me! Pls forget my previous comment. I did only upgrade the VCSA appliance and not the ESXi host itself 😉
William - This has appeared to fix my performance probs, too. I have a slightly newer version than the one you posted. Just interesting if you knew of any differences btwn the two.
Module: vmw_ahci
Module File: /usr/lib/vmware/vmkmod/vmw_ahci
License: BSD
Version: 1.0.0-39vmw.650.1.26.5969303
Build Type: release
Provided Namespaces:
Required Namespaces: com.vmware.vmkapi@v2_4_0_0
Containing VIB: vmw-ahci
VIB Acceptance Level: certified
Curtis, did you run some performance benchmarks against your SSD? What write speeds are you getting?
No, nothing formal. Just general VM usage...migrations, vm boots and general ops. Before, it was noticeably stalled - now, quick/normal.
Hello!
I downloaded this new version to test it on my HP Proliant Microserver Gen8 and the performance was not as expected.
If I put it in RAID mode in HD 2 TB RED:
Read: 136 and Write 78
AHCI support
Read: 96 and Write 48
What's going on?
Thank you very much!
Do we have to wait for HP to release the HP ISO image of ESXi 6.5 U1 to get this fix OR can we just update that one driver file while on ESXi 6.5.0 of the HP ISO image of ESXi?
Look at that, the HP Custome Image for ESXi 6.5U1 is already out at
https://my.vmware.com/group/vmware/details?downloadGroup=OEM-ESXI65U1-HPE&productId=614
This apparently "bricked" my instance. All VMs are appearing as /vmfs/volume/* and no storage instances appear. This was due to the "trick" described by disabling the module. I re-enabled (no upgrade) and am now sitting with several VMs that won't boot and "invisible" storage instances.
Any suggestions?
I reinstalled from the 6.5u1 iso and my data was still there.
All changes with ESXi 6.,5 U1
# esxcli system version get
Product: VMware ESXi
Version: 6.5.0
Build: Releasebuild-5969303
Update: 1
Patch: 26
# esxcli software vib list | grep ahci
sata-ahci 3.0-26vmw.650.1.26.5969303 VMW VMwareCertified 2017-08-22
vmw-ahci 1.0.0-39vmw.650.1.26.5969303 VMW VMwareCertified 2017-08-22
# esxcli storage core path list
(edited)
sata.vmhba0-sata.0:5-t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____
UID: sata.vmhba0-sata.0:5-t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____
Runtime Name: vmhba0:C0:T5:L0
Device: t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____
Device Display Name: Local ATA Disk (t10.ATA_____Samsung_SSD_850_PRO_2TB_________________S3D4NX0J702488T_____)
Adapter: vmhba0
Channel: 0
Target: 5
LUN: 0
Plugin: NMP
State: active
Transport: sata
Adapter Identifier: sata.vmhba0
Target Identifier: sata.0:5
Adapter Transport Details: Unavailable or path is unclaimed
Target Transport Details: Unavailable or path is unclaimed
Maximum IO Size: 33554432
# esxcli system module set --enabled=false --module="vmw_ahci"
# reboot
After reboot
# esxcli system module list | grep ahci
ahci true true
vmw_ahci false false
# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"
655360+0 records in
655360+0 records out
real 6m 20.89s
user 0m 0.00s
sys 0m 0.00s
# esxcli system module set --enabled=true --module=vmw_ahci
#reboot
After reboot
# esxcli system module list | grep ahci
ahci true true
# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"
655360+0 records in
655360+0 records out
real 5m 20.66s
user 0m 0.00s
sys 0m 0.00s
Are these figures normal ?
on a SATA datastore I got
[root@localhost:/vmfs/volumes/589566b4-b40e0cf2-8cac-0cc47a7c09b4] time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"
real 13m 29.23s
I can't answer that, but I noticed that both of us are using a Samsung SSD Pro. Could the problems be specific to this brand/type of SSD ?
Read performance is fixed, but write performance is still low with 6.5U1.
While I can copy from and AHCI SSD or HDD to my RAID with 200 MB/s - 400 MB/s the other direction (AHCI write) is constantly 30 MB/s. No matter, if SSD or HDD.
Would you be able to perform these operations with and with OUT AHCI enabled and generate a support bundle which I can share w/Engineering team?
I found out that the SSD I had used for testing is not good (Tanscend TS480GSSD220S). When I tried it out in my desktop machine it performed bad as well.
After switching to another SSD everything is fine.
CT525MX300SSD1, 500GB SSD SATA, vmw_ahci
# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"
real 0m 53.31s
CT525MX300SSD1, 500GB SSD SATA, attached at SAS2108 RAID controller. Only disk cache.
Write Through, DirectIO, No Read Ahead, Disc Cache Policy: Enable
# time sh -c "dd if=/dev/zero of=test bs=16k count=640k conv=sync"
real 0m 55.31s
Still much slower than in my desktop system, but in the same system vmw_ahci is not slower than the RAID controller.
Do we need to updade the vCenter server from 6.5 to 6.5 update 1, before updating the host from 6.5 to 6.5 update 1. Is there a co-relation between vCenter and ESXi build versions?
FYI This does not fix the issue with RDM. The native driver (vmw_ahci) completely botches your RDM disks.
My 8TB disks were showing up as 1.3TB, and RAW.
Thankfully switching back to legacy ahci fixed the issue, but that was scary for a minute.
How can I disable vmw_ahci in a USB boot installer for ESCi 6.5 ?
I edited boot.cfg and removed vmh_ahci from the boot.cfg. This made the installer not hang at scanning or installing.
However now installer hangs at 68 % after unexpected error FileNotFoundError vmh_ahci.v00
Is there a way to execute "esxcli system module set --enabled=false --module=vmw_ahci" in an installer?
Using Mac Mini 7,1
Any update in relation to this? same issue here
Had exactly this problem using Crucial MX300 SSDs and Intel S2600JF motherboard.
Pre-update, (ESXi 6.5.0) with the default AHCI driver, I was seeing about 2MBps write speeds.
I used the legacy AHCI driver and the write speeds were about 400MBps - great!
Except for one particular revision of Intel S2600JF motherboard where creating/migrating VMs was very unreliable.
When I was troubleshooting I saw that the write errors were increasing at about 0.5% of the write operations - which is very unusably bad.
I have upgraded to vmw_ahci Version: 1.0.0-39vmw.650.1.26.5969303 and am using the 'new' AHCI driver and now my problems are solved!
Thank you!
Update - That particular Intel S2600JF motherboard revision is still showing problems with random write failures and read and write errors.
Sorry to confirm:
I am using a Samsung 840 PRO.
esxcli system module list | grep ahci
vmw_ahci true true
time sh -c "dd if=/dev/zero of=test bs=16k count=100k conv=sync"
102400+0 records in
102400+0 records out
real 0m 38.12s
user 0m 0.00s
sys 0m 0.00s
part2
esxcli system module list | grep ahci
ahci true true
vmw_ahci false false
time sh -c "dd if=/dev/zero of=test bs=16k count=100k conv=sync"
102400+0 records in
102400+0 records out
real 0m 15.74s
user 0m 0.00s
sys 0m 0.00s
Waiting for fix, VMware...
Guys, I was finally able to get results by unmounting/detaching then physically removing the SSD. Once re-attached, and created a new datastore read/writes were in the 500+ MBs.
This was tested under 6.7.
The performance issue is back in ESXi 6.7. AHCI controller is Sunrise Point-H AHCI Controller. The performance seems ok in another box with Intel Chipset AHCI Controller.
have you managed to make it work with sunrise point h ahci controller?
Also the problematic native ahci driver doesn't support physical RDM disk greater than 2TB. Partition shows as RAW instead of NTFS in Windows. Issue disappears after switch to the legacy ahci driver.
ESXI 6.7u1 (10302608) still/again shows this problem with Intel Xeon D2100 / Lewisburg chipset on SuperMicro X11SDV-8C-TP8F mainboard. Work-around still works.
Same here: ESXi 6.7u1 has problems with latency and device loss with X10SDV-8C with vmw_ahci driver that resolves with use of legacy sata-ahci driver
Unfortunately this didn't work for me. ESXi 6.7U2 in 2019. Supermicro motherboard with onboard SAS + SATA.
Was having all manner of issues with speed, and eventually system hanging and crashing when accessing drives. As well, the general page loading when doing anything related to disk was terrible (browsing folders actually took loading time). If this sounds like your symptoms, the solution for my system might help you too!
Supermicro X9DR3-LN4F+ here, with onboard SAS and SATA, but probably X10 and maybe X11 is the same - as well as many other motherboards with the same chipset, I would imagine.
I tried EVERYTHING - BIOS settings, different drives, this vmw_ahci driver, all manners of things. Was banging me head against the wall.
Then I realised I'd never updated the BIOS on this board. Checked the version, and it was a super old (2 major versions, and 6 (!!!) years older than the current version).
Updated motherboard BIOS, reinstalled ESXi and _immediately all of the datastores from before worked great_! Bonus: the data is even there.
It's running with the stock ahci driver as well from ESXi. Seems there is no need to change it if this fixes it for you.
TL;DR - check your BIOS version on your motherboard. A BIOS update might finally fix it for you too. Digging through the release notes for the BIOS firmware from this mobo, I see there are a number of SATA and/or SAS controller changes and/or updates through the revisions. Makes sense I guess.
SSD Performance issues with Intel NUC 6.7.0 Update 3 (Build 14320388)
I am also facing performance issues with my SSD.
ESXi is on latest version and I have not updated/changed any drivers.
I tried to get back to the older driver sata-ahci but that lead to a not bootable environment.
How do I go back in 6.7 U3 to the older SATA driver?
I tried this but than I had no driver left that the SSD could use:
esxcli system module set --enabled=false --module=vmw_ahci
I also tried this
esxcfg-module --enable sata_ahc.v00
But the driver did not show up.
Are there any steps I can try to get it back and to see if this gives be better performance?
Right now I am at 80 MB/sec when copying inside the Data store.
Below is my environment. I am trying to update my SSD bios but that isn't easy having the disk running in a ESXi ...
Any tip is highly appreciated.
Daniel Nashed
Intel NUC NUC8i3BEH2
Samsung SSD 840
0000:00:17.0 Mass storage controller SATA controller: Intel Corporation Cannon Point-LP SATA Controller [AHCI Mode] [vmhba0]
esxcli system module get -m vmw_ahci
Module: vmw_ahci
Module File: /usr/lib/vmware/vmkmod/vmw_ahci
License: BSD
Version: 1.2.8-1vmw.670.3.73.14320388
Build Type: release
Provided Namespaces:
Required Namespaces: com.vmware.vmkapi@v2_5_0_0
Containing VIB: vmw-ahci
VIB Acceptance Level: certified
Hi all,
I have experienced performance issues for several years on my i7 NUC, and always thought it was related to the ahci drivers. But it turns out it was the USB Passthrough that was causing all the issues. Disabling Passthrough magically fixed all intermittent High Disk Latency issues, as well as intermittent lockups of my Ubuntu Guest. 15-60 seconds just to log in to it.
Maybe this is not the right place to inform of my findings, but if you have the same issues as I had - you will most likely end up here too, to find a solution...
I have two NUCs - the i7 version has the Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode]. The i5 NUC has the Intel Corporation Lynx Point SATA Controller [vmhba0]. But I only experienced the Passthrough problem with the Wildcat Controller. I still want to use the Passthrough feature - but can live without it for the sake of performance.
Cheers
Hi Mike,
Appreciate you sharing your findings and observations. I think this still comes up from time to time and potential workaround depending on customer needs from homelab standpoint.
I've seen this performance issue still with ESXi 6.7u3 in combination with a Samsung EVO sata SSD. Disabling the vmware ahci driver fixed the issue for me.
I am unable to get this to work with a Transcend 256GB SATA III 6Gb/s MTS430S 42 mm M.2 on 6.7u3. I cap out at 40MB/s. The host sees the PCI device info as:
Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller
ID: 0000:00:12.0
Device ID: 0x5ae3
Vendor ID: 0x8086
Function: 0x0
Bus: 0x0
lspci -v | grep "Class 0106" -B 1
0000:00:12.0 Mass storage controller SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller [vmhba0]
Class 0106: 8086:5ae3
I tried disabling vmw_ahci but after the reboot I lose the datastore. I thought the sata_ahci should recognize it, but that doesn't get loaded. What would prevent the legacy ahci from loading? I installed the VFrontDE sata_xahci which then depends on and loads sata_ahci, but still hard caps at 20MB/s.
In the BIOS on a Fitlet2 I can only turn the SATA port on or off and use AHCI type, so I don't have any many options there.
esxcli system module get -m vmw_ahci
Module: vmw_ahci
Module File: /usr/lib/vmware/vmkmod/vmw_ahci
License: BSD
Version: 2.0.5-2vmw.670.3.132.17167734
Build Type: release
Provided Namespaces:
Required Namespaces: com.vmware.vmkapi@v2_5_0_0
Containing VIB: vmw-ahci
VIB Acceptance Level: certified
Hello, i have a weird situation. where i have a h11dsi-nt mainboard. and have ssd directly connected to the board. In esxi 6.5 u3 under hardware it was using the fch sata controller by amd. i have a total of 4 ssd but it is only showing 3 sata controller but in the disks view I can see all 4 ssd and able to use all 4 ssd. but I was trying to passthrough the ssd to a windows machine but only able to pass 3 since only 3 sata controller showed up. Does this have to do with vmw-ahci? Not sure where to begin on this.
lately we have updated our hosts to the latest esxi VMware ESXi, 7.0.3, 19482537
i've seen this article and i dont know if it related to my AHCI but the speed is really slow.
i have a VM running the latest vMware tools 11365
12 CPU ( Cores per Socket 6 ( Sockets : 2 )
32GB Memory
HDD of 200 SSD on the RAID 10 ( HPE SSD hard disk ).
Hardware version is 19
but the speed is very poor.
i hope someone can point me to the right direction