WilliamLam.com

AHCI (vmw_ahci) performance issue resolved in ESXi 6.5 Update 1

by // 44 Comments

For customers who had SATA controllers that consumed the VMware Advanced Host Controller Interface (AHCI) driver found that after upgrading to ESXi 6.5, the disk performance for those devices were significantly impacted. Basic operations such as cloning or uploading an OVF/OVA would literally double if not triple in time. In fact, I too had observed this same behavior when I had upgraded my Intel NUC (not an officially supported platform) to ESXi 6.5. One thing I had noticed at the time when others were reporting simliar issues was that their HW platforms were also not on the VMware HCL, so I was not sure if this was limited to only home-lab environments?

In any case, I and others eventually stumbled onto this blog article by Sebastian Foss who I believe may have been the first to identify a workaround which was to simply disable the new AHCI Native Driver which loads by default and forcing it fall back to using the legacy AHCI driver which made the issue go away after a reboot. Although the folks who had reported seeing simliar issue were all using hardware platforms that were not officially on the VMware HCL, I decided to still file an internal bug and hoped someone could take a look to see what was going on.

With the release of ESXi 6.5 Update 1, I am happy to report the observed performance issues with the Native AHCI driver have now been resolved! I have been running on earlier release of ESXi 6.5 Update 1 build for couple of weeks now and have not seen any of the problems I had before. For those interested, the official fix went is in version 1.0.0-37vmw or greater of the vmw_ahci driver.

You can easily verify for this by running the following ESXCLI command to retrieve the version of your vmw_ahci driver:


If you had disabled the Native AHCI driver, you will definitely want to re-enable it. You can check if its been disabled by running the following ESXCLI command and checking the second column to see if it shows "false":

esxcli system module list | grep vmw_ahci

If the Native AHCI driver is disabled as shown in the previous command, then you can re-enable it by running the following ESXCLI command:

esxcli system module set --enabled=true --module=vmw_ahci

Once you have re-enabled the driver, you will need to reboot for the changes to go into effect.

Quick Tip - Locating SRM Placeholder VMs using the vSphere API

by // 5 Comments

I had received a question the other day from a reader where they were trying to distinguish between the running VM and its placeholder VM due to their use of VMware Site Recovery Manager (SRM). Since the VM name is exactly the same in both vCenter Servers, it was not clear how to identify between the two. As mentioned in my reply to the reader, there are a couple of ways. You could use the SRM API in-conjunction with the vSphere API (in his case, he was using PowerCLI) to be able to check whether the VM in question was the placeholder VM or not.

Another option is to simply use the vSphere API and querying for the managedBy property which is populated when SRM and/or other solutions are associated with managing a set of VMs. In the case of SRM, you will see an extensionKey with value of "com.vmware.vcDR" and type with value of "placeholderVm" which tells you that the VM is an SRM Placeholder VM, pretty easy, right!? 🙂

Since I did not have an SRM environment handy, the next best thing was to check out VMware Hands-On-Lab environment which anyone can access for free. Lab HOL-1705-SDC-1 was exactly what I needed and here is a quick screenshot of the vSphere MOB showing you what the managedBy property looks like in the vSphere API.

To demonstrate the use of this vSphere API, I wrote a quick PowerCLI function called PlaceholderVMs.ps1 and below is an example of running the Get-PlaceholderVM command:

vSphere Content Library versioning 

by // 1 Comment

A question came up the other day on how versioning is handled when using the vSphere Content Library feature and specifically when you update an existing VM template that already exists in the Content Library (CL, which I will be using throughout the article). Having used CL since vSphere 6.0 which I have also written about it extensively here, I knew that it had a built-in versioning mechanism which would automatically increment as new updates were applied to the VM Templates stored in CL.

One thing that I do not think many folks are aware of is that you can actually retrieve the CL Item Version within the vSphere Web Client. You can do by just adding the "Content Version" column by right clicking on column headers and select show/hide columns to add or remove specific fields.


I figured, this was pretty straight forward and shared my findings with the individual and thought that was it. While working on my Content Library PowerCLI community module, I came to learn there was more than one version that was being returned by the CL API.

Here is an example screenshot showing the three different versions using the Get-ContentLibraryItems (not to be confused with the OOTB Get-ContentLibraryItem cmdlet):


Taking a look at the API documentation, it was still not 100% clear on what all these versions provided and also when would they increment? I reached out to one of the CL Engineers, Eric who then provided me more information on how each of these versions are being used. Lets go through each one of them and I will include an example which also helped me understand how these different "versions" actually work.

Content Version - This particular version will increment as changes are made to the actual file content itself. This is also the same version that is available in the vSphere Web Client UI as shown earlier.

Lets now go through an example workflow to show when this version will get incremented:

  1. Create a new VM called Foo (ContentVersion=N/A)
  2. Clone VM Foo into CL called Foo VM Template (ContentVersion=1)
  3. Clone completes for Foo VM Template (ContentVersion=2) <--The reason this is 2 and not 1 is that when the initial CL Item is created (think of it as a placeholder), the Content Version is incremented
  4. Deploy Foo VM Template to new VM called Bar (ContentVersion=2, no changes on deploy)
  5. Update VM Bar with some changes and clone Bar back to Foo VM Template (ContentVersion=3) <--Changes were made and we now increment the Content Version

Metadata Version - This particular version will only increment for changes made to either the name or description of the CL template.

Version - This particular version is used for concurrency control and this will always increment along with the Metadata Version for local libraries.

Lets now go through an example workflow for a local library to show how these two versions get incremented:

  1. Create a new VM called Foo (MetadataVersion=N/A)
  2. Clone VM Foo into CL called Foo VM Template (MetadataVersion=1, Version=1)
  3. Update the description of Foo VM Template (MetadataVersion=2, Version=2)
  4. Deploy Foo VM Template to new VM called Bar (MetadataVersion=2, Version=2, no changes on deploy)
  5. Update VM Bar with some changes and clone Bar back to Foo VM Template (MetadataVersion=3, Version=3)

Lets now go through an example workflow for a subscribed library to show how these two versions get incremented:

  • Create a new VM called Foo on a local library named CL1 (MetadataVersion=N/A)
  • Clone VM Foo into CL called Foo VM Template (MetadataVersion=1, Version=1)
  • Update the description of Foo VM Template (MetadataVersion=2, Version=2)
  • Create a new subscribed library to CL1 which will download Foo VM Template (MetadataVersion=2, Version=1) <--MetadataVersion must match the publisher, but version does not have to match

After going through a few of these exercises with a few dummy VMs and using my Get-ContentLibaryItems function, I was able to get a better grasp on CL versioning. Hopefully this helpful for anyone who might want to use the CL APIs to track changes between their VM templates and/or files or just being able to see this within the vSphere Web Client UI.