WilliamLam.com

How to Automate the disabling of the VAAI UNMAP primitve in ESXi 5

by // 5 Comments

I just saw an interesting article on Jason Boche's blog about VMware's recall of the new VAAI UNMAP primitive for vSphere 5. VMware released a new KB article KB2007427 documenting the details along with a recommendation to disable the VAAI UNMAP primitive for now (a patch will be released in the future to automatically disable this until the issue is resolved).

One thing that caught my in the VMware KB article is to disable the VAAI UNMAP primitive, you need to manually login to ESXi 5 Tech Support Mode (SSH) to run the local version of esxcli command. This is trivial if you have several hosts, but it can be time consuming if you have several hundred hosts to manage. Even though the "VMFS3.EnableBlockDelete" is a hidden parameter that can not be seen using any of the supported utilities, you can enable and disable the property using the remote version of esxcli which is part of vMA 5 or vCLI 5.

Here is an example of the command when connecting directly to an ESXi 5 host:

esxcli --server himalaya.primp-industries.com --username root system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

Here is an example of command when connecting to vCenter Server:

esxcli --server reflex.primp-industries.com --vihost himalaya.primp-industries.com --username administrator system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

As you can see, you can easily wrap this in a simple for-loop to disable the VAAI UNMAP primitive. So here is a script to help with exactly that called vaaiUNMAP.sh

The script accepts 4 parameters:

  1. A list of ESXi 5 hosts to enable or disable VAAI UNMAP primitive
  2. Name of the vCenter Server managing the ESXi 5 hosts
  3. vCenter auth file which contains the username/password
  4. 0 to disable or 1 to enable VAAI UNMAP primitive

The auth file is just a file that contains the following:

VI_USERNAME=administrator
VI_PASSWORD=y0mysuperdupersecurepassword

Here is an example of disabling VAAI UNMAP primitive on 3 ESXi hosts being managed by a vCenter Server:

To help extract all ESXi 5 hosts from your vCenter Server, you can use the following vSphere SDK for Perl getESXi5Hosts.pl

Here is an example of running the script and just save the output to a file:

One of the unfortunate thing about the VMFS3.EnableBlockDelete is that it is a hidden parameter along with others, so it will not automatically display when using local or remote ESXCLI, but thanks to Craig Risinger, you can still get the information using the remote ESXCLI by specifying the --option parameter which is great because you do not need to login to the ESXi Shell to retrieve the information.

esxcli --server himalaya.primp-industries.com --username root system settings advanced list --option /VMFS3/EnableBlockDelete

I would also recommend adding an entry into your ESXi 5 kickstart to automatically disable VAAI UNMAP by default until a fix is released.

%firstboot --interpreter=busybox

#disable VAAI UNMAP primitive
esxcli system settings advanced set --int-value 0 --option /VMFS3/EnableBlockDelete

Script - Automate VAAI Configurations in vSphere 4.1 (vaaiHWAccelerationMgmt.pl)

by // Leave a Comment

With the release of vSphere 4.1, we finally get to see the first revision of the vStorage API for Array Integration (VAAI) features. This initial release provides the following SCSI driver primitives with VAAI supported storage arrays:

  • Write Same/Zero - Eliminating redundant and repetitive write commands, tells array to repeat via SCSI commands.
  • Full Fast/Copy - Leverage array to mass copy, snapshot and move blocks via SCSI commands.
  • Atomic Set and Test - Stop locking LUNs and start locking blocks.
  • Thin Provisioning Stun - Reporting array TP state to ESX.

The above definitions were taken off of an EMC presentation found here. VMware has also published a new VMware KB article regarding VAAI FAQ, take a look here at KB1021976.

Though these features are not specific to any one storage array vendor, the vendors themselves are required to implement these primitives within their array OS software for them to be available. If all prerequisites are met, and you have an ESX or ESXi host running on vSphere 4.1 and VAAI supported storage array, these new storage operations will now offload to the array versus running within the VMkernel.

However, if you do not have an array that supports VAAI, the new version of ESX and ESXi will still try to use these features. As I understand from an earlier discussion of VAAI, there is one additional operation that is performed and it's impact is supposed to be insignificant (please correct me if I'm wrong). Though if you would like to disable these VAAI features or would like to see the difference between a non-VAAI and VAAI operation, it is controlled with the following three advanced host configurations.

VMFS3.HardwareAcceleratedLocking - Atomic Test and Set
DataMover.HardwareAcceleratedMove - Full/Fast Copy
DataMover.HardwareAcceleratedInit - Write Same

By default, all three of these configurations are enabled with a default installation of vSphere 4.1. The following vSphere SDK for Perl script allows a user to enable or disable VAAI configuration on a set of hosts defined in an input file. The script allows you to connect to vCenter if your hosts are being managed by vCenter or directly to a specific ESX or ESXi host and provide the following parameters:

--hostlist = Lists of ESX(i) hosts to perform operation _IF_ they're being managed by vCenter (default is ALL hosts in vCenter)

--operation = Operation to perform (query|enable|disable)

Download: vaaiHWAccelerationMgmt.pl

Here is an example of the host input file:

[vi-admin@makalu scripts]$ cat hosts
esxi4-2.primp-industries.com
esxi4-3.primp-industries.com
Here is an example of querying for VAAI configurations:

Here is an example of disabling VAAI configurations:

Here is an example of disabling VAAI configurations:

For more information about vStorage APIs, take a look here.

The vStorage API, do you really know what it is?

by // 2 Comments

I’ve seen this question posed quite a few times both in the VMware developer and VMTN forum asking what exactly is the vStorage API?

So what is it?

If your answer was VMware marketing term, then you win 50 Schrute Bucks!

The VMware vStorage API is actually a blanket umbrella term that encompasses 4 separate individual APIs, all with different functionalities:

  1. vStorage API for Data Protection (VADP)
  2. vStorage API for Site Recovery Manager (VASRM)
  3. vStorage API for Multi-pathing (VAMP)
  4. vStorage API for Array Integration (VAAI)

I’m actually going to quote one of Chad Sakac’s reply on a blog about these 4 APIs as he has done a great job of explaining the differences:

1) vStorage API for Data Protection – a set of APIs focused on local backup/recovery use cases.

2) vStorage APIs for Site Recovery Manager – a set of APIs focused on array vendor remote replication such that they can be orchestrated by Site Recovery Manager.

3) vStorage APIs for Multipathing (otherwise known as the Pluggable Storage Architecture). A set of APIs for 3rd parties to extend vSphere’s core multipathing architectures.

4) vStorage APIs for Array Integration (VAAI). Technically not in vSphere 4, but have been discussed in VMworld events in the past. Will be available in future vSphere-generation releases. This allow array vendors to “offload” various tasks from the ESX host’s vmkernel stack – things like writing blocks that make up VMs, copying/snapshotting blocks, doing thin-provisioning out of space handling, and also a much more advanced global locking mechanism than VMFS uses today. These each will make common actions 5x-10x faster (clone, deploy from template, create a FT VM), and improve VMFS scaling by an order of magnitude. More on that here, for folks that are interested (note that when I wrote this, vStorage API for Data Protection was called the VCB Backup Framework)

Generally when the vStorage APIs are brought up, most people think about backups and the new Change Block Tracking feature in vSphere. That is because VMware and other 3rd party backup vendors has done a good job of marketing this "must have" feature. Leveraging Change Block Tracking helps decrease the duration of a backup by only copying the blocks that have changed and some users have seen up to 5-10x increase in speed.

This is great! But now you might ask the question, how do I use vStorage API for Data Protection and how does it tie into VMware's VCB product? Both the vStorage API for Data Protection and VCB are backup API frameworks, they are similar in functionality but different in features (for a break down of the two, take a look at this VMware blog post). VADP will be pretty much hidden from the end user's perspective, you just need to ensure that backup vendors are implementing it and utilizing Change Block Tracking to efficiently backup your VMs. VADP is only available in vSphere and not in VI 3.5.

If you want to write your own backup application, then you will need to know how to hook into VADP. As far as I understand today, and correct me if I'm wrong, the vStorage API for Data Protection is actually the combination of the vSphere 4.0 API + VMware Virtual Disk Development Kit (VDDK) which are both available to users to develop against. There's actually a guide within the VDDK page on Designing Backup Solutions for VMware vSphere which goes into great detail on creating a backup solution using the Change Block Tracking feature.

The three other APIs (VASRM, VAMP and VAAI) are targeted at third party hardware/software vendors and storage array providers to hook in their special sauces with the various VMware solutions. This includes hooking into Site Recovery Manager, PSA (Pluggable Storage Architecture) plugins and offloading VMware operations such as VM cloning and storage vMotions, etc. onto the actual storage array. These APIs are only exposed to partners who provide solutions to one of these features and are not available to the public for general use.

I personally think vStorage API is going to be a game changer and Change Block Tracking is just one of the many cool features to come!

Hopefully this all made sense and if you're interested to learn more about the vStorage API and some of the upcoming features, take a look at these additional resources:

http://www.ntpro.nl/blog/archives/1461-Storage-Protocol-Choices-Storage-Best-Practices-for-vSphere.html
http://virtualgeek.typepad.com/virtual_geek/2008/09/so-what-does-vs.html
http://www.vmware.com/products/vstorage-apis-for-data-protection/
http://www.yellow-bricks.com/2009/03/19/pluggable-storage-architecture-exploring-the-next-version-of-esxvcenter/