WilliamLam.com

Automating Dead Space Reclamation in ESXi 5.0u1

by // 4 Comments

VMware released vSphere 5.0 Update 1 last week, which mainly included bug fixes but it also brought back one very cool feature that was initially introduced with the release of vSphere 5.0 called Thin Provisioning UNMAP primitive for an ESXi host. You can read more about the details in this article by my colleague Cormac Hogan.

As you can see from From Cormac's article, the process of reclaiming of dead space on a thin provisioned LUN is currently a manual process, but does it have to be? The answer is, No of course, we can can definitely automate this!

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production system. This script is provided as an example on you can automate this manual process.

Before you proceed, please understand that the UNMAP operation can potentially take a few minutes up to a few hours depending on the size of your datastore and how your array handles this operation. You should consider performing this operation during a maintenance window or during off peak hours else you could impact VMs residing on the datastore. You should also ensure you have a VAAI-capable storage array before performing running this script.

I wrote a simple shell script called reclaimMyDeadsSace.sh which needs to be executed on the ESXi Shell via SSH. The script will also perform some validation such as ensuring you are running ESXi 5.0 Update 1 and that your host is in maintenance mode as a per-caution to ensure no running VMs are on the host during this process.

You will only need to run the script on one of the hosts connected to all the datastores you wish to reclaim dead space on. You may use scp or WinSCP to transfer the script to your ESXi host and ensure you set the execute permission on the script (chmod +x reclaimMyDeadSpace.sh)

The script can be executed in two ways:

  1. Identify ALL VMFS3 and VMFS5 volumes and perform the reclaim based on the percentage entered by the user
  2. Reclaim on specific datastores specified by the user as well as the percentage to be reclaimed (this is recommended, that way script does not choose all datastores including local ones)

Here is an example of selecting ALL VMFS3 and VMFS5 datastores to reclaim 60% of free space:

Here is an example of selecting just 4 datastores specified in a file and we will be reclaiming 60% of free space:

In this example, we created a file called "datastore_list.txt" (you may name the file anything you want) which contains the following:
iSCSI-1
iSCSI-2
iSCSI-3
iSCSI-4

So if you are using thin provisioned LUNs and would like to reclaim some of that dead space back and have a VAAI-capable storage array, be sure to check out the UNMAP functionality in ESXi 5.0u1.

How to Access USB Storage in ESXi Shell

by // 29 Comments

While performing some experiments in my home lab, I needed to access a USB storage key directly on my ESXi host (not pass-through to VMs) and found it required a small trick after some tinkering. I thought I share the process in case this comes in handy for others.

Disclaimer: This is mainly for educational and testing purposes as this is not officially supported by VMware. Please use at your own risk.

Before I begin, you should know that only USB storage devices formatted with FAT16 can be accessed in the ESXi shell and is applicable to both ESXi 4.1 and 5.0.

Step 1 - Login to ESXi Shell via SSH and disable the USB Arbitrator service (this is automatically enabled by default to allow pass-through of USB devices to your VMs) using the following command: /etc/init.d/usbarbitrator stop

Step 2 - Plug-in your USB device to your ESXi host and you can verify by using the two ESXCLI commands: verifying the storage device using the command: esxcli storage core device list | grep -i usb or viewing the mounted filesystems using the command: esxcli storage filesystem list

Step 3 - Lastly, after you verify the USB device can be seen by the ESXi host, you can of course browse and access your USB device by looking under /vmfs/volumes/

Te re-enable pass-through of USB devices to your VMs, you just need to start the usbarbitrator service.

Datastore File Management using vCLI vifs

by // Leave a Comment

There are many useful scripts that are bundled with the VMware vCLI, one such script, that is not very well known is the vifs utility which provides datastore file management. When you right click on a datastore and browse using the vSphere Client, you can create a new folder, download/upload, delete and move files.

Using the vCLI's vifs utility, you can perform the same set of operations via the command-line and behind the scenes it uses the vSphere API fileManager to perform these operations. You can also browse datastore by just having access to a web browser, just point it to the following address: https://[ESXI_HOSTNAME]/folder and you can access the datastores by clicking through the links.

To browse the datastore using vifs, you will need vCLI installed on either a Windows/Linux system or you may use VMware vMA.

To browse a specific datastore for an ESXi host, you will need to first list the available datastores by using the following command: vifs --server [SERVER] --username [USERNAME] --listds

Once you have identified the datastore you are interested in, you will then use the --dir flag to list the contents of the directory and their sub-directories by using the following command: vifs --server [SERVER] --username [USERNAME] --dir '[DATASTORENAME]'

Note: The format of the datastore name must be in brackets '[datastorename]' which is how a datastore path is identified in the vSphere API. To list sub-directories, you will need a space between the datastore name and the directory name and do not forget to quote the parameter

Let's say you would like to download the .vmx configuration file for in the directory, you can use --get flag to by using the following command:

vifs --server [SERVER] --username [USERNAME] --get '[DATASTORENAME] somedir/somefile.vmx'

Note: In the example above, we are downloading the file in the current working directory denoted by the "." (period). If you wish to download it somewhere else or even renaming the file, you will need to specify the full path to the destination


If you wanted to automate the downloading of say all .vmx configuration files, it might be pretty tedious to run through the directory discovery, so here is a quick shell script called getVMVMX.sh that is more user friendly that allows you to easily download all .vmx configurations for a given datastore.

To use the script, you will need vCLI installed on either a Linux system or use VMware vMA and be sure to set the executable permission on the shell script. You will need to specify the credentials to the ESX(i) host and the specific datastore you wish to either "list" or "download" all .vmx configuration files.

Using the --listds flag, you will need to identify the datastore you wish to use. Next you will use the following command to "list" all .vmx configuration file: ./getVMVMX.sh [ESXI_SERVER] [USERNAME] "[PASSWORD]" [DATASTORE] list

To download all .vmx configuration file you will use the following command:

./getVMVMX.sh [ESXI_SERVER] [USERNAME] "[PASSWORD]" [DATASTORE] download [FOLDER]

where FOLDER is a directory that will automatically be created for you to store all .vmx configuration files

Note: You can easily modify the script to add an additional "for loop" at the beginning to automatically download .vmx configurations for all datastores. I will leave that as an exercise for the reader.

So if you ever need to grab a vmware.log file for a specific VM or upload an ISO to datastore, you can do so from the command-line using the vifs utility that is bundled with the vCLI.