WilliamLam.com

Identifying & Fixing Virtual Machines Affected By SvMotion / VDS Issue

by // 24 Comments

UPDATE 07/13/2012 - vSphere 5.0 Update 1a has just been released which resolves this issue, please take a look here for more details for the patch as this script is no longer required.

Duncan Epping recently wrote an article about Clarifying the SvMotion / VDS problem in which he describes the scenario that would impact your VMs as well as a way to remediate those impacted VMs. I would recommend you go through Duncan's article before moving any further.

The challenge now, is how to easily identify all VMs that are currently impacted by this problem in your environment? The answer is of course Automation and leveraging the vSphere API! I created the following vSphere SDK for Perl script called querySvMotionVDSIssue.pl which searches for all VMs that are connected to a VDS and checks whether or not it's expected dvPortgroup file exists in the appropriate datastore. To use the script, you just need a system with the vCLI installed or you can just use the vMA appliance.

UPDATE: The script has now been updated to support remediation for VMs connected to both a VMware VDS as well as Cisco N1KV. The solution, thanks to one of our internal engineers was to "move" the VM's dvport from one to another, all while staying within the existing dvPortgroup which will also force the creation of the .dvsdb port file. Once the dvport move has successfully completed, we will move it back to it's original dvport that it initially resided on. We no longer have to rely on creating a temporally dvPortgroup and best of all, we can now remediate both VDS and N1KV. The script now combines both the "query" and "remediation" into single script. Please take a look at the examples below on usage.

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production systems.

Here is a sample output of the script running in "query" mode:

Only impacted VMs will be listed in the output. To remediate, I have combined the remediation script into the query script, if you wish to remediate ALL VMs that were listed as being impacted, you can specify the --fix flag and providing the option "true". This will go ahead and remediate all impacted VMs that were listed as before.

Here is a sample output of the script running in "remediation" mode:

In the screenshot above, you may noticed a few interesting details with VM3 and VM4. If you run out of dvports in a dvPortgroup, the script will automatically increase the number of ports to satisfy the swap (max of 10 due to number of ethernet interfaces a VM can have). Once the VM has been remediated, the dvportgroup will be reconfigured to it's original configured number of ports as shown with VM3.

If you have an impacted VM that is connected to an ephemeral dvportgroup, we will not be able to remediate due to the nature of how an ephemeral binding works. You will get a message on the specific interface and you will need to manually remediate using the steps outlined by Duncan or using the "old" remediation script which will create a temporally dvPortgroup (again, this will only work for VMware VDS' only).

If you run into any issues or have questions, feel free to leave a comment.

Extracting SSL Thumbprint from ESXi

by // 13 Comments

While browsing the VMTN forums earlier this week, I noticed an interesting request from a user who was trying to compile an inventory of the SHA1 Thumbprints for all his ESXi hosts. The challenge the user had, was that he was capturing this information manually by "looking" at the DCUI screen which is where the SHA1 Thumbprint for an ESXi host is displayed by default.

As you might have guessed, this can be very tedious and error prone by copying down this very long string by just looking at the screen. Even if you do not make a mistake copying this long string, I bet your eyes will eventually give out. Luckily, there are a few ways to retrieve this information and I will show you some methods to help automate this across all of your ESXi hosts.

UPDATE (05/22/16) - Here's how you can extract SSL Thumbprint using PowerShell

Option 1 - Retrieve SSL Thumbprint using the DCUI as shown above, this is going to be the most manual method.

Option 2 - If you have remote SSH or direct console access to ESXi Shell, you can login to your ESXi host and using openssl utility, you can retrieve the SSL Thumbprint which you can then use or copy off to a remote host.

openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout

Option 3 - You can remotely retrieve the SSL Thumbprint by leveraging just the openssl utility and you do not even need to login to the ESXi host. This not only allows you to retrieve the SSL Thumbprint from a centralized location, but you can easily automate this across all your hosts.

echo -n | openssl s_client -connect 172.30.0.252:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1

Using Option 3, you can easily wrap this in a simple "for" loop to iterate through all your ESXi hosts as long as you have either the hostname/IP Address. Here is a simple shell script that you can use to iterate through all your ESXi hosts to extract the SSL Thumbprint.

In the script above, I have a list of three ESXi hosts and it is simply going through each host and executing the two commands to extract the SSL Thumbprint and displaying it on the screen.

Option 4 - You can also retrieve the SSL Thumbprint using the vSphere API, but the property is only displayed when it is connected to a vCenter Server. There is a property on the ESXi host called sslThumbprint that is populated when querying against the vCenter Server that is managing the ESXi host. You can use the vSphere Health Check script which captures this and other useful information about your vSphere infrastructure.

As you can see, there are several options on obtaining the SSL Thumbprint for an ESXi host, you definitely do not have to manually read it off the DCUI screen. Automation FTW again! 🙂

Auditing vMotion Migrations

by // 3 Comments

I saw an interesting question this week about auditing vMotion events and the number of times a VM has migrated to a particular ESX(i) host for license compliance. You can view this information using the Task/Events in your vCenter Server but you can also extract out the various types of events using the EventManager in the vSphere API. You will be able to go as far back in time as your vCenter Server's database retention policy allows you to. We will be searching for the VmMigratedEvent Event which will include variety of information including the source and destination host for the VM. The destination host will only be populated upon a successful vMotion.

Of course I had to write a script to help automate this, so here is a vSphere SDK for Perl script called getNumberOfvMotions.pl that accepts the name of an existing VM and will return the number of vMotions that has been performed on the VM as well as the list of destination hosts and the number of times it has migrated to those hosts. You will need a system that has the vCLI installed or you can you use vMA.

Note:  If you want to look at past vMotion for a VM that no longer exists, this is still possible, but you will need to parse the "message" within the Event as you can no longer look up that VM object in vCenter.

Here is an example of the script running:

You can easily modify the script to audit all VM's in your environment or just use a simple "for" loop to go through a set of VM's you are interested in, but I will leave that as an exercise for you.