William Lam

Quick Tip - smartd configurable polling interval in vSphere 6.0

02.20.2015 by William Lam // 1 Comment

In vSphere 5.1, one of the major storage enhancements that was part of the new I/O Device Management (IODM) framework was the addition of SMART (Self Monitoring, Analysis And Reporting Technology) data for monitoring FC, FCoE, iSCSI, SAS protocol statistics, this is especially useful for monitoring the health of an SSD device. The SMART data is provided through a SMART daemon which lives inside of ESXi and runs every 30 minutes to gather statistics and diagnostic information from the underlying storage devices and provides the information through the following ESXCLI command:

esxcli storage core device smart get -d [DEVICE]

If you would like to learn more about IODM and SMART, be sure to check out Cormac Hogan's in-depth article here.

The default polling interval for the SMART daemon in vSphere 5.1 was not configurable and 30 minutes was the system default. For most customers, the out of the box configuration should be sufficent. However, for some customers who wish to have greater flexibility in the polling frequency, the default can now be adjusted in vSphere 6.0. The smartd process now includes a new -i option which specifies the polling interval.

[root@mini:~] smartd -h
smartd: option requires an argument -- 'h'
smartd <options>
-i Polling interval (in minutes) for smartd
(default Polling interval is 30 minutes)

If you wish to change the default, you will need to modify the /etc/init.d/smartd init script and include the interval option. One issue that I have found is that changes to the init script do not persist reboots as modifications to these files should not be performed by users. In the case of adjusting the polling interval, we need to add the additional option for smartd startup.

We can still accomplish this by adding the following to /etc/rc.local.d/local.sh make the necessary adjustments and restarting the smartd process:

SMARTD_POLL_INTERVAL=35
/etc/init.d/smartd stop
sed -i "s/^SMARTD_SCHED_PARAM.*/SMARTD_SCHED_PARAM=\"-i ${SMARTD_POLL_INTERVAL} ++group=smartd\"/g" /etc/init.d/smartd
/etc/init.d/smartd start

Note: The -i option is only visible when smartd process is not running

If you wish to see the changes live immediately, then you can run /etc/rc.local.d/local.sh command once or this will automatically happen upon ESXi booting up. If we perform a process look up using "ps", we can see that our smartd is now configured to poll every 35 minutes instead of the default 30.

Did you know of an additional cool vMotion capability in vSphere 6.0?

02.19.2015 by William Lam // 59 Comments

There was an excellent blog post from Duncan a couple of weeks back going over the new vMotion capabilities in vSphere 6.0 which includes: Cross vSwitch vMotion, Cross vCenter vMotion (xVC-vMotion) and Long Distance vMotion (LD-vMotion). If you have not checked out his article, I highly recommend you give it a read before proceeding further. After reading through Duncan's article, I noticed he had missed out on one additional vMotion capability which might not be obvious as the option is no where to be found in the vSphere Web Client UI. In fact, I was only aware of this additional capability after hearing about it from Engineering during the development of vSphere 6.

The additional vMotion capability actually extends the Cross vCenter Server vMotion (xVC-vMotion) workflow which allows an administrator to live migrate a running virtual machine between two vCenter Servers that are part of the same SSO Domain. By virtue of being in the same SSO Domain using the new Enhanced Linked Mode feature, both vCenter Servers will be visible in the vSphere Web Client and will be available to be selected either as a source or destination for a vMotion operation.

This extended Cross vCenter Server vMotion capability (unofficially I am calling it ExVC-vMotion) allows an administrator to live migrate a running virtual machine between two vCenter Servers which are NOT part of the same SSO Domain. How cool is that!? In my opinion, this is actually a pretty big deal because I think it truly removes any boundaries for a vSphere virtual machine and will open up an entire new class of mobility use cases that were never thought possible before. This will definitely make it interesting for customers who wish to migrate workloads from their on-premises datacenter into a completely different vSphere environment or even one that is hosted by a service provider or maybe even vCloud Air?

The ExVC-vMotion operation is currently only available today using the vSphere API, not because it is a private API but because there is no UI wizard for this operation. The reason the current xVC-vMotion is so seamless today is that both your source and destination vCenter Server is visible by being part of the same SSO Domain. If you have two completely different vCenter Servers which are not joined to the same SSO Domain or have completely different SSO Domains, then you will need to use the vSphere API to perform this operation.

All vMotion operations including vMotion without shared storage uses the vSphere API RelocateVM_Task() method. In vSphere 6.0, the method has been enhanced to accept a new property called ServiceLocator which provides a service endpoint to a vCenter Server where a VM can be migrated to. One important thing to note is that if you wish to migrate a VM between two vCenter Servers located in the same SSO Domain, there is an sslThumbprint property that is not required to be set. However, if the two vCenter Servers are NOT part of the same SSO Domain, then you need to set that property. In addition, if the VM is migrated to a different vCenter Server, additional properties such as the ESXi host, vSphere Cluster/Resource Pool and Datastore must be specified as part of the migration spec.

UPDATE (05/25/16) - I have re-factored and simplified my xVC-vMotion script which supports additional capabilities. Please have a look at the blog post here for more details.

To demonstrate this awesome ExVC-vMotion operation, I have created a simple PowerCLI script called run-cool-ExVC-vMotion.ps1 which accepts 12 command-line parameters which are described in more detail below:

Variable	Description
sourceVC	The hostname or IP Address of the source vCenter Server
sourceVCUsername	The username to connect to source vCenter Server
sourceVCPassword	The password to connect to source vCenter Server
destVC	The hostname or IP Address of the destination vCenter Server
destVCUsername	The username to connect to the destination vCenter Server
destVCPassword	The password to connect to the destination vCenter Server
destVCThumbprint	The SSL Thumbprint (SHA1) of the destination vCenter Server (can be retrieved using either this or this)
datastorename	The destination vSphere Datastore where the VM will be migrated to
clustername	The destination vSphere Cluster where the VM will be migrated to
vmhostname	The destination vSphere ESXi host where the VM will be migrated to
vmnetworkname	The destination vSphere VM Portgroup where the VM will be migrated to
vmname	The name of the source VM to be migrated

In my lab environment, I have configured two vCenter Server's which are part of two different SSO Domains as seen in the screenshot below:

I have tiny Linux VM (vMA) that I am using which I will be migrating from vcenter60-4 to vcenter60-5 which has a completely different datastore and VM portgroup (if you have stretched/extended L2, then the VM would stay online during this migration). I then execute the script using the following parameters based on my own environment and we can see the migration is kicking off:

.\run-cool-ExVC-vMotion.ps1 vcenter60-4.primp-industries.com *protected email* VMware1! vcenter60-5.primp-industries.com *protected email* VMware1! 82:D0:CF:B5:CC:EA:FE:AE:03:BE:E9:4B:AC:A2:B0:AB:2F:E3:87:49 vesxi60-8-local-storage NY-Cluster vesxi60-8.primp-industries.com NY-VM-Network vMA

One the migration has completed, if we now take a look at our vSphere Web Client, we can see the VM has now been migrated to the other vCenter Server.

I really hope to see the vSphere Web Client get enhanced to support this cool vMotion capability, but in the mean time you can easily perform this operation using the above PowerCLI script or any other scripting/programming language calling into the vSphere API. Set your VM free and let it migrate where your heart desires 🙂

Quick Tip - "poor mans" history command in ESXi

02.19.2015 by William Lam // 1 Comment

When I am logged into the ESXi Shell, I often forget that history command is not implemented in ESXi which can be helpful when recalling the list of operations that had been executed in the past. I especially rely on the history command when I am tinkering around with things and once I am successful with the end result, I can easily go back and see the exact steps I took. Recently, I tried running history command only to be let down again as I forgot it was not implemented 🙁

I was thinking there had to be a way on ESXi and then it hit me! Starting with ESXi 5.1, all operations executed in ESXi Shell and Console were automatically logged to /var/log/shell.log. The information I was looking was there but instead of having to manually view the contents of the log, I could simply create an "alias" to a history command which could display the last N-number of entries using the tail command.

Here is an example alias to "history" command to view the last 50 lines in /var/log/shell.log:

alias history="tail -50 /var/log/shell.log"

To make the alias permanent and persist across reboots, we just need to add the entry to /etc/profile.local

Now, I can run the history command on ESXi and get exactly what I want.

Note: Entries in /var/log/shell.log contain more operations executed by all users. You can further refine the aliased command to search only for the current user, such as the root account.