William Lam

Dynamic memory resizing for vCenter Server 6.0

02.23.2015 by William Lam // 32 Comments

In previous releases of vSphere, scaling up resources such as storage or memory for vCenter Server was a huge pain-point for our customers. Before the various vCenter Server services can consume the new resources, some additional manual steps were required. Though this type of an operation is usually infrequent, there is still an operational overhead which can potentially lead to increased downtime of your vCenter Server.

For example, increasing storage capacity for the VCSA was an offline operation that required adding an additional disk and then copying the existing content to the new disk which can be quite error prone and lead to a significant amount of downtime. In vSphere 6.0, the VCSA now uses LVM which provides the ability for online storage capacity increase without any downtime to vCenter Server. Increasing memory was also challenging because you had to manually adjust several configuration files that manages the JVM heap settings for various vCenter Server services as described in this VMware KB. Having complex workflows to perform basic resource expansion can increase risk of errors, especially when the process is foreign to those performing it for the very first time.

To help solve this problem, in vSphere 6.0 vCenter Server (Windows & VCSA) now includes a built-in dynamic memory reconfiguration process that automatically runs at boot up. This process includes a dynamic algorithm that inspects the current amount of CPU, Memory and Storage that is available to determine the appropriate size to configure the vCenter Server. This means that if you no longer have to tweak individual JVM settings for the various services within vCenter Server, this will happen automatically by analyzing the resources that are available and then calculating the configuration based on the supported maximums for vCenter Server.

Note: In vSphere 6.0, there are additional services going beyond just the core vCenter Server, vSphere Web Client, vCenter SSO and Inventory Services.

The dynamic memory algorithm is configured to understand the minimal amount of resources for running a vCenter Server and is bounded between a "Tiny" configuration which is 2vCPU and 8GB memory and a "Large" configuration which is 16vCPU and 32GB memory. This is important to note because if you try to configure the vCenter Server with less memory than the minimal supported, though the algorithim will dynamically distribute the available memory to the various resources, it could lead to performance degradation as the different services may not be receiving the amount of memory they require to run. YMMV if you decide to reduce the supported amount of memory but the algorithm will distribute what's available.

The process which does all the magic is a utility called cloudvm-ram-size and there are several useful options to be aware of. To view the current memory assignment for the various vCenter Server services including the OS, you can run the following command on the VCSA as an example:

cloudvm-ram-size -l

From the screenshot above, we can see a very simple break down of the current memory assignment for a "Tiny" deployment which has 8GB of memory.

To show that the dynamic memory algorithm is in fact running when more memory is added, the example below is of a VCSA that was initially configured with 8GB of memory. I then capture the running configuration and then shut down the vCenter Server and increased its memory to 10GB. I then power on the VCSA and capture the running state and you can see differences in the screenshot below.

Another useful command to be aware is being able to see the current memory usage for all services. You can do this by running the following command:

cloudvm-ram-size -S

As you can see the dynamic memory algorithm is a very much welcome feature for vCenter Server and will greatly simplify the operational tasks when needing to scale up or down resources such as CPU and Memory. I know this is definitely one of the enhancements I have been waiting for and I am glad to see it here in the new vSphere 6.0 release! As of right now, a system reboot is required but who knows maybe in the future we can increase memory while the VCSA is still running and simply reloading the services ...

Quick Tip - smartd configurable polling interval in vSphere 6.0

02.20.2015 by William Lam // 1 Comment

In vSphere 5.1, one of the major storage enhancements that was part of the new I/O Device Management (IODM) framework was the addition of SMART (Self Monitoring, Analysis And Reporting Technology) data for monitoring FC, FCoE, iSCSI, SAS protocol statistics, this is especially useful for monitoring the health of an SSD device. The SMART data is provided through a SMART daemon which lives inside of ESXi and runs every 30 minutes to gather statistics and diagnostic information from the underlying storage devices and provides the information through the following ESXCLI command:

esxcli storage core device smart get -d [DEVICE]

If you would like to learn more about IODM and SMART, be sure to check out Cormac Hogan's in-depth article here.

The default polling interval for the SMART daemon in vSphere 5.1 was not configurable and 30 minutes was the system default. For most customers, the out of the box configuration should be sufficent. However, for some customers who wish to have greater flexibility in the polling frequency, the default can now be adjusted in vSphere 6.0. The smartd process now includes a new -i option which specifies the polling interval.

[root@mini:~] smartd -h
smartd: option requires an argument -- 'h'
smartd <options>
-i Polling interval (in minutes) for smartd
(default Polling interval is 30 minutes)

If you wish to change the default, you will need to modify the /etc/init.d/smartd init script and include the interval option. One issue that I have found is that changes to the init script do not persist reboots as modifications to these files should not be performed by users. In the case of adjusting the polling interval, we need to add the additional option for smartd startup.

We can still accomplish this by adding the following to /etc/rc.local.d/local.sh make the necessary adjustments and restarting the smartd process:

SMARTD_POLL_INTERVAL=35
/etc/init.d/smartd stop
sed -i "s/^SMARTD_SCHED_PARAM.*/SMARTD_SCHED_PARAM=\"-i ${SMARTD_POLL_INTERVAL} ++group=smartd\"/g" /etc/init.d/smartd
/etc/init.d/smartd start

Note: The -i option is only visible when smartd process is not running

If you wish to see the changes live immediately, then you can run /etc/rc.local.d/local.sh command once or this will automatically happen upon ESXi booting up. If we perform a process look up using "ps", we can see that our smartd is now configured to poll every 35 minutes instead of the default 30.

Did you know of an additional cool vMotion capability in vSphere 6.0?

02.19.2015 by William Lam // 59 Comments

There was an excellent blog post from Duncan a couple of weeks back going over the new vMotion capabilities in vSphere 6.0 which includes: Cross vSwitch vMotion, Cross vCenter vMotion (xVC-vMotion) and Long Distance vMotion (LD-vMotion). If you have not checked out his article, I highly recommend you give it a read before proceeding further. After reading through Duncan's article, I noticed he had missed out on one additional vMotion capability which might not be obvious as the option is no where to be found in the vSphere Web Client UI. In fact, I was only aware of this additional capability after hearing about it from Engineering during the development of vSphere 6.

The additional vMotion capability actually extends the Cross vCenter Server vMotion (xVC-vMotion) workflow which allows an administrator to live migrate a running virtual machine between two vCenter Servers that are part of the same SSO Domain. By virtue of being in the same SSO Domain using the new Enhanced Linked Mode feature, both vCenter Servers will be visible in the vSphere Web Client and will be available to be selected either as a source or destination for a vMotion operation.

This extended Cross vCenter Server vMotion capability (unofficially I am calling it ExVC-vMotion) allows an administrator to live migrate a running virtual machine between two vCenter Servers which are NOT part of the same SSO Domain. How cool is that!? In my opinion, this is actually a pretty big deal because I think it truly removes any boundaries for a vSphere virtual machine and will open up an entire new class of mobility use cases that were never thought possible before. This will definitely make it interesting for customers who wish to migrate workloads from their on-premises datacenter into a completely different vSphere environment or even one that is hosted by a service provider or maybe even vCloud Air?

The ExVC-vMotion operation is currently only available today using the vSphere API, not because it is a private API but because there is no UI wizard for this operation. The reason the current xVC-vMotion is so seamless today is that both your source and destination vCenter Server is visible by being part of the same SSO Domain. If you have two completely different vCenter Servers which are not joined to the same SSO Domain or have completely different SSO Domains, then you will need to use the vSphere API to perform this operation.

All vMotion operations including vMotion without shared storage uses the vSphere API RelocateVM_Task() method. In vSphere 6.0, the method has been enhanced to accept a new property called ServiceLocator which provides a service endpoint to a vCenter Server where a VM can be migrated to. One important thing to note is that if you wish to migrate a VM between two vCenter Servers located in the same SSO Domain, there is an sslThumbprint property that is not required to be set. However, if the two vCenter Servers are NOT part of the same SSO Domain, then you need to set that property. In addition, if the VM is migrated to a different vCenter Server, additional properties such as the ESXi host, vSphere Cluster/Resource Pool and Datastore must be specified as part of the migration spec.

UPDATE (05/25/16) - I have re-factored and simplified my xVC-vMotion script which supports additional capabilities. Please have a look at the blog post here for more details.

To demonstrate this awesome ExVC-vMotion operation, I have created a simple PowerCLI script called run-cool-ExVC-vMotion.ps1 which accepts 12 command-line parameters which are described in more detail below:

Variable	Description
sourceVC	The hostname or IP Address of the source vCenter Server
sourceVCUsername	The username to connect to source vCenter Server
sourceVCPassword	The password to connect to source vCenter Server
destVC	The hostname or IP Address of the destination vCenter Server
destVCUsername	The username to connect to the destination vCenter Server
destVCPassword	The password to connect to the destination vCenter Server
destVCThumbprint	The SSL Thumbprint (SHA1) of the destination vCenter Server (can be retrieved using either this or this)
datastorename	The destination vSphere Datastore where the VM will be migrated to
clustername	The destination vSphere Cluster where the VM will be migrated to
vmhostname	The destination vSphere ESXi host where the VM will be migrated to
vmnetworkname	The destination vSphere VM Portgroup where the VM will be migrated to
vmname	The name of the source VM to be migrated

In my lab environment, I have configured two vCenter Server's which are part of two different SSO Domains as seen in the screenshot below:

I have tiny Linux VM (vMA) that I am using which I will be migrating from vcenter60-4 to vcenter60-5 which has a completely different datastore and VM portgroup (if you have stretched/extended L2, then the VM would stay online during this migration). I then execute the script using the following parameters based on my own environment and we can see the migration is kicking off:

.\run-cool-ExVC-vMotion.ps1 vcenter60-4.primp-industries.com *protected email* VMware1! vcenter60-5.primp-industries.com *protected email* VMware1! 82:D0:CF:B5:CC:EA:FE:AE:03:BE:E9:4B:AC:A2:B0:AB:2F:E3:87:49 vesxi60-8-local-storage NY-Cluster vesxi60-8.primp-industries.com NY-VM-Network vMA

One the migration has completed, if we now take a look at our vSphere Web Client, we can see the VM has now been migrated to the other vCenter Server.

I really hope to see the vSphere Web Client get enhanced to support this cool vMotion capability, but in the mean time you can easily perform this operation using the above PowerCLI script or any other scripting/programming language calling into the vSphere API. Set your VM free and let it migrate where your heart desires 🙂