WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Cross vCenter Server operations (clone / migrate) between versions of vSphere 6.x

02.27.2017 by William Lam // 7 Comments

When cross vCenter Server operations such as clone and migrate was first introduced in vSphere 6.0, it required that both the source and destination vCenter Server (includes ESXi hosts) to be running the same vSphere version. With the release of vSphere 6.5, this base requirement still holds true (e.g. vSphere 6.5 for both source and destination), especially when performing these operations using the vSphere Web Client where mixed-vSphere versions is not supported outside of a rolling upgrade.

Having said that, it is possible and supported to clone or migrate a VM across different versions of vSphere 6.x, for example a vSphere 6.5 and a vSphere 6.0 Update 3 environment. This can be accomplished by performing a xVC-vMotion or xVC-Clone operation using the vSphere API. For the the xVC-vMotion use case, I have extensively written about it here and here and with PowerCLI 6.5r1, the Move-VM cmdlet has even been updated based on my feedback to support this capability natively. Furthermore, you can even perform these operations across completely different vCenter Single Sign-On Domains, which enables a new level of mobility for your VMs and access to resources of independently deployed vCenter Server instances.

UPDATE (11/01/17) - The following VMware KB 2106952 has just been updated to reflect what is officially supported in terms of Cross vCenter Operations ( Clone / Migrate) across different versions of vSphere. The matrix in the KB reflects what has been tested by Engineering and one thing you may notice is that Cross vCenter vMotion/Clone from vSphere 6.x to vSphere 6.5 is only supported when running at least vSphere 6.0 Update 3. After speaking with the PM, the reason for this change is that pre-vSphere 6.0 Update 3, there were no pre-checks in the code to prevent Cross vCenter Operations for un-supported target hosts such as ESXi 5.5, which could lead to poor user experience as well as undefined failure scenarios. In addition, vSphere 6.0 Update 3 also includes additional enhancements to properly clean up failed provisioning operations which will make Cross vCenter Operations much more robust. Due to these reasons, though it is possible to perform Cross vCenter vMotion from earlier versions, it will not be officially supported. I have also updated my summarized table below to reflect what is in the VMware KB, but please use the KB as your official source of truth for what VMware supports.

To help make sense of the different combinations of vMotions and cloning operations, below are a few tables to help outline what is possible and supported today.

vMotion

Source vCenter Server Destination vCenter Server Supported UI or API
vSphere 6.0 vSphere 6.0 Yes UI and API
vSphere 6.x (pre 6.0 Update 3) vSphere 6.5 Possible but Not Supported N/A
vSphere 6.0 Update 3 vSphere 6.5 Yes API
vSphere 6.5 vSphere 6.5+ Yes UI and API
vSphere 6.5 vSphere 6.x No No
vSphere 6.5+ VMware Cloud on AWS Yes UI and API
VMware Cloud on AWS vSphere 6.5+ Yes UI and API

Cold Migrate

Source vCenter Server Destination vCenter Server Supported UI or API
vSphere 6.0 vSphere 6.0 Yes UI and API
vSphere 6.x (pre 6.0 Update 3) vSphere 6.5 Possible but Not Supported API
vSphere 6.0 Update 3 vSphere 6.5 Yes API
vSphere 6.5 vSphere 6.5 Yes UI and API
vSphere 6.5 vSphere 6.x No No
vSphere 6.5+ VMware Cloud on AWS Yes UI and API
VMware Cloud on AWS vSphere 6.5+ Yes UI and API

Clone

Source vCenter Server Destination vCenter Server Supported  UI or API
vSphere 6.0 vSphere 6.0 Yes UI and  API
vSphere 6.x (pre 6.0 Update 3) vSphere 6.5 No N/A
vSphere 6.0 Update 3 vSphere 6.5 No N/A
vSphere 6.5 vSphere 6.5+ Yes UI and API
vSphere 6.5 vSphere 6.x No N/A
vSphere 6.5+ VMware Cloud on AWS Yes UI and API
VMware Cloud on AWS vSphere 6.5+ Yes UI and API

Virtual Networking Migration

Source Type Destination Type Supported
VDS VDS Yes
VDS VSS No
VSS VSS Yes
VSS VDS Yes

Note1: vMotioning and/or cloning of VMs which uses the new vSphere Encryption feature introduced in vSphere 6.5 is not supported.

Note2: "Compute" only xVC-vMotion insufficient space issue has now been resolved with vSphere 6.0 Update 3, see this post here for more details.

Note3: xVC-vMotion is not supported on 3rd party switches as we can not checkpoint the switching state.

Here are some additional xVC-vMotion and vMotion articles that may also useful to be aware of:

  • Are Affinity/Anti-Affinity rules preserved during Cross vCenter vMotion (xVC-vMotion)?
  • Duplicate MAC Address concerns with xVC-vMotion in vSphere 6.0
  • Network Compatibility Checks During vMotion Between vCenter Server Instances
  • Auditing vMotion Migrations

Categories // Automation, vSphere 6.0, vSphere 6.5 Tags // Cross vMotion, ExVC-vMotion, vSphere 6.0, vSphere 6.5, vSphere API, xVC-vMotion

Quick Tip - vSphere 6.0 Update 3 resolves "Compute" only Cross-vCenter vMotion operation

02.27.2017 by William Lam // Leave a Comment

Previously when a "Compute" only Cross-vCenter vMotion (xVC-vMotion) was initiated, which only migrates the VMs compute from one vCenter Server to another while maintaining its current storage configuration, an insufficient space error may be thrown under certain conditions. This behavior was due to the way vCenter Server used to calculate the available space on the destination vCenter Server.

Prior to vSphere 6.0 Update 3, vCenter Server used the Managed Object Reference (MoRef) ID of the vSphere Datastore determine whether the source and destination was the same. Even if you have the exact same vSphere Datastore mounted in both the source and destination vCenter Server, there was a good chance the MoRef ID will be different which then causes this calculation to occur. Now, the "insufficient space" error only occurs IF, the free space on the current vSphere Datastore is less than the size of the VM to be migrated which is why this behavior was only observed in some environments. Some customers workaround the problem by simply freeing up enough capacity which then allowed them to perform the operation.

The good news is this has now been resolved in the latest vSphere 6.0 Update 3 release which came out last Friday and has been outlined as one of the resolved issues in the release notes:

  • Attempts to perform an exclusive compute resource cross vCenter vMotion might fail.
    When a VM is migrated using vMotion or cold migrate from a vCenter to another vCenter and space available on datastore is less than size of the Virtual Machine Disk (VMDK), an error similar to the following is displayed:
    insufficient space

Rather than using the MoRef ID to determine if the vSphere Datastore is the same in both the source and destination vCenter Server, it is now using the datastore URL path. This means, if you want the correct behavior for a "Compute" only xVC-vMotion, you should ensure that the vSphere Datastore is mounted using the same name in both the source and destination vCenter Server.

Categories // vSphere 6.0 Tags // Cross vMotion, ExVC-vMotion, vSphere 6.0 Update 3, xVC-vMotion

How to easily disable vMotion & Cross vCenter vMotion for a particular Virtual Machine?

07.20.2016 by William Lam // 6 Comments

The question of disabling vMotion for a specific set of Virtual Machine(s) is not a new one. In fact, this topic comes up on some what of a frequent basis and usually driven by arcane change management processes or worse licensing restrictions. Do not get me wrong, there are definitely some valid use cases where you would not want a particular VM to be migrated off. The classic example is a 3rd Party VM solution that provides Anti-Malware, Intrusion Detection & Firewall capabilities for your workload VMs. For this particular use case, VMware provides our partners with an integration hook into the vSphere platform called ESX Agent Manager (EAM) that ensures these "Service VMs" are not allowed to be powered off or migrated to another ESXi host, even in the case of a Maintenance Mode operation. This solutuion even allows you to configure custom icons for your Service VMs!

For all other use cases outside of the "Service VMs", there really is not an easy way of disabling vMotion for a particular VM. There have been many solutions that have been suggested in the past ranging from disabling DRS for a specific VM, DRS Affinity Rules, VM miss-configurations to break vMotion compatibility to using vSphere Permissions to prevent vMotion operations. However, many of these solutions do not work very well or is very difficult to manage at scale. I actually like the idea of using vSphere Permissions to prevent a vMotion, however, I have seen some customers push back on this because the vSphere Administrator still has the ability to perform this operation. For these cases, customers just want to be able to completely disable vMotion for a given VM and prevent anyone from migrating the VM, including the vSphere Administrators.

Given that this topic had recently come up again, I was wondering if there was an easier way in which this could be achieved and made more manageable for our customers. After thinking about about how EAM handles "disabling" certain operations for a VM and recalling an article I wrote last year which leveraged this exact capability to resolve an NSX Controller issue, I thought why not apply it to this use case here?

UPDATE (09/27/18) - As of vSphere 6.5, the MigrateVM_Task() method has been deprecated in favor of the RelocateVM_Task() which is used to handle BOTH vMotion,  Storage vMotion and other variants. This means that as of 6.5, you no longer have the ability to disable a specific migration type and when you apply the settings, both migration types will be disabled for the specific VM.

Disclaimer: The use of internal APIs are not officially supported by VMware and can change at any time. Please use at your own risk.

Each VM has a property called DisableMethod which lists the specific vSphere API methods that are currently disabled. These are not governed by vSphere Permissions but rather the runtime state of the VM. For example, if you have a VM that is currently powered on, then the PowerOnVM_Task API would not be available and would show up in the disabled list.

Here is a quick PowerCLI snippet on how to retrieve the current set of disable methods for a VM:

$vm = Get-Vm -Name TestVM-1
$vm.ExtensionData.DisabledMethod

disable-vmotion-for-vm-1
As mentioned in this article, the ability to enable and disable these methods are only available as an internal vCenter Server API. However, it is possible to access these APIs using the vSphere MOB, but it is not very user friendly nor intuitive. Below is a screenshot of invoking the disableMethods API using the vSphere MOB.

disable-vmotion-for-vm-0
A couple of weeks back I started to investigate on how we might be able to automate against the vSphere MOB. The result of that investigation lead to the creation of a simple PowerCLI script that allows you to automate operations using the vSphere MOB which I had published here. That work became the foundation for the new PowerCLI script that I had created for disabling and enabling the vMotion capability for a particular VM.

You can download the PowerCLI script here called enable-disable-vsphere-api-method.ps1 which includes two functions Enable-vSphereMethod and Disable-vSphereMethod. You will need to edit the script to provide a couple of pieces of information.

  1. Credentials to your vCenter Server
  2. Name of the VM you wish to either disable or enable vMotion capability on
  3. Name of the vSphere API method you wish to disable (by default this is MigrateVM_Task which maps to the vMotion capability)

By default, I have commented out both functions usage, you will need to manually uncomment one of the lines based on the operation you wish to perform.

To Disable the vMotion capability, run the following:

Disable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -disable_method $method_name

To Enable the vMotion capability if you had disabled it, run the following:

Enable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -enable_method $method_name

After the script has completed, you can now re-run the command that we ran earlier to see which methods have been disabled and you should see that the MigrateVM_Task is now part of the disable methods.

disable-vmotion-for-vm-5
If we now login to either the vSphere Web/C# Client and right click on the VM that we had disabled vMotion on, you should also see that the Migrate option is now grayed out and unavailable. This behavior will be true for ALL users including those in the vSphere Administrators group.

disable-vmotion-for-vm-3
It is important to note that vMotion is not only disabled from the UI, but it is also disabled from the vSphere API standpoint which the UI is built on top of. Here is an example of trying to perform a vMotion using the PowerCLI Move-VM cmdlet and you can see that an error is thrown immediately stating that the method has been disabled.

disable-vmotion-for-vm-4
Note: The "self" text output from the PowerCLI command is actually something that you can specify as part of disabling the vMotion capability. This might be useful to specify a change control ID or some string to signal to the user who might be trying to perform the operation. Please refer to the script and search for the "self" keyword if you wish to change it.

The really nice thing about this solution is not only is it really easy to enable or disable, but it can also be managed at scale which many of the other solutions mentioned earlier start to break down. The last thing anyone would want is additional operational overhead to manage manage complex DRS rules (which can still be overridden through manual migrations) or additional vSphere Permissions which also runs into the same problem where a vSphere Administrator can still override by performing a manual migration. This solution does prevent both standard vMotion as well as the new Cross vCenter vMotion capability (both between same/different SSO Domain) that was introduced in vSphere 6.0. You do not need to be running vSphere 6.0 to be able to leverage this solution, this should actually work for almost all versions of vSphere. Lastly, enabling or disabling the functionality does not require any type of system restart or impact to your VM other than the ability to vMotion.

Limitations

Beyond artificially limiting what vSphere DRS and HA can do, I did observe an interesting behavior when a Maintenance Mode operation is performed. If you leave the "Move powered-off and suspended virtual machines to other hosts in the cluster" uncheck, then all VMs will be migrated off and the VMs that have vMotion disabled will reside on the ESXi host while it goes into Maintenance Mode. However, if you do check the box, I did find that the system would override setting and actually move the VM to another ESXi host. This is something to be aware of and may not be a bad thing depending on your requirements.

disable-vmotion-vm-6

Disabling Storage vMotion

You may have noticed that if the VM is in a powered off state, that the Migrate option is still available in the UI. The reason for this is that we only disabled vMotion but you are still allowed to perform a Storage vMotion. If you wish to also disable the Storage vMotion capability, then you will need to disable RelocateVM_Task vSphere API method as well.

Auditing vMotion and Storage vMotion Operations

With or without this solution, you may still want another level of confidence that a VM has either not migrated or migrated to authorized set of ESXi hosts. We can easily do so by auditing the VM's Event system and looking for migration events. The name of the vMotion event is called VmMigratedEvent and the name of the Storage vMotion event is called VmRelocatedEvent. Here is a sample script using the vSphere SDK for Perl that exercises this specific vSphere API and provides you with all the ESXi hosts a given VM might have migrated to. For those that rather consume the vSphere API using something like PowerCLI, here is a quick one-liner to extract vMotion events:

$vm = Get-VM TestVM-2
Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmMigratedEvent"} | Select CreatedTime, UserName, FullFormattedMessage | ft -wrap -AutoSize

disable-vmotion-for-vm-5

Categories // Automation, PowerCLI, vSphere, vSphere Web Client Tags // Cross vMotion, disableMethods, enableMethods, ExVC-vMotion, Managed Object Browser, MigrateVM_Task, PowerCLI, RelocateVM_Task, Storage vMotion, svmotion, vSphere, vSphere MOB, xVC-vMotion

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...