vMotion across different VDS version between onPrem and VMC

For those of you who have attempted a vMotion (whether that is within a vCenter Server or between different vCenter Servers (including across SSO Domains), if the VM is running on a Distributed Virtual Switch (VDS) and the version of the VDS is not the same between the source and destination (VDS 6.5 to VDS 6.7), the operation will fail with the following error message (UI and API):

Currently connected network interface 'Network adapter 1' cannot use network 'DVPG-VM-Network (VDS-67)', because the destination distributed switch has a different version or vendor than the source distributed switch.

This behavior is no different on VMware Cloud on AWS (VMC) or at least, I thought it was, until I recently learned about a really neat feature that was introduced in the VMC 1.4p2 release, here is a snippet from the release notes:

Cross VDS version vMotion Compatibility
With this advanced configuration option enabled, bi-directional vMotion between on-premises and VMware Cloud on AWS can be achieved across different virtual distributed switch (VDS) versions (greater than or equal to version 6.0). This must be enabled on the on-premises vCenter.

It turns out there is actually a way to allow vMotions across different VDS versions, this is important for VMC because the software stack will always be using a newer version than what we ship to our onPrem customers. However, due to this limitation, we could not benefit from the latest VDS version but had to default it to VDS 6.0 to ensure that customers could migrate their workloads. The advanced setting mentioned in the release notes allows us to disable the strict compatibility check which is performed on the destination vCenter Server when a vMotion is initiated, this setting is now enabled by default on the VMC vCenter Server which is why you can perform migration across different VDS without having to do anything special on your onPrem vCenter Server.

UPDATE (11/07/21) - Thanks to Robert Cranedonk, it looks like you can now also vMotion across different NSX-T Logical Switches by adding vCenter advanced setting called  config.vpxd.network.allowVmotionBetweenLogicalSwitches and set the value to true.

UPDATE (01/02/21) - If you are running vSphere 7.x, an additional advanced setting must be configured called config.vmprov.enableHybridMode and set the value to true. For more details, you can refer to this VMware KB 79446. Thanks to reader Marc Alumbaugh for sharing this finding!

UPDATE (10/16/18) - With the release of vSphere 6.7 Update 1, customers can now also vMotion VMs from on-prem running on a VDS to VMC with NSX-T N-VDS.

To allow vMotions to go the other direction (VMC to onprem), the onPrem vCenter Server would need to be configured with an advanced setting which can be applied non-disruptively. What the release note does not go into details about is the actual process. If you wish to enable migrations back to onPrem, simply login to your vCenter Server and go to the Advanced Settings as shown in the screenshot below. You will then need to add the following property: config.migrate.test.NetworksCompatibleOption.AllowMismatchedDVSwitchConfig and set the value to true and click OK to save.

Once this is done, you will now be able to perform vMotion bi-directionally between VMC and your onPrem environment. Pretty cool!? This actually got me thinking, if this setting allows the destination vCenter Server to relax this compat check, I wonder if this could also be applicable for vMotions for a pure onPrem to onPrem deployment?

It turns out the answer is yes and in fact, the screenshot above is actually a failed vMotion of a VM with from a source vCenter Server running VDS 6.5 to vCenter Server running VDS 6.7. After applying the setting on my destination vCenter Server (6.7), I was able to successfully perform the vMotion using this quick PowerCLI snippet (more details here) and as you can see from the screenshot below, the migration was successful!

$sourceVC = Connect-VIServer -Server mgmt-vcsa-03 -User *protected email* -Password VMware1!
$targetVC = Connect-VIServer -Server mgmt-vcsa-04 -User *protected email* -Password VMware1!

$targetVmhost = "vesxi-05.cpbu.corp"
$targetDatastore = "vsanDatastore"
$targetNetwork = "DVPG-VM-Network"
$sourceVM = "TinyVM-01"

Move-VM -VM (Get-VM -Server $sourceVC $sourceVM) -VMotionPriority High `
    -Destination (Get-VMhost -Server $targetVC -Name $targetVmhost) `
    -Datastore (Get-Datastore -Server $targetVC -Name $targetDatastore) `
    -Portgroup (Get-VDPortgroup -Server $targetVC -Name $targetNetwork)

Before you go and start enabling this feature, there are few things to be aware of

• You must be running vSphere 6.0 Update 3, vSphere 6.5 Update 2 and vSphere 6.7+ and customers with NSX-V, you will need to be running at least NSX-V 6.3.6 or greater for your onPrem vCenter Server (includes ESXi host version)
• When performing vMotion across different VDS, we are only migrating the port of the VM and not the actual switch configuration where the VM resides. This means for switch level configurations like NIOC, IPFIX or NSX-V features like Distributed Firewall (DFW) will NOT be migrated to the destination vCenter Server. If you are relying on these features, you will need to ensure that they have been configured on the destination vCenter Server prior to migration. This behavior is the same whether you are going from onPrem to VMC or onPrem to onPrem
• For vSphere 7.x environment, you will also need to configure config.vmprov.enableHybridMode = true as mentioned in the note above

In general, we recommend customers to vMotion between the same VDS version. I think the biggest benefit of this capability are for customers who wish to migrate their workloads from older vSphere deployments (6.0 and 6.5) and go directly to a fresh vCenter Server, which probably has refreshed hardware. After the workloads have been migrated, they can simply decomission the vCenter Server and not have to worry about upgrades, which is something I had quite a few times during my conversations at VMworld.

Its really nice to see that we were able to solve a particular challenge in VMC and the results also directly benefit our onPrem customers!