I had been hearing a lot of cool things about VMware's Hybrid Cloud Extension (HCX) but never tried the solution myself nor had a good understanding of what it actually provided. With the recently announced Hybrid Cloud Extension (HCX) on VMware Cloud on AWS (VMWonAWS) offering being available, I thought this was a great way to get hands on with HCX and take advantage of my VMWonAWS infrastructure. Having only spent a couple of days with the solution, I can see why customers are excited for HCX and the new offering on VMWonAWS makes it super easy to consume. I also recently learned that HCX is now free for all VMWonAWS customers and you can easily live migrated your VMs to/from your onPrem environment!
There are a number of impressive capabilities that HCX offers, but two that really stood out to me which I thought was quite unique and interesting compared to other VM-based "migration" options. The first is that HCX can perform live VM migrations (vMotion) or replicated migrations (vSphere Replication) which includes scheduled switch over across different versions of vSphere (vSphere 5.x to/from vSphere 6.x). This is great for customers who may not be able to upgrade their underlying vSphere environment to 6.0 or later and take advantage of things like Cross vCenter vMotion feature which only supports VM migration between vSphere 6.0u3 to/from 6.x.
The second capability is that HCX can abstract and protect the underlying ESXi hosts by not requiring direct connectivity between the source and destination ESXi hosts. Traditionally, for vMotion and vSphere Replication traffic, you either had to stretch the VLAN or ensure the VMkernel interface was routable so that it can communicate with the destination ESXi hosts for data transfers. This was not always possible and adds additional networking requirements which can be challenging to implement depending on how your network infrastructure is configured. The way HCX solves this problem is by using a special HCX Cloud Gateway which securely proxy vMotion and vSphere Replication traffic from the on-premises environment out to the respective HCX Cloud Gateway Peer which then gets transfered to destination vSphere environment. Below is a diagram to help illustrate this:
Note: HCX also supports WAN optimization (compression and de-duplication) out of the box, which the diagram includes as that is what I had deployed in my env. This is an optional virtual appliance that can be deployed at each location ensuring efficient data transfer between the source and destination vSphere environments.
While going through and getting HCX configured on both my VMWonAWS and onPrem environment, I had ran into a few minor gotchas and to help others avoid some of the issues I had ran into, I figure I would outline the process and include some additional tips that can be help.
Step 1 - Once you have been granted access to HCX Cloud (https://cloud.vmware.com/vmware-hcx) which comes in the form of an invite with an activation code, you will then select the specific VMWonAWS SDDC that you wish to enable HCX. This will take a few minutes while a new HCX Manager is being deployed within your SDDC, which can monitor the progress using the vSphere H5 Client.
Step 2 - Once the deployment of HCX Manager has completed, you should now see an "Open HCX" option. Click on that to connect to your HCX Cloud instance. You will login using the same credentials (*protected email*) as your vCenter Server within VMWonAWS.
Step 3 - Next, we need to deploy an onPrem HCX Manager within your environment. You can download the OVA by clicking on the Administration tab and under System Updates, you will find a direct download or link as highlighted in the screenshot below. I found that the link was not clickable immediately and it took a few second before it was enabled.
Step 4 - Import the HCX Manager OVA (must use vSphere Web Client or OVFTool, H5 does not work) into your vCenter Server environment that you wish to enable HCX migrations from and power on the VM after completing the OVF property requirements. Once the onPrem HCX Manager has has fully booted up, you will need to configure it by opening a browser to following URL: https://[HCX-Manager]:9443. You will login with username admin and password you had set earlier in the OVF properties.
Step 5 - Before you can configure your onPrem HCX Manager, you will need to activate it with a different key which you should have also been provided via email. Follow the wizard to provide your vCenter Server, NSX (if you have that deployed) and SSO configuration and if everything was successful, you should see the following page below asking you to restart the application for the changes to take effect. Go ahead and click on restart and wait for the process to finish.
Step 6 - Once the restart completes, you should be re-directed to the Dashboard view where you can get a high level view of your onPrem HCX configuration as shown in the screenshot below. There are two more mandatory configurations that we need to setup before we can proceed in pairing our onPrem HCX Manager with our HCX Manager Cloud instance.
Step 7 - The first it configuring who can access the HCX pairing between onPrem and VMWonAWS, deployment and setup of HCX Cloud Gateway as well as performing VM migration, network extensions and DR operations. Here is where you would add in your specific AD User/Group(s) for the two specific roles.
Step 8 - The last configuration is importing the SSL Certificate from our HCX Manager Cloud instance to establish a trust between the two HCX Managers. Simply select URL and provide the IP Address of your HCX Manager Cloud instance (e.g. https://[HCX-Manager-Cloud-Instance-IP]) and then click Apply.
At this point, we are done configuring our onPrem HCX Manager and we can logout of the interface.
Step 9 - Login to your vCenter Server using the vSphere Web (Flex) Client and you should now see a new HCX vSphere Web Client plugin on the left hand side. Go ahead and open that and you should your onPrem HCX Manager that we had just finished configuring. Next, we will pair this with our HCX Cloud instance running on VMWonAWS by clicking on the "New Site Parings" option
You will be asked to provide Site URL, which is simply the URL of your HCX Manager Cloud Instance (e.g. https://[HCX-Manager-Cloud-Instance-IP]) and the cloudadmin username and credential.
Step 10 - Next, we will specify what HCX Services to enable. For both Live Migration and Replication Migration, you will need to select the HCX Interconnect Service which deploys the HCX Cloud Gateway which will require access to both the vMotion and VR Networks that your ESXi hosts sits on. If you would like to enable compression/de-dupe capabilities, go ahead and also select the HCX Optimization Service which will results in an additional virtual appliance to be deployed. The wizard will take you though the rest of configurations required for each of the appliance and then start the deployment before completing the site pairing.
Once the deployment finishes, you should now see a new icon denoting the VMWonAWS location (US West or East) within the UI. At this point, you have successfully deployed HCX onPrem and paired it with the HCX Cloud instance running on VMWonAWS.
If we now take a look at our onPrem vSphere Inventory, you will see two VMs that were deployed. The first is the HCX Cloud Gateway and the other is the WAN Optimization Appliance. You might also notice what looks to be an ESXi host that has been added to the Datacenter level. If you look at the IP Address, you will see that this is actually our HCX Cloud Gateway which actually acts as proxy for both vMotion and vSphere Replication traffic, which is pretty slick. If you login to your VMWonAWS vCenter Server, you will also notice an equivalent set of HCX VMs that were also deployed on VMWonAWS automatically for you by the HCX Service without requiring any additional configuration or user interaction.
Before initiating a migration or consuming HCX Services, you can head over to the Interconnect tab and verify that both HCX VMs are up and running and that the tunnel between the onPrem HCX Gateway and VMWonAWS HCX Gateway can communicate with each other. This is denoted by the status on the upper right hand side as shown in the screenshot below.
To initiate a VM Migration(s), click on the Migration tab. Here you will find a list of all your onPrem VMs and you have a number of configurable options to apply on either a global level or on a per individual VM basis which is quite nice. If you select the traditional vMotion option, the VM will migrate (compute/storage) immediately like a normal Cross vCenter vMotion but the operation can take some time depending on the networking bandwidth, size and number of VMs that wish to migrate. The change window for the VM migration can be quite large and unpredictable if something goes wrong. This option supports the source vSphere environment to be running 5.5 or later.
The next option is Bulk Migration, which provides customers with greater flexibility on when the migration occurs but more importantly it also helps reduce the size of the change window and the amount of time it takes to migrate the VM. With this option, vSphere Replication (built into HCX Cloud Gateway) is used to proactively replicate the VM(s) cold data to VMWonAWS while the VM is still running. Customers then have the option to either switch over immediately in which the VM will start running on VMWonAWS after the initial sync or they can schedule the switch over at a later date. In the latter scenario, HCX will continuously replicate changes from the VM until the date of the switch over. For customers looking to move hundreds of gigabytes if not several terabytes of VM data, this option ensures that the switch over is a quickly and predictably as possible. vSphere environments running 5.0 or later is supported can be used with this option.
Here is a screenshot of one of my running VMs being migrated using the Bulk Migration option.
Although this article specifically focused on using the migration feature in HCX with VMWonAWS, HCX also supports Layer 2 Network Extension as well which can be used with VMWonAWS. For more information about HCX Cloud and VMWonAWS, be sure to check out their site here.
Vern Bolinius says
Great write-up William, thanks!
Great explanation and informative..
Is there any API or commands to perform HCX operation ?
William Lam says
Yes, both HCX Cloud and HCX Enterprise have REST APIs which you can use to automate everything you would do manually via the UIs
Glad to know about it.
Could you please share any reference documents, would help..
William Lam says
There's currently nothing public, this is something I had asked about as well and I believe the team is working on putting collateral together to show the common workflows and API examples. Depending on time, perhaps I may provide a few examples but without the official API docs, its probably not going to be terribly useful
Thanks much for the information. It would be great if we could see a few early posts, would definitely be excited.
It won't be terrible anyway 🙂
Steven Martin says
Does the bulk migration tool use real-time replication after the initial sync or does it use snapshots? I'm trying to determine if the RPO is near-zero, and therefore suitable for mission-critical apps like SQL.
At least Disaster Recovery is based on snapshots. You can set the RPO in "Disaster Recovery" as low as 5 minutes. Snapshot interval goes as low as one hour, and you can set up to 24 snapshots. But i assume that you'll will probably use the most recent snapshot in case of a DR scenario.
Can we use HCX for migration
From 5.5 to 6.5.
instead of using xvc
Brianna Williams says
Hi, just a question about your role mapping. I see you used your vsphere.local\administrator, but it also seems like you have a customized one. Did you customize a role in vCenter so that they are only able to perform HCX related tasks? If so, how did you do this?
William Lam says
No, HCX only has two default roles afaik "System Admins" and "Enterprise Admins" and you can then assign vCenter Users (which can include AD users that you've got added as an Identity Source) and this would allow you to restrict who can perform what operations given those two default roles
Brianna Williams says
Yes, but since orchestrating migrations takes place within the vCenter, how do you create a role that is restricted to only performing HCX-related tasks within the vCenter?
William Lam says
HCX operations are performed through the HCX Plugin, if you don't have permissions, the user will not be able to initiate any operations. Recommend you give it a try 🙂
Hi, quick question, and an important one pls. If the datastore usage is at 100% and there is no room to grow on the storage array, do we have to start the HCX based replication from scratch on the VMs that are sitting on that datastore or can we use HCX API (if there is one for storage vMotioning the VMs)? I'm not sure if we can recover (failover) the VM to a different datastore either. Any solution would be greatly appreciated.
Pedro Rocha says
Dear Gabe, we are having a tough time putting HCX proximity routing to work. VMware support is insisting that it is not supported when both sites are private clouds both with local DLRs. Have you heard about it? Documentation says anything about it (so I assumed it was supported). But it does not work. After a vMotion, VM loses connectivity. It only works when I do bulk migrations.
Matthieu Paturot says
Hi ! I have a question related to the HCX migration
I have a site under ESXi 6.5 and I would like to create a new site for disaster recovery purposes and take this opportunity to run VMware Cloud Foundation.
I know that the migration is mandatory but can I continue to using the first site under ESXi 6.5 considered legacy afterwards ?
I don't want to lose it.
The final plan is to build a private cloud with two sites :
- the legacy site under ESXi 6.5
- the new site under VMware Cloud Foundation
William Lam says
Once you've migrated your workloads to your new VCF environment, then you can do whatever you want with previous setup 🙂
Matthieu Paturot says
Got it 🙂
I thank you for your fast reply.