vSphere – Page 94

Considerations when migrating VMs between vCenter Servers

05.02.2014 by William Lam // 19 Comments

Something that I really enjoy when I get a chance to, is to speak with our field folks and learn a bit more about our customer environments and some of the challenges they are facing. Last week I had quick call with one of our TAMs (Technical Account Managers) regarding the topic of Virtual Machine migration between vCenter Servers. The process of migrating Virtual Machines between two vCenter Servers is not particularly difficult, you simply disconnect the ESXi hosts from one vCenter Server and re-connect to the new vCenter Server. This is something I have performed on several occasions when I was a customer and with some planning it works effortlessly.

However, there are certain scenarios and configurations when migrating VMs between vCenter Servers that could potentially cause Virtual Machine MAC Address collisions. Before we jump into these scenarios, here is some background. By default, a Virtual Machine MAC Address is automatically generated by vCenter Server and the algorithm is based on vCenter Server's unique ID (0-63) among few other parameters which is documented here. If you have more than one vCenter Server, a best practice is to ensure that these VC IDs are different, especially if they are in the same broadcast domain.

As you can imagine, if you have two vCenter Servers that are configured with the same VC ID, there is a possibility that a duplicate MAC Address could be generated. You might think this is probably a rare event given the 65,000 possible MAC Address combinations. However, it actually happens more frequently than you think, especially in very large scale environments and/or Dev/Test for continuous build/integration environments which I have worked in the past and I have personally seen these issues before.

Going back to our vCenter Server migration topic, there are currently two main scenarios that I see occurring in customr environments and we can explore each in a bit more detail and their implications:

Migrate ALL Virtual Machines from old vCenter Server to new vCenter Server
Migrate portion of Virtual Machines from old vCenter Server to new vCenter Server

Migrate all Virtual Machines:

In the diagram above, we have vCenter Server 1 and vCenter Server 2 providing a before/after view. To make things easy, lets say they have VC ID 1 and 2. If we migrate ALL Virtual Machines across, we can see their original MAC Addresses will be preserved as we expect. For any new Virtual Machines being created, the 4th octet of the MAC Address will differ as expected and the vCenter Server will guarantee it is unique. If you want to ensure that new Virtual Machines keep a similar algorithm, you could change the vCenter Server ID to 1. No issues here and the migration is very straight forward.

Migrate A portion of Virtual Machines:

In the second diagram, we still have vCenter Server 1 and vCenter Server with unique VC IDs. However, in this scenario we are only migrating a portion of the Virtual Machines from vCenter Server 1 to vCenter Server 2. By migrating VM2 off of vCenter Server 1, the MAC Address of VM2 is no longer registered with that vCenter Server. What this means is that vCenter Server 1 can potentially re-use that MAC Address when it generates a new request. As you can see from the above diagram, this is a concern because VM2 is still using that MAC Address in vCenter Server 2, but vCenter Server 1 is no longer aware of its existence.

The scenario above is what the TAM was seeing at his customer's site and after understanding the challenge, there are a couple of potential solutions:

Range-Based MAC Address allocation - Allows you to specify a range of MAC Addresses to allocate from which may or may not helpful if the migrated MAC Addresses are truly random
Prefix-Based MAC Address allocation - Allows you to modify the default VMware OUI (00:50:56) which would then ensure no conflicts would be created with previously assigned MAC Addresses. Though this could solve the problem, you potentially could run into collisions with other OUI's within your environment
Leave VMs in a disconnected state - This was actually a solution provided by another TAM on an internal thread which ended up working for his customer. The idea was that instead of disconnecting and removing the ESXi host when migrating a set of Virtual Machines, you would just leave it disconnected in vCenter Server 1. You would still be able to connect the ESXi host and Virtual Machines to vCenter Server 2 but from vCenter Server 1 point of view, the MAC Addresses for those Virtual Machines are seen as in use and it would not be reallocated.

I thought option #3 was a pretty interesting and out of the box solution that the customer came up with. The use case that caused them to see this problem in the first place is due to the way they provision remote environments. The customer has a centralized build environment in which everything is built and then shipped off to the remote sites which is a fairly common practice. Since the centralized vCenter Server is not moving, you can see how previously used MAC Addresses could be re-allocated.

Although option #3 would be the easiest to implement, I am not a fan of seeing so many disconnected systems from an operational perspective as it is hard to tell if there is an issue with the ESXi host and Virtual Machines or because it has been migrated. I guess one way to help with that is to create a Folder called "Migrated" and move all disconnected ESXi hosts into that folder which would help mask that away and disable any alarms for those hosts.

Some additional per-requisite checks that you can perform prior to the partial Virtual Machine migration:

Ensure that the destination vCenter Server is not configured with the same VC ID else you can potentially run into duplicate MAC Address conflicts. You can do this either manually through the vSphere Web/C# Client or leveraging our CLI/API to do so.

Here is an example using PowerCLI to retrieve the vCenter Server ID:

Get-AdvancedSetting -Entity $server -Name instance.id

Ensure no duplicate MAC Addresses exists by comparing the MAC Addresses of the Virtual Machines to be migrated to the Virtual Machines in the new environment. Again, you can either do this by hand (which I would not recommend) or leveraging our CLI/API to extract this information.

Here is an example using PowerCLI to retrieve the MAC Addresses for a Virtual Machine:

Get-VM | Select-Object -Property Name, PowerState, @{"Name"="MAC";"Expression"={($_ | Get-NetworkAdapter).MacAddress}}

If there are other scenarios or solutions that you have seen with Virtual Machine migrations between vCenter Serves, feel free to leave a comment. I am sure others can benefit from past experiences or any other lesson learns.

How to run Nested ESXi on the vCloud Hybrid Service?

05.02.2014 by William Lam // 7 Comments

Today I was granted access to VMware's vCloud Hybrid Service and the first order of business for me of course, was to provision a Nested ESXi VM! After going through the vCHS UI (which is very slick and easy to use by the way) and the vCloud Director UI, I realized the ESXi guestOS type has not been enabled on the backend of the vCloud Director Database. This totally makes sense, as vCHS is a production ready service and they definitely would not want to run anything that is not officially supported.

Having said that, I can see the benefits to customers who would like build out a Nested ESXi environment on vCHS for lab purposes instead of having to manage their own. Some customers even leverage Nested ESXi as part of their development and testing of software and it can be challenging at times to quickly spin up a brand new environment. Instead, they go to vCHS and with just a couple of of clicks in the UI or automatically using the vCloud APIs, provision a couple of Nested ESXi instances for testing. You can easily discard the resources once you are done or keep them running a bit longer.

Having worked with vCloud Director in the past, I knew that you could import an OVF/OVA and I thought maybe I could just import the Nested ESXi OVF templates that I built and potentially workaround vCHS "limitation" 🙂

Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware nor is it supported on vCHS

I tried to upload one of the OVF templates that I built, but it turns out vCloud Director does not supported the Dynamic Disks feature, so I had to perform two additional steps.

Step 1 - Download one of the following Nested ESXi OVF templates

Step 2 - Import the OVF template in an existing vSphere environment and ensure you are doing so using the vSphere Web Client, as some of the properties may not be imported properly

Step 3 - Once deployed, go ahead and re-export the image to an OVF/OVA (I choose OVA as it is a single file) and this will generate the empty VMDKs for you so the image should still be very small (< 1MB)

Step 4 - Login into to your vCHS account and click on your Virtual Datacenter. Select Virtual Machines and then click on Manage in vCloud Director. Import the OVF/OVA that you have just exported

Step 5 - Once the import has been completed, you now have a Virtual Machine that has been configured with the correct guestOS type which should be VMware ESXi 5.x as seen in the screenshot below

Step 6 - At this point, you can either mount an ESXi ISO over your browser or upload it into the vCloud Director Catalog so you can mount it locally and begin your installation of ESXi. Below is a screenshot of 3 Nested ESXi VMs running on vCHS

Note: It looks like some of the advanced VM settings that are part of my OVF template are ignored as part of the vCloud Director import. This means that if you would like to run a Nested VSAN environment on vCHS, you will not be able to rely on the SSD emulation setting but instead, you will need to run through the ESXCLI claim rules to mark particular disks as "SSD" devices. It would have been really nice if vCloud Director would preserve all the advanced VM settings but at least you can still run a Nested VSAN environment.

So there you have it, Nested ESXi running on vCHS! I am kind of curious if this is the first instance of a Nested ESXi VM running on vCHS without having admin access on the backend system?

Note: One limitation to be aware of is that since the backend of vCloud Director is not properly enabled for Nested Virtualization support, this means you will NOT be able to run Nested VMs on top of the Nested ESXi instances. This is due to the lack of having Network Pool which has both Promiscuous & Forge Transmits enabled which is a requirement for proper Nested VMs connectivity. I wonder if vCHS should provide Nested Virtualization capabilities? I know I definitely would like to see it, what do you think? Leave a comment if you have some thoughts on this topic.

UPDATE (05/4/14) - If you wish to run a Nested VSAN environment on vCHS, you will need to take a look at this blog post here on how to "fake" an SSD on one of the devices by using ESXCLI claim rules. The rason for this is that you will not be able to leverage the other method of emulating an SSD device via advanced setting as that requires access to the underlying vSphere environment which you will not have in vCHS.

Other handy vSphere VOBs for creating vCenter Alarms

04.24.2014 by William Lam // 8 Comments

Lately I have been writing on a variety of topics regarding the use of VOBs (VMkernel Observations) for creating useful vCenter Alarms such as:

I figure it would also be useful to collect a list of all the vSphere VOBs, at least from what I can gather by looking at /usr/lib/vmware/hostd/extensions/hostdiag/locale/en/event.vmsg on the latest version of ESXi. The list below is quite extensive, there are a total of 308 vSphere VOBs not including the VSAN VOBs in my previous articles. For those those of you who use vSphere Replication, you may also find a couple of handy ones in the list.

[Read more...]