mac address – Page 2

Duplicate MAC Address concerns with xVC-vMotion in vSphere 6.0

03.05.2015 by William Lam // 4 Comments

In vSphere 6.0, the mobility options for a Virtual Machine is truly limitless. This has all been possible with a new set of vMotion capabilities introduced in vSphere 6.0 which you can learn more about them here and here. In the past, one area of concern when migrating a VM from one vCenter Server to another is the possibility that a migrated VM's MAC Address might be re-provisioned by the source vCenter Server resulting in a MAC Address conflict. In fact, this is actually a topic I have covered before in my considerations when migrating VMs between vCenter Servers article. I highly encourage you check out that article before proceeding further as it provides some additional and necessary context.

When looking to leverage the new Cross vCenter Server vMotion (xVC-vMotion) capability in vSphere 6.0, are MAC Address conflicts still a concern? To answer that question, lets take a look at an example. Below I have a diagram depicting two different vSphere 6.0 deployments. The first is comprised of three vCenter Servers who are joined to the same SSO Domain called vghetto.local and VM1 is currently being managed by VC1. The second is a single vCenter Server connected to a completely different SSO Domain called vmware.local. I will also assume we are being a good VI Admin and we have deployed each vCenter Server using a unique ID (more details here on why having different VC ID matters).

Lets say we now migrate VM1 from VC1 to VC2. In previous releases of vSphere, this potentially could lead to VC1 re-provisioning the MAC Address that VM1 was associated with because that MAC Address was no longer being managed by VC1 and from its point of view, it is now available. Though this type of a scenario is probably rare in most customer environments, in a high churn continuous integration or continuous delivery environment, this can be a real issue. So has anything been improved in vSphere 6.0? The answer is yes, of course 🙂

In vSphere 6.0, vCenter Server now maintains a VM MAC Address Blacklist which upon a successful xVC-vMotion will update this blacklist with the MAC Addresses associated with the migrated VM. This ensures that the source vCenter Server will not re-provisioned these MAC Addresses to newly created VMs and these MAC Addresses are basically "blacklisted" from being used again as shown in the diagram below.

If we decide to migrate VM1 from VC2 back to VC1, the blacklist is automatically updated and "blacklisted" MAC Addresses will be removed. If we decide to migrate VM1 to a completely different vCenter Server which is not part of the same SSO Domain, ~~then the MAC Address could potentially be re-used~~, but it will depend on your environment if VC4 is on a completely different L2 segment, then a MAC Address conflict would not occur.

As of right now, there is no automatic way of reclaiming blacklisted MAC Addresses, it is a manual process that must be initiated through a private vSphere API. I am hoping we will be able to get this documented in an official VMware KB, so that in case this is required, you can easily follow the simple steps to execute the necessary APIs. Automatic reclamation is being looked at by Engineering and hopefully we will see this in a future patch/update in vSphere. Overall, this should should not really be a concern given that vCenter Server can uniquely generate about 65,000 unique MAC Addresses and you would have to perform quite a few xVC-vMotions before ever needing to reclaim from the blacklist.

One thing to be aware of when performing xVC-vMotion or ExVC-vMotion is that there are currently no pre-flight checks for MAC Address conflicts at the destination vCenter Server (something Engineering is looking update in a future patch/update release). Having said that, there are two additional measures you can implement in you environment to prevent MAC Address conflicts:

Create vCenter Server alarm which can detect and notify you of a duplicate MAC Address in you environment (also applicable to vSphere 5.5)
Pro-actively check to see if the existing MAC Addresses of your VM is currently in use prior to performing a xVC-vMotion, this is especially useful when performing ExVC-vMotion.

To help with with number 2, I have created a simple PowerCLI script called check-vm-mac-conflict.ps1 which accepts both your source and destination vCenter Server as well as the name of the VM in the source VC to be migrated. It will check the VM's MAC Addresses in the destination VC and ensure that there are no conflicts. If there is a conflict, it will output the name of the destination VM and the MAC Address that is in conflict as seen in the screenshot below.

Hopefully with these additional measures, you can easily prevent MAC Address conflicts when performing xVC-vMotions in your vSphere environment which can be a pain to troubleshoot.

Detecting duplicate VM MAC Address using vCenter Server Alarm

02.25.2015 by William Lam // 6 Comments

Having a duplicate VM MAC Address in your environment can lead to an extremely painful day of troubleshooting and it can also be tough to prevent depending on how and where you provision your VMs.

There are two cases that I can think of where a duplicate MAC Address can potentially occur:

You manually assign a static MAC Address versus using dynamic assignment (includes VM import) and it conflicts with an already assigned MAC Address
You migrate a VM from one vCenter Server to another and the destination vCenter Server has already assigned the MAC Address of the migrated VM

In both of these scenarios, when a duplicate MAC Address occurs, time is of the essence to quickly pin-point the source of the duplicated entry and quickly resolving the conflict. What would be nice is to be able to automatically detect that a MAC Address conflict has occurred and provide the necessary information of the offending VMs.

UPDATE (4/22) - Thanks to Petr, it turns out there is another MAC Address conflict event which I did not know about specifically for detecting duplicate entries for manually assigned MAC Addresses called "VM static MAC conflict". I definitely recommend creating an alarm for both Events for the vCenter Alarm.

While performing some research in my lab environment the other day, I accidentally stumbled onto this little tidbit in vCenter Server. It turns out there is an out of the box event called "VM Mac conflict" which can be triggered using a vCenter Server alarm when a duplicated MAC Address is detected for a VM. I was actually surprised that this was not one of the pre-created default alarms in vCenter Server as I can see this being extremely useful to have out of the box. In any case, it is simple enough to create a new vCenter Server Alarm and in the example below I called it "Dupe VM Mac Address".

To test our new alarm, I created a new VM called "VM1" which has been configured with static MAC Address that matches "VM2". Once the VM has been created, we can see that the alarm is immediately triggered and by clicking into the alarm details, it provides the details of the MAC Address and the offending VMs.

In my opinion this is an alarm that everyone should create in their environment to ensure that if this problem ever occurs, you can quickly get notified and resolve the problem. I have also reported this internally and asked if we can have this alarm created by default, so hopefully this will not be necessary in the near future 🙂

Considerations when migrating VMs between vCenter Servers

05.02.2014 by William Lam // 19 Comments

Something that I really enjoy when I get a chance to, is to speak with our field folks and learn a bit more about our customer environments and some of the challenges they are facing. Last week I had quick call with one of our TAMs (Technical Account Managers) regarding the topic of Virtual Machine migration between vCenter Servers. The process of migrating Virtual Machines between two vCenter Servers is not particularly difficult, you simply disconnect the ESXi hosts from one vCenter Server and re-connect to the new vCenter Server. This is something I have performed on several occasions when I was a customer and with some planning it works effortlessly.

However, there are certain scenarios and configurations when migrating VMs between vCenter Servers that could potentially cause Virtual Machine MAC Address collisions. Before we jump into these scenarios, here is some background. By default, a Virtual Machine MAC Address is automatically generated by vCenter Server and the algorithm is based on vCenter Server's unique ID (0-63) among few other parameters which is documented here. If you have more than one vCenter Server, a best practice is to ensure that these VC IDs are different, especially if they are in the same broadcast domain.

As you can imagine, if you have two vCenter Servers that are configured with the same VC ID, there is a possibility that a duplicate MAC Address could be generated. You might think this is probably a rare event given the 65,000 possible MAC Address combinations. However, it actually happens more frequently than you think, especially in very large scale environments and/or Dev/Test for continuous build/integration environments which I have worked in the past and I have personally seen these issues before.

Going back to our vCenter Server migration topic, there are currently two main scenarios that I see occurring in customr environments and we can explore each in a bit more detail and their implications:

Migrate ALL Virtual Machines from old vCenter Server to new vCenter Server
Migrate portion of Virtual Machines from old vCenter Server to new vCenter Server

Migrate all Virtual Machines:

In the diagram above, we have vCenter Server 1 and vCenter Server 2 providing a before/after view. To make things easy, lets say they have VC ID 1 and 2. If we migrate ALL Virtual Machines across, we can see their original MAC Addresses will be preserved as we expect. For any new Virtual Machines being created, the 4th octet of the MAC Address will differ as expected and the vCenter Server will guarantee it is unique. If you want to ensure that new Virtual Machines keep a similar algorithm, you could change the vCenter Server ID to 1. No issues here and the migration is very straight forward.

Migrate A portion of Virtual Machines:

In the second diagram, we still have vCenter Server 1 and vCenter Server with unique VC IDs. However, in this scenario we are only migrating a portion of the Virtual Machines from vCenter Server 1 to vCenter Server 2. By migrating VM2 off of vCenter Server 1, the MAC Address of VM2 is no longer registered with that vCenter Server. What this means is that vCenter Server 1 can potentially re-use that MAC Address when it generates a new request. As you can see from the above diagram, this is a concern because VM2 is still using that MAC Address in vCenter Server 2, but vCenter Server 1 is no longer aware of its existence.

The scenario above is what the TAM was seeing at his customer's site and after understanding the challenge, there are a couple of potential solutions:

Range-Based MAC Address allocation - Allows you to specify a range of MAC Addresses to allocate from which may or may not helpful if the migrated MAC Addresses are truly random
Prefix-Based MAC Address allocation - Allows you to modify the default VMware OUI (00:50:56) which would then ensure no conflicts would be created with previously assigned MAC Addresses. Though this could solve the problem, you potentially could run into collisions with other OUI's within your environment
Leave VMs in a disconnected state - This was actually a solution provided by another TAM on an internal thread which ended up working for his customer. The idea was that instead of disconnecting and removing the ESXi host when migrating a set of Virtual Machines, you would just leave it disconnected in vCenter Server 1. You would still be able to connect the ESXi host and Virtual Machines to vCenter Server 2 but from vCenter Server 1 point of view, the MAC Addresses for those Virtual Machines are seen as in use and it would not be reallocated.

I thought option #3 was a pretty interesting and out of the box solution that the customer came up with. The use case that caused them to see this problem in the first place is due to the way they provision remote environments. The customer has a centralized build environment in which everything is built and then shipped off to the remote sites which is a fairly common practice. Since the centralized vCenter Server is not moving, you can see how previously used MAC Addresses could be re-allocated.

Although option #3 would be the easiest to implement, I am not a fan of seeing so many disconnected systems from an operational perspective as it is hard to tell if there is an issue with the ESXi host and Virtual Machines or because it has been migrated. I guess one way to help with that is to create a Folder called "Migrated" and move all disconnected ESXi hosts into that folder which would help mask that away and disable any alarms for those hosts.

Some additional per-requisite checks that you can perform prior to the partial Virtual Machine migration:

Ensure that the destination vCenter Server is not configured with the same VC ID else you can potentially run into duplicate MAC Address conflicts. You can do this either manually through the vSphere Web/C# Client or leveraging our CLI/API to do so.

Here is an example using PowerCLI to retrieve the vCenter Server ID:

Get-AdvancedSetting -Entity $server -Name instance.id

Ensure no duplicate MAC Addresses exists by comparing the MAC Addresses of the Virtual Machines to be migrated to the Virtual Machines in the new environment. Again, you can either do this by hand (which I would not recommend) or leveraging our CLI/API to extract this information.

Here is an example using PowerCLI to retrieve the MAC Addresses for a Virtual Machine:

Get-VM | Select-Object -Property Name, PowerState, @{"Name"="MAC";"Expression"={($_ | Get-NetworkAdapter).MacAddress}}

If there are other scenarios or solutions that you have seen with Virtual Machine migrations between vCenter Serves, feel free to leave a comment. I am sure others can benefit from past experiences or any other lesson learns.