If an ESXi host was connected to a Distributed Virtual Switch (DVS) and you were not able to properly remove it from the vCenter inventory, which cleans up the VDS as part of the removal, then you will be left with a stale VDS configuration.
Of course, the ideal workflow is to clean this up within vCenter Server but if you some how lost access or the vCenter Server was deleted and you might be wondering how to manually clean this up?
Earlier this week I needed test something which required a VMware Distributed Virtual Switch (VDS) and this had to be a physical setup, so Nested ESXi was out of the question. I could have used my remote lab, but given what I was testing was a bit "experimental", I prefered using my home lab in the event I need direct console access. At home, I run ESXi on a single Apple Mac Mini and one of the challenges with this and other similar platforms (e.g. Intel NUC) is that they only have a single network interface. As you might have guessed, this is a problem when looking to migrate from a Virtual Standard Switch (VSS) to VDS, as it requires at least two NICS.
Unfortunately, I had no other choice and needed to find a solution. After a couple minutes of searching around the web, I stumbled across this serverfault thread here which provided a partial solution to my problem. In vSphere 5.1, we introduced a new feature which would automatically roll back a network configuration change if it negatively impacted network connectivity to your vCenter Server. This feature could be disabled temporarily by editing the vCenter Server Advanced Setting (config.vpxd.network.rollback) which would allow us to by-pass the single NIC issue, however this does not solve the problem entirely. What ends up happening is that the single pNIC is now associated with the VDS, but the VM portgroups are not migrated and the reason that this is problematic is that the vCenter Server is also running on the ESXi host which it is managing and has now lost network connectivity 🙂
I lost access to my vCenter Server and even though I could connect directly to the ESXi host, I was not able to change the VM Network to the Distributed Virtual Portgroup (DVPG). This is actually an expected behavior and there is an easy work around, let me explain. When you create a DVPG, there are three different bindings: Static, Dynamic, and Ephemeral that can be configured and by default, Static binding is used. Both Static and Dynamic DVPGs can only be managed through vCenter Server and because of this, you can not change the VM network to a non-Ephemeral DVPG and in fact, it is not even listed when connecting to the vSphere C# Client. The simple work around is to create a DVPG using the Ephemeral binding and this will allow you to then change the VM network of your vCenter Server and is the last piece to solving this puzzle.
Disclaimer: This is not officially supported by VMware, please use at your own risk.
Here are the exact steps to take if you wish to migrate an ESXi host with a single NIC from a VSS to VDS and running vCenter Server:
Step 1 - Change the following vCenter Server Advanced Setting config.vpxd.network.rollback to false:
Note: Remember to re-enable this feature once you have completed the migration
Step 2 - Create a new VDS and the associated Portgroups for both your VMkernel interfaces and VM Networks. For the DVPG which will be used for the vCenter Server's VM network, be sure to change the binding to Ephemeral before proceeding with the VDS migration.
Step 3 - Proceed with the normal VDS Migration wizard using the vSphere Web/C# Client and ensure that you perform the correct mappings. Once completed, you should now be able connect directly to the ESXi host using either the vSphere C# Client or ESXi Embedded Host Client to confirm that the VDS migration was successful as seen in the screenshot below.
Note: If you forgot to perform Step 2 (which I initially did), you will need to login to the DCUI of your ESXi host and restore the networking configurations.
Step 4 - The last and final step is to change the VM network for your vCenter Server. In my case, I am using the VCSA and due to a bug I found in the Embedded Host Client, you will need to use the vSphere C# Client to perform this change if you are running VCSA 6.x. If you are running Windows VC or VCSA 5.x, then you can use the Embedded Host Client to modify the VM network to use the new DVPG.
Once you have completed the VM reconfiguration you should now be able to login to your vCenter Server which is now connected to a DVPG running on a VDS which is backed by a single NIC on your ESXi host 😀
There is probably no good use case for this outside of home labs, but I was happy that I found a solution and hopefully this might come in handy for others who might be in a similar situation and would like to use and learn more about VMware VDS.
Over the weekend I received an email from a fellow colleague over in the NSBU who was interested in retrieving statistics on an individual Distributed Virtual Port using the vSphere API and had asked if I had done this before. Although I have written about using the vSphere API to access various properties of a Distributed Virtual Switch (here & here) and migrating from a Virtual Standard Switch (VSS) to Distributed Virtual Switch (VDS) and from a VDS back to VSS (here & here), I have not done a whole lot with the vSphere API with regards to individual Distributed Virtual Ports.
Having said that, with the easy to use PowerCLI Distributed Virtual Switch cmdlets being available, it is actually not too difficult to retrieve this information using the vSphere API. For those of you who have not worked with a Distributed Virtual Switch before, to view the statistics of an individual Distributed Virtual Port, you must drill down into a specific Distributed Virtual Portgroup which you can find using either the vSphere Web/C# Client. Below is a screenshot from my vSphere Web Client.
To retrieve information on a Distributed Virtual Port using the vSphere API, you can refer to the DistributedVirtualPort property. Within this object, there is a state property which you can then drill down further to retrieve statistics using the stat property. If you would like to be able to pull other properties as shown in the screenshot, you can use the runtimeInfo property which provides additional information on the Distributed Virtual Port such as connected entity, MAC Address, VLAN ID, state, etc.
The following two PowerCLI VDS cmdlets: Get-VDPortgroup & Get-VDPort will help us retrieve the information we are looking. However, out of the box these cmdlets do not support pulling statistics for a Distributed Virtual Port. To do so, we will need to access the ExtensionData property which will allows us to access the underlying vSphere API from the PowerCLI object.
Below is a simple PowerCLI script which connects to a vCenter Server and retrieves a specific Distributed Virtual Portgroup (dvPg) that we are interested in using the Get-VDPortgroup cmdlet. Once we have the dvPg, we can then identify a specific Distributed Virtual Port (dvP) by using the Get-VDPort and passing in the dvPg from the previous command as well as the specific port by using the -key property. From here, we can then access a few properties from the dvP and most importantly, the statistics which we were originally interested in. In the example below, I am displaying the entire "stat" property, but you can easily filter out the specific stats you care about.
$server = Connect-VIServer -Server reflex.primp-industries.com # Retrieve a Distributed Portgroup $vdportgroup = Get-VDPortgroup -Name dv-vlan333 # Retrieve a specific Distributed Virtual Port from the Distributed Portgroup using the "key" property $vdport = Get-VDPort -VDPortgroup $vdportgroup -Key 4 # Sample Distributed Virtual Port properties Write-Host "Link Up: " $vdport.ExtensionData.State.RuntimeInfo.Linkup Write-Host "Connectee: " $vdport.ExtensionData.State.RuntimeInfo.LinkPeer Write-Host "Runtime MAC Address: " $vdport.ExtensionData.State.RuntimeInfo.macAddress # Distributed Virtual Port Stats $vdport.ExtensionData.State.Stats Disconnect-VIServer -Server $server -Confirm:$false
Here is an example of the output from the script above and we can see that Distributed Virtual Port "4" matches to what we see in the vSphere Web Client screenshot.
