WilliamLam.com

NSX

vMotion across different VDS version between onPrem and VMC

by 13 Comments

For those of you who have attempted a vMotion (whether that is within a vCenter Server or between different vCenter Servers (including across SSO Domains), if the VM is running on a Distributed Virtual Switch (VDS) and the version of the VDS is not the same between the source and destination (VDS 6.5 to VDS 6.7), the operation will fail with the following error message (UI and API):

Currently connected network interface 'Network adapter 1' cannot use network 'DVPG-VM-Network (VDS-67)', because the destination distributed switch has a different version or vendor than the source distributed switch.


This behavior is no different on VMware Cloud on AWS (VMC) or at least, I thought it was, until I recently learned about a really neat feature that was introduced in the VMC 1.4p2 release, here is a snippet from the release notes:

Cross VDS version vMotion Compatibility
With this advanced configuration option enabled, bi-directional vMotion between on-premises and VMware Cloud on AWS can be achieved across different virtual distributed switch (VDS) versions (greater than or equal to version 6.0). This must be enabled on the on-premises vCenter.

It turns out there is actually a way to allow vMotions across different VDS versions, this is important for VMC because the software stack will always be using a newer version than what we ship to our onPrem customers. However, due to this limitation, we could not benefit from the latest VDS version but had to default it to VDS 6.0 to ensure that customers could migrate their workloads. The advanced setting mentioned in the release notes allows us to disable the strict compatibility check which is performed on the destination vCenter Server when a vMotion is initiated, this setting is now enabled by default on the VMC vCenter Server which is why you can perform migration across different VDS without having to do anything special on your onPrem vCenter Server.

UPDATE (01/02/21) - If you are running vSphere 7.x, an additional advanced setting must be configured called config.vmprov.enableHybridMode and set the value to true. For more details, you can refer to this VMware KB 79446. Thanks to reader Marc Alumbaugh for sharing this finding!

UPDATE (10/16/18) - With the release of vSphere 6.7 Update 1, customers can now also vMotion VMs from on-prem running on a VDS to VMC with NSX-T N-VDS.

[Read more...] about vMotion across different VDS version between onPrem and VMC

New SDDC Certificate Replacement Fling

by 9 Comments

Certificate lifecycle management is not something anyone looks forward to, it is time consuming and usually not automated. However, it is a necessity for many of our customers. The process gets even more challenging when needing replace certificates across multiple VMware products, not only careful orchestration but also properly reestablishing trust between product just adds another layer of operational complexity. Within the Integrated System Business Unit (ISBU) at VMware, which produces both the VMware Validated Design (VVD) and VMware Cloud Foundation (VCF), the team has been working on a way to simplify certificate management, not only for individual products (working with product teams) but also holistically at the VMware SDDC level.

This initially started with the development of a tool called Certificate Generation Utility (CertGen), which helps customers generate new certificates for various products within the VMware SDDC. Although it was developed for the VVD, any VMware customer who consumed products within the VVD, could also leverage this tool. We all know certificate generation can be a pain, but it is not as challenging or as complex as the actual certificate replacement process itself which is also fully documented by the VVD team here.

This is where the new Fling comes in, the SDDC Certificate Tool, which automates the manual steps outlined by the VVD and helps customers easily replace certificates that they have created (CertGen or another process) and automatically orchestrates this across the different products within the SDDC. The tool is command-line driven and uses a JSON configuration file which can contain all or a subset of the VMware SDDC products, which is great for supporting different environments and allows for easy source control. Extensive pre-checks are also built into the tool to validate the certificates themselves (e.g. expiry, chain validation, etc) also also preventing miss-match of information (e.g. SAN entries, number of nodes, etc) which then get compared against your actual environment before any changes are applied. The JSON also contains a section referred to as Service Accounts, which is merely other VMware product accounts that the tool supports to reestablish trust after replacing the certificate for given product. 

[Read more...] about New SDDC Certificate Replacement Fling

Quick Tip - How do I tell if NSX-V or NSX-T is installed?

by 1 Comment

This question came up last week asking for a programmatic method to identify whether NSX-V or NSX-T is deployed in your environment. With NSX-V, vCenter Server is a requirement but for NSX-T, vCenter Server is not a requirement, especially for multi-hypervisor support. In this post, I will assume for NSX-T deployments, you have configured a vCenter Compute Manager.

Both NSX-V and NSX-T uses the ExtensionManager API to register themselves with vCenter Server and we can leverage this interface to easily tell if either solutions are installed. NSX-V uses the com.vmware.vShieldManager string to identify itself and NSX-T uses the com.vmware.nsx.management.nsxt string to identify itself.

Here is a quick PowerCLI snippet that demonstrates the use of the vSphere API to check whether NSX-V or NSX-T is installed and provides the version shown in the registration:

Here is a screenshot from my environment which has both NSX-V (6.4) and NSX-T (2.1) installed:


Note: Due to some current testing, I have not upgraded my NSX-T deployment to the latest 2.2 release, so I do not know if the version gets bumped to match the actual released version

VPN Configuration to VMware Cloud on AWS using pfSense

by 1 Comment

Provisioning a new SDDC on VMware Cloud on AWS (VMC) is not an operation that I perform on a regular basis. Usually, one of the first tasks after a new SDDC deployment is setting up a VPN connection between your on-premises datacenter and your VMC environment. Given this is not a frequent activity, I always forget the specific configurations required for my particular VPN solution and figure I would document this for myself in the future as well as anyone else who might also have a simliar setup.

Since the VMC Gateways are just NSX-v Edges, any VPN solution that supports the NSX-v configurations will also work with VMC. In my environment, I am using pfSense which is a popular and free security Virtual Appliance that many folks run in their VMware home lab. Before getting started, it is also important to note that there are two gateway endpoints that you can setup separate VPN connections to. The first is the Management Gateway which provides access to the management infrastructure such vCenter Server, NSX and ESXi hosts and the second is the Compute Gateway which provide access to the VM workloads running within VMC. Since the instructions are exactly the same for setting up the VPN for either gateways, I am just going over the Management Gateway configuration and where applicable, I will note the minor differences.

Step 1 - Login to the VMC Portal (vmc.vmware.com) and select one of your deployed SDDCs. Click on the Network tab and you should be taken to a page like the one shown in the screenshot below. Here is where you will be applying your VPN configuration from the VMC side. Start off by making a note of the public IP Address for the Management Gateway (highlighted in yellow), this will needed when configuring the VPN configuration on the on-prem side. It is probably a good idea to also note down the Compute Gateway IP Address if you plan on configuring that as well.


[Read more...] about VPN Configuration to VMware Cloud on AWS using pfSense

ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

by 22 Comments

The ESXi MAC Learn dvFilter Fling was released a little over two years ago and it has become a must have when it comes to running our ESXi Hypervisor within a VM, also referred to as Nested ESXi. The reason this Fling has become such a popular hit amongst our customers and partners is that it greatly improves the performance when “Promiscuous Mode” is enabled on a Virtual or Distributed Virtual Portgroup, which is a requirement for using Nested ESXi. Although this Fling works great, there are a couple of limitations with this solution today. The first of which is called out in the original Fling release notes, that once a MAC Address has been learned, it never ages out which is not ideal for long running Nested ESXi environments that generates a large amount of new MAC Addresses. The second, is the lack of vMotion support where the learned MAC Address table is not transfered to the destination ESXi host and must be re-learned.

To help address both of these limitations, the folks over in the Network and Security Business Unit (NSBU) have been working hard to improve upon the existing solution and have developed a new native MAC Learning VMkernel module called the Learnswitch. This new Learnswitch not only helps improves Nested ESXi workloads but it can also potentially benefit other workloads such as Nested Containers or other 3rd Party network inspection software. One immediate difference from the previous MAC Learn dvFilter solution is that rather than operating on the Network IO Chain, the filtering is now performed within the outer virtual switch layer itself which will provide some additional performance gains. The other added benefit from an internal VMware standpoint is that the Learnswitch is now vmkapi compatible, which means we will have a better backwards compatible story for supporting old releases of ESXi. One downside to this new solution compared to the previous one is that because the dvFilter operated below the virtual switch layer, it could support both a Virtual Standard Switch as well as the Distributed Virtual Switch. With the new Learnswitch, a Distributed Virtual Switch will be required. If you currently do not meet the requirements of the new Learnswitch, you can continue using the dvFilter, but it is recommended that you do not mix both on a single system but you can definitely make use of both solutions across different ESXi hosts depending on the constraints of your environment.

Here are some of the new capabilities provided by the new Learnswitch module:

  • Overlay Network based that learning and filtering are done in Etherswitch forwarding check
  • MAC Address learning is based on VLAN ID or VXLAN ID on uplink and leaf port
  • Packet is filtered on uplink and leaf port if the MAC is learned on a different port
  • MAC Address table size is 32k per system
  • MAC Address aging support with default aging time of 5 minutes and configurable
  • Unknown unicast packet is flooded by default and configurable to drop
  • vMotion support that the MAC table learned on the port is transferred to destination host and RARP packet is sent
  • Standalone VMkernel module available as a VIB
  • net-learnswitch CLI to display MAC Address table, configuration and stats

[Read more...] about ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

Potential ESXi Host Preparation issues with NSX 6.3

by 16 Comments

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.


There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

[Read more...] about Potential ESXi Host Preparation issues with NSX 6.3