Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

Nested Virtualization is an extremely useful tool that helps customers easily test and try out new VMware products and solutions before rolling that into a proper development environment for further validation. This is especially handy for those wanting to setup an NSX-based environment and simulating their actual deployment topology, configuration and upgrade workflows.

In this past year, I have seen a 10x increase in the number of NSX-T based questions that have come up from customers and our field, the adoption of NSX-T is definitely in full swing. As expected, questions about running Nested ESXi on top of a physical NSX-T deployment has come up and there has actually been several variations that have been asked about whether that is Nested ESXi using VSS, VDS, NSX-V or even NSX-T running on top of an N-VDS, which is the virtual switch that NSX-T uses.

Luckily all of these combinations work and just require some basic configuration changes within NSX-T. However, before I continue, let me remind folks again that VMware does NOT officially support Nested Virtualization.

As a refresher, if you are just using a VSS or VDS with no NSX-V or NSX-T, you no longer need to enable Promiscuous Mode. For VSS, you can use ESXi MAC Learn dvFilter Fling and for VDS as of vSphere 6.7 and can simply take advantage of the new native MAC Learning feature. With NSX-T, we can also take advantage of MAC Learning but the changes must now be applied within NSX-T Manager as it is responsible for managing the virtual switch versus vCenter Server. In both the configurations mentioned below, you are making the changes to your physical NSX-T deployment, not the Nested layer.

VSS/VDS or NSX-V on NSX-T

MAC Learning in NSX-T is configured as part of the MAC Discovery Policy for a Network Segment Profile which can be found under the Segments tab in the NSX-T Policy UI. By default, MAC Learning is disabled in the default MAC Discovery Profile which is applied to all Network Segments that are created if not overridden by a user. To enable MAC Learning, simply create a new Segment Profile and enable the MAC Learning feature which we will use to override when creating a new Network Segment.

Now when you create a new Network Segment, in the MAC Discovery configuration, we can change profile to point to our MAC Learning enabled Profile and now you are ready to deploy Nested ESXi VMs that can be configured with VSS, VDS or NSX-V running on top of an N-VDS.

Nested NSX-T on NSX-T

In addition to enabling MAC Learning in your physical NSX-T deployment, if you wish to run NSX-T on top of physical NSX-T deployment, you will need to make one additional change when creating your Overlay Transport Zone which is only possible when using the NSX-T API. The property that needs to be added is called nested_nsx which is boolean value and ensures that networking will function properly when running a Nested NSX-T instance on top of your physical NSX-T deployment.

The easiest way to configure this is using PowerCLI (but you can certainly use the NSX-T REST API) and below is the snippet required to create a Transport Zone with the required setting:

Connect-NsxtServer -Server nsx.primp-industries.com

$transportZoneService = Get-NsxtService -Name "com.vmware.nsx.transport_zones"
$overlayTZSpec = $transportZoneService.help.create.transport_zone.Create()
$overlayTZSpec.display_name = "TZ-Nested-NSX-T"
$overlayTZSpec.host_switch_name = "nsxswitch"
$overlayTZSpec.transport_type = "OVERLAY"
$overlayTZSpec.nested_nsx = "true"
$transportZoneService.create($overlayTZSpec)

If the operation was successful, you should get a new Transport Zone created with the required setting and you can setup your Nested NSX-T instance.