WilliamLam.com

ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

by // 23 Comments

The ESXi MAC Learn dvFilter Fling was released a little over two years ago and it has become a must have when it comes to running our ESXi Hypervisor within a VM, also referred to as Nested ESXi. The reason this Fling has become such a popular hit amongst our customers and partners is that it greatly improves the performance when “Promiscuous Mode” is enabled on a Virtual or Distributed Virtual Portgroup, which is a requirement for using Nested ESXi. Although this Fling works great, there are a couple of limitations with this solution today. The first of which is called out in the original Fling release notes, that once a MAC Address has been learned, it never ages out which is not ideal for long running Nested ESXi environments that generates a large amount of new MAC Addresses. The second, is the lack of vMotion support where the learned MAC Address table is not transfered to the destination ESXi host and must be re-learned.

To help address both of these limitations, the folks over in the Network and Security Business Unit (NSBU) have been working hard to improve upon the existing solution and have developed a new native MAC Learning VMkernel module called the Learnswitch. This new Learnswitch not only helps improves Nested ESXi workloads but it can also potentially benefit other workloads such as Nested Containers or other 3rd Party network inspection software. One immediate difference from the previous MAC Learn dvFilter solution is that rather than operating on the Network IO Chain, the filtering is now performed within the outer virtual switch layer itself which will provide some additional performance gains. The other added benefit from an internal VMware standpoint is that the Learnswitch is now vmkapi compatible, which means we will have a better backwards compatible story for supporting old releases of ESXi. One downside to this new solution compared to the previous one is that because the dvFilter operated below the virtual switch layer, it could support both a Virtual Standard Switch as well as the Distributed Virtual Switch. With the new Learnswitch, a Distributed Virtual Switch will be required. If you currently do not meet the requirements of the new Learnswitch, you can continue using the dvFilter, but it is recommended that you do not mix both on a single system but you can definitely make use of both solutions across different ESXi hosts depending on the constraints of your environment.

Here are some of the new capabilities provided by the new Learnswitch module:

  • Overlay Network based that learning and filtering are done in Etherswitch forwarding check
  • MAC Address learning is based on VLAN ID or VXLAN ID on uplink and leaf port
  • Packet is filtered on uplink and leaf port if the MAC is learned on a different port
  • MAC Address table size is 32k per system
  • MAC Address aging support with default aging time of 5 minutes and configurable
  • Unknown unicast packet is flooded by default and configurable to drop
  • vMotion support that the MAC table learned on the port is transferred to destination host and RARP packet is sent
  • Standalone VMkernel module available as a VIB
  • net-learnswitch CLI to display MAC Address table, configuration and stats

[Read more...]

Potential ESXi Host Preparation issues with NSX 6.3

by // 16 Comments

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.


There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

[Read more...]

Unable to apply VSAN VM Storage Policy on NSX Controller/Edge VMs?

by // 6 Comments

This post was inspired by a recent Twitter conversation with Joep Piscaer who ran into an interesting challenge with VSAN and NSX.

The issue that Joep encountered was not being able to apply a VSAN VM Storage Policy onto an NSX Controller VM which resided on a VSAN Datastore. Below is a screenshot of the error message "The method is disabled by vShield_SVM" if you tried to apply the VM Storage Policy.

disabled_methods_on_vms_1
The reason Joep is seeing this error is because the NSX Controller VM is a special "Service VM" that is being managed by a specific solution, in this case it is VMware NSX. To ensure that users do not accidentally modify these "Service VMs", certain set of functionality has been disabled on these VMs from regular users. Any configuration changes that are required are initiated through the solution itself which has full administrative access to these VMs. This issue is actually not specific to the NSX Controller VMs but also applies to the NSX Edge VMs: ESR (Edge Service Router) and DLR (Distributed Logical Router).

In fact, this applies to any "Service VMs" which are being managed by a VMware Solution or 3rd party Solution. You will notice that you will not be able to edit these VMs like you would normally on other VMs. One suggestion from a community member was to check out the VMware KB 2008957 which has users manually tweaking the VCDB, which I am not a big fan of if I can help it. If you want to know why I do not recommend this, check out this post here where kittens might get harmed. Now, getting back to Joep's request, is there a solution for him? After all, his request is a valid one where he has deployed an NSX Controller VM on a VSAN Datastore and wishes to apply a specific VSAN VM Storage Policy.

Luckily, there is cleaner work around that does not involve messing around with the VCDB and crafting ugly SQL queries. The way these "methods" or operations are disabled on a particular set of VMs is through the use of a private vSphere API available through vCenter Server called disableMethods. You can actually view the list of disableMethods by viewing a particular VM using the vSphere MOB under config->disableMethod property as seen in the screenshot below.

disabled_methods_on_vms_2
The list of disableMethods map to the specific vSphere API calls for a VM and in the case of modifying a VM which includes applying a VM Storage Policy, the method that is used is is called the ReconfigVM_Task which we can see in the screenshot mapping to vim.VirtualMachine.reconfigure. If we want to be able to apply a VM Storage Policy, we simply just need to temporarily remove this particular operation from the disabbleMethods list.

Here are the instructions for enabling ReconfigVM_Task method:

Step 1 - You will need to find the MoRef (Managed Object Reference) ID of the VM that you wish to enable the method on. You can do this by either browsing through the vSphere MOB, using this vSphere SDK for Perl script or this PowerCLI snippet:

Get-VM -Name [VM-NAME] | ft -Property Id

Step 2 - Open a web browser to the following URL:

https://[VC-IP]/mob/?moid=AuthorizationManager&method=enableMethods

Step 3 - You will need to replace the following two parameters (make sure to replace the VM MoRef ID with the one you found in Step 1):

parameter value
entity <entity type="ManagedEntity" xsi:type="ManagedObjectReference">vm-35</entity>
method <method>ReconfigVM_Task</method>

Step 4 - Once you have updated fields as shown in the screenshot below, to execute the API call you just need to click on the "Invoke Method" link on the bottom right.

disabled_methods_on_vms_3
If everything was successful, you should see some output from the operation listing the methods that are still currently disabled. You can also confirm that everything is working by refreshing the vSphere Web Client or if you are using the vSphere C# Client, the "Edit Settings" option should now be available. Lastly, if I now apply a VSAN VM Storage Policy, I will no longer get the error and as you can see from the screenshot below, I now have successfully applied my "VSAN-Platinum-VM-Storage-Policy" for my NSX Controller VM. I would strongly recommend that you re-enable the original disable method by following the instructions below.

disabled_methods_on_vms_4

Here are the instructions for disabling ReconfigVM_Task method:

Step 1 - You will need to find the MoRef (Managed Object Reference) ID of the VM that you wish to enable the method on. You can do this by following Step 1 from the enable instructions

Step 2 - Open a web browser to the following URL:

https://[VC-IP]/mob/?moid=AuthorizationManager&method=disableMethods

Step 3 - You will need to replace the following three parameters (make sure to replace the VM MoRef ID with the one you found in Step 1 and any random number works for sourceId property):

parameter value
entity <entity type="ManagedEntity" xsi:type="ManagedObjectReference">vm-35</entity>
method <DisabledMethodRequest><method>ReconfigVM_Task</method><DisabledMethodRequest>
sourceId 1234

Step 4 - Once you have updated fields as shown in the screenshot below, to execute the API call you just need to click on the "Invoke Method" link on the bottom right.

disabled_methods_on_vms_5
If everything was successful, you should see a void output and if you go to your vSphere Web/C# Client, you should see that the "Edit Settings" option is now disabled again for this VM. Though applying a VSAN VM Storage Policy is pretty trivial, there are some additional things to be aware of when working with special "Service VMs" like the NSX Controller. Hopefully we can improve this workflow in the future and provide for a better user experience but in the mean time, you can use this workaround.