NSX – Page 3

Potential ESXi Host Preparation issues with NSX 6.3

02.17.2017 by William Lam // 16 Comments

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.

There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

[Read more...]

Unable to apply VSAN VM Storage Policy on NSX Controller/Edge VMs?

01.07.2015 by William Lam // 6 Comments

This post was inspired by a recent Twitter conversation with Joep Piscaer who ran into an interesting challenge with VSAN and NSX.

I want to apply a VSAN VM Storage Policy to a NSX Controller, but I’m getting an "The method is disabled by ‘vShield_SVM’” error. Any ideas?

— Joep Piscaer (@jpiscaer) December 31, 2014

The issue that Joep encountered was not being able to apply a VSAN VM Storage Policy onto an NSX Controller VM which resided on a VSAN Datastore. Below is a screenshot of the error message "The method is disabled by vShield_SVM" if you tried to apply the VM Storage Policy.

The reason Joep is seeing this error is because the NSX Controller VM is a special "Service VM" that is being managed by a specific solution, in this case it is VMware NSX. To ensure that users do not accidentally modify these "Service VMs", certain set of functionality has been disabled on these VMs from regular users. Any configuration changes that are required are initiated through the solution itself which has full administrative access to these VMs. This issue is actually not specific to the NSX Controller VMs but also applies to the NSX Edge VMs: ESR (Edge Service Router) and DLR (Distributed Logical Router).

In fact, this applies to any "Service VMs" which are being managed by a VMware Solution or 3rd party Solution. You will notice that you will not be able to edit these VMs like you would normally on other VMs. One suggestion from a community member was to check out the VMware KB 2008957 which has users manually tweaking the VCDB, which I am not a big fan of if I can help it. If you want to know why I do not recommend this, check out this post here where kittens might get harmed. Now, getting back to Joep's request, is there a solution for him? After all, his request is a valid one where he has deployed an NSX Controller VM on a VSAN Datastore and wishes to apply a specific VSAN VM Storage Policy.

Luckily, there is cleaner work around that does not involve messing around with the VCDB and crafting ugly SQL queries. The way these "methods" or operations are disabled on a particular set of VMs is through the use of a private vSphere API available through vCenter Server called disableMethods. You can actually view the list of disableMethods by viewing a particular VM using the vSphere MOB under config->disableMethod property as seen in the screenshot below.

The list of disableMethods map to the specific vSphere API calls for a VM and in the case of modifying a VM which includes applying a VM Storage Policy, the method that is used is is called the ReconfigVM_Task which we can see in the screenshot mapping to vim.VirtualMachine.reconfigure. If we want to be able to apply a VM Storage Policy, we simply just need to temporarily remove this particular operation from the disabbleMethods list.

Here are the instructions for enabling ReconfigVM_Task method:

Step 1 - You will need to find the MoRef (Managed Object Reference) ID of the VM that you wish to enable the method on. You can do this by either browsing through the vSphere MOB, using this vSphere SDK for Perl script or this PowerCLI snippet:

Get-VM -Name [VM-NAME] | ft -Property Id

Step 2 - Open a web browser to the following URL:

https://[VC-IP]/mob/?moid=AuthorizationManager&method=enableMethods

Step 3 - You will need to replace the following two parameters (make sure to replace the VM MoRef ID with the one you found in Step 1):

parameter	value
entity	<entity type="ManagedEntity" xsi:type="ManagedObjectReference">vm-35</entity>
method	<method>ReconfigVM_Task</method>

Step 4 - Once you have updated fields as shown in the screenshot below, to execute the API call you just need to click on the "Invoke Method" link on the bottom right.

If everything was successful, you should see some output from the operation listing the methods that are still currently disabled. You can also confirm that everything is working by refreshing the vSphere Web Client or if you are using the vSphere C# Client, the "Edit Settings" option should now be available. Lastly, if I now apply a VSAN VM Storage Policy, I will no longer get the error and as you can see from the screenshot below, I now have successfully applied my "VSAN-Platinum-VM-Storage-Policy" for my NSX Controller VM. I would strongly recommend that you re-enable the original disable method by following the instructions below.

Here are the instructions for disabling ReconfigVM_Task method:

Step 1 - You will need to find the MoRef (Managed Object Reference) ID of the VM that you wish to enable the method on. You can do this by following Step 1 from the enable instructions

Step 2 - Open a web browser to the following URL:

https://[VC-IP]/mob/?moid=AuthorizationManager&method=disableMethods

Step 3 - You will need to replace the following three parameters (make sure to replace the VM MoRef ID with the one you found in Step 1 and any random number works for sourceId property):

parameter	value
entity	<entity type="ManagedEntity" xsi:type="ManagedObjectReference">vm-35</entity>
method	<DisabledMethodRequest><method>ReconfigVM_Task</method><DisabledMethodRequest>
sourceId	1234

Step 4 - Once you have updated fields as shown in the screenshot below, to execute the API call you just need to click on the "Invoke Method" link on the bottom right.

If everything was successful, you should see a void output and if you go to your vSphere Web/C# Client, you should see that the "Edit Settings" option is now disabled again for this VM. Though applying a VSAN VM Storage Policy is pretty trivial, there are some additional things to be aware of when working with special "Service VMs" like the NSX Controller. Hopefully we can improve this workflow in the future and provide for a better user experience but in the mean time, you can use this workaround.

Does the ESXi Mac Learn dvFilter work with Nested ESXi on NSX VXLAN's?

09.19.2014 by William Lam // 3 Comments

After publishing my article on the new ESXi Mac Learn dvFilter which helps improve CPU/Network performance when using promiscuous mode with Nested ESXi, I received a couple of questions asking whether the dvFilter would work with NSX VXLAN's? At the time, I had only tested the Mac Learn dvFilter using standard VSS/VDS and not with any VXLAN based networks. I had reached out to a couple of folks asking whether this would work and to my surprise, I actually got back a mix set of answers to it will not work to it could work. One of the reasons that was given to me on why this may not work is that NSX-v (NSX for vSphere) leverages a different "virtual switch" than VSS/VDS and hence the Mac Learn dvFilter would not properly function. This actually would make sense, but because I received other responses negating that fact, I figured I probably should just test it for myself and see.

NSX 6.1 was recently released and I figured this would be a great opportunity for me to learn a bit more about NSX, as I have never played with it before and also test whether Mac Learn dvFilter would in fact work with NSX VXLAN's. In my lab environment I have deployed NSX and I have 3 physical ESXi hosts running VSAN (go SDS!). I deployed both an NSX ESR (Edge Service Router) hosting 2 Logical Networks (aka VXLAN segments) and an NSX DLR (Distributed Logical Router) hosting another 2 Logical Networks.

Here is a screenshot of the 4 Logical Networks, the first two on NSX ESR and the last two on NSX DLR:

Here is a screenshot of both the NSX ESR and DLR:

Note: If you would like to learn more about NSX ESR and DLR, check out this great article by Brad Hedlund who goes into more detail.

For my test, I first enabled Promiscuous Mode and Forged Transmit on the respective Logical Switches which is just a dvPortgroup on the VDS for my NSX ESR setup. I then had 2 Nested ESXi VMs running (without the Mac Learn dvFilter), a Windows "Jump Box" VM and vMA all connected to the same VLXAN network.

I then transfer an ISO from the Windows VM to vMA while running ESXTOP on the physical ESXi host which is hosting these four VMs. As I expected, both the Nested ESXi VMs and vMA were receiving network packets. Next, I installed the Mac Learn dvFilter VIB on the physical ESXi host and added the required VM Advanced Settings to both the Nested ESXi VMs and then re-ran the test. To my surprise, both the Nested ESXi VMs were no longer receiving the erroneous packets! So it seems that using VLXAN with NSX ESR, the Mac Learn dvFilter is working as expected.

To be thorough, I also ran through same test but now for the VXLAN segments backed by NSX DLR. This time, I was really surprised by the results. The test was prior to installing the Mac Learn dvFilter and my expectation was that the two Nested ESXi VMs would be seeing the duplicated network packets from the VDS, but to my surprise, they did not! Both the Nested ESXi VMs were pretty much idling at 0 packets as nothing was being sent to them. I am not exactly sure why I was seeing this behavior, perhaps there is some type of optimization in the DLR? This is something I hope to get an answer from someone in Engineering on why I might be seeing this positive behavior.

To summarize, this myth has been busted and the Mac Learn dvFilter does in fact work with VXLAN networks. If you are using NSX ESR for your VXLAN setup, then you will need to install the dvFilter and if you are using NSX DLR, it seems like you do not need to make any additional changes. After briefly speaking with Christian Dickmann, the creator of the dvFilter as I wanted to share the results with him, I also learned about some interesting tidbits. Christian was not surprised by the results actually, the reason for this is that the VMkernel networking stack was architected and designed to be modular. This meant that, one could switch out the "virtual switch" with other implementations and the underlying dvFilter framework would still continue to work regardless of the "virtual switch" being used.

Additional Note:

I did not get a chance to test with vCNS and VXLAN, but I believe it should work given NSX-v is functional. If you are able to test this, feel free to leave a comment on whether the expected behavior is seen with the Mac Learn dvFilter.
I did not get a chance to test this with vCloud Director with VXLAN based networks, but as I mentioned, this should work. Please leave a comment if you can confirm
I also noticed when creating the Logical Switches, there is a Mac Learning capability, but from my testing, I found it did not benefited Nested ESXi and the Mac Learn dvFilter was still required.