How to debug NSX-T API Automation with PowerCLI?

I recently needed to deploy the latest version of NSX-T (2.5) for some work I was doing with Project Pacific and of course it was related to Automation 🙂 It has been some time since I have touched the NSX-T Manager API (2.0) and although most of my existing code still worked, there were some things that broke due to API deprecation and also net new functionality that I needed to use.

I normally use PowerCLI for my Automation work and/or for prototyping purposes, not only is it easy to do but PowerCLI is still one of the most popular tool used by our customers and it means that they can easily benefit from my work. However, one of my pet peeves when working with the NSX-T APIs and PowerCLI is simply the lack of useful error messages. Here is the generic error message that you would normally see even checking the $Error[0].Exception.ServerError variable, it generally does not contain anything useful or actionable.

A server error occurred: 'com.vmware.vapi.std.errors.invalid_request': . Check $Error[0].Exception.ServerError for more details.

Here is a concrete example where I am attempting to create a new Transport Zone but I am purposing leaving out a required parameter and as you can see from the output, the same generic error message is shown and not very actionable.

I normally debug NSX-T API issue whether it is a syntax or usage problem by SSH'ing to the NSX-T Manager and monitoring the actual API logs to figure out what is actually going. It usually has exactly what I am looking for in terms of the actual server error message along with details on how to fix the problem.

The log file that you should take a look at is /var/log/proton/nsxapi.log and I normally would tail the log and grep for either fail or error which will usually catch the actual error that was thrown by the server. You also simply tail the log file to see all responses but I find that it can be a little verbose.

tail -f /var/log/proton/nsxapi.log | grep -i fail

Here is what the log entry looks like for our example above and we can clearly see what is missing from our API invocation:

2019-10-22T12:22:28.434Z INFO http-nio-127.0.0.1-7440-exec-11 NsxBaseRestController - - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" subcomp="manager"] UserName:'admin' ModuleName:'common-services' Operation:'POST@/api/v1/transport-zones'Operation status: 'failure' Error: Field level validation errors: {required property transport_type is missing}

Ideally, PowerCLI would return this exact same message as seen by NSX-T Manager and this normally would help me figure out the actual issue without having to SSH to NSX-T Manager.

In digging around for a bit, I ended up learning that this data was actually accessible as part of the error object that is returned from PowerCLI but for whatever reason, we do not actually default this output as part of the payload nor provide any guidance that this information is actually accessible!

If you run the following command after getting an error:

$Error[0].Exception.ServerError.data

You will actually see the same error message from NSX-T Manager! I was pretty happy to see that this information was actually available, just in a non-intuitive way if you ask me. I was about to file a feature enhancement but it looks like this was something that had already been identified and suggested awhile back, so I will be following up with the PowerCLI PM 🙂

Although you can retrieve the NSX-T Manager error message when using PowerCLI, I still think there is value in using the log file for other things like understanding the request payload that has been submitted from PowerCLI or any other SDK or API Client for that matter. For example, if you are unsure if the payload was constructed correctly, you can submit the request and then see how its processed on the server. You can also see how requests from the UI translate to specific API calls and their respective payloads.

If we take a look at our example again in creating a Transport Zone, the following log entry is captured which shows the entire payload and its parameters:

2019-10-22T12:22:28.434Z INFO http-nio-127.0.0.1-7440-exec-11 AuditingServiceImpl - FABRIC [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="1e7e340e-876b-4f92-b3ce-5f489e68bf70" subcomp="manager" username="admin"] UserName="admin", ModuleName="TransportZone", Operation="CreateTransportZone", Operation status="failure", New value=[{"host_switch_name":"nsswitch","host_switch_mode":"STANDARD","nested_nsx":false,"is_default":false,"display_name":"TZ-OVERLAY","_protection":"UNKNOWN"}]

I personally find this to be very valuable and I would love to see any PowerCLI cmdlet that talks to our REST API to have a special flag that would allow a user to easily see the payload request which take s the PowerCLI Object and output the actual JSON payload, so you can easily compare that against the respective REST API. I actually do this for some of the PowerCLI modules I have created such as the NSX-T Policy module where you can specify -Troubleshoot parameter which then outputs the raw payload request for inspection and debugging. I will also suggest this to our PowerCLI PM and hopefully get this into the backlog 🙂