If you are using the latest version of vMA 5.5, there is currently a known issue when using resxtop. After successfully logging into the ESXi host, the command prompt is returned immediately and resxtop is not actually loaded.

Here is a screenshot of the behavior:

I have received word from engineering there is a workaround that is required to get resxtop working, which is to run the following command:

sudo mv /usr/lib/vmware-rcli/lib/ /usr/lib/vmware

Once you have executed the above command (only required once), you now will be able to connect to your ESXi host using resxtop.

Engineering is working on getting the vMA release notes updated as well as a KB article published and we will have fix for this in the a sub-sequent update/patch release of vMA.