WilliamLam.com

esxcli Part1 - What is esxcli?

by // 9 Comments

esxcli is a new CLI (commandline interface) framework in vSphere that provides a modular architecture for various components called namespaces running in the VMkernel. Some of these namespaces are nmp (Native Multipathing Plugin) for the new VMware Pluggable Storage Architecture, corestorage for claim rules used for masking certain devices to a host, and swiscsi for managing iSCSI interface.

esxcli can be executed within the classic ESX Service Console, the unsupported Busybox console in ESXi or using the vCLI's remote version of esxcli. There are currently 3 namespaces (nmp,corestorage and swiscsi) with the current release of vSphere and we may see others introduced in future releases of vSphere. One important thing to note is that because these modules run within the host, using the vCLI's version of esxcli, you will need to authenticate to the host first to see what modules will be available for access. Currently, esxcli is not vCenter aware, meaning you must connect to a specific ESX or ESXi host when performing an operation.

Here is an example of esxcli executed without connecting to the host first:

Here is an example of esxcli being executed after connecting to the host:

When invoking the esxcli command, you may also notice an esxcli.log is generated. If the command is successfully executed, the log will generally be empty, but if there was an error you may want to take a look at esxcli.log if the command does not provide any output to the screen.

Here is an example of using an auth configuration file and because of the case sensitivity of esxcli, the VI_PROTOCOL entry is failing with HTTPS vs https:

[vi-admin@scofield ~]$ cat esxcli.log

[root CRITICAL] Exception:Unsupported protocol
[root CRITICAL] Traceback (most recent call last):
File "esxcli.py", line 387, in _GetStub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 239, in stub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 299, in Login
Exception: Unsupported protocol

There is not a whole lot of information available to the public about esxcli. From what I understand after talking to a few VMware engineers, esxcli has an API, but it is currently not exposed to the public for consumption. Not only is there an API, but 3rd party providers or users can potentially create their own modules and install it using the VIB format also known as vSphere Installation Bundle.

Some well known packages that are currently being distributed in the VIB format today are: Cisco Nexus 1000V VEM, HP Insight Manager Agent, EMC PowerPathV/E, Xsigo ESX IB drivers and VMware ESX/ESXi/vMA updates, to name a few. Hopefully in the future, VMware will expose the esxcli API functionality to the developer ecosystem.

Here are a few blog posts with detail examples on using esxcli with the various namespaces:

Stay tuned for Part2 and Part3 where we will look at automating esxcli operations using both the vSphere SDK for Perl and Windows PowerShell!

How to remove stale targets from vMA

by // Leave a Comment

If you have used vMA's vi-fastpass authentication, you will know how easy it is to setup using vifp utility which supports both ESX/ESXi and vCenter targets.

Here's an example of adding ESXi target:

[vi-admin@scofield ~]$ sudo vifp addserver esxi3-1.primp-industries.com
*protected email*'s password:

Here's an example of the listing of the available fastpass targets:

[vi-admin@scofield ~]$ sudo vifp listservers
esxi3-1.primp-industries.com ESXi

During this process, two accounts (vi-userXX & vi-adminXX) are created on the target host with a password that vMA management creates and caches it locally in an obfuscated but not encrypted form. This will allow you to initialize a fastpass target using vifpinit utility and execute commands against the target host without having to manually type in the credentials.

The fastpass targets are stored in 2 configuration files on vMA:

1) The obfuscated cached credentials is stored in /home/vi-admin/.vmware/credstore/vicredentials.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   vi-admin00
   XXXXXXXXXXXXXXXXXXXXXXX

2) A More detailed configuration for each of the targets along is stored in /etc/vmware/viconfig/viconfig.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   443
   524d18f6-8bbb-2c5f-a366-6d191813fbe3
   https
   /sdk
   vi-admin00
   vi-user00
   true
   ESX
   1276121961

What happens when you rebuild your host, or the system is no longer available because it has been decommissioned or being used for another purpose? vMA will still think it's managing the host and the fastpass credentials will no longer function as the account is no longer valid the host. If you try to remove the old target, you will see the following error:

[vi-admin@scofield ~]$ sudo vifp removeserver esxi3-1.primp-industries.com
*protected email*'s password:

Error: Failed to connect. Please make sure the server is up and is of supported version.

The reason this occurs is that vMA is unable to login to the host and remove the two accounts that were initially created and fails to remove the target. What you will need to do is actually pass in an additional parameter to vifp command "--force" which will forcefully remove the target from vMA management. This command actually does not require the user to enter the correct password to the host even if it is still reachable by vMA. By specifying this flag and providing some input when prompted for the password, vMA will purge the target from it is system.

[vi-admin@scofield ~]$ sudo vifp removeserver esxi3-1.primp-industries.com --force
*protected email*'s password:

After a target is removed from vMA, it is also removed from the two above files. You do not manually tweak either of these configuration files or it may lead to issues on your vMA host.

Best practice for decommissioning a host that has been added to vMA's management is the following:

  1. Disable vilogger if you've enabled it for the host
  2. Remove target from vMA management
  3. Verify the host is no longer being managed by vMA
  4. Decomission host

Script - hostops-lamw.pl

by // 1 Comment

I recently noticed a question on the ESXi forum about trying to add a host to a vCenter server that had the "SSL host certificate verification" enabled while using the vSphere SDK for Perl Utility hostops.pl on vMA. The user encountered the following error when trying to add the host:

Error:
SOAP Fault:
-----------
Fault string: Authenticity of the host's SSL certificate is not verified.
Fault detail: SSLVerifyFault

The SSL host verification is a feature that came with the release of vSphere that provides a security measure to verify the validity of a host before adding it to your VMware infrastructure. This feature is disabled by default, but when it is enabled, a user will need to accept a dialog box to confirm the SHA1 thumbprint of the host in question.

This particular use case was not handled properly by hostops.pl which caused the error message to be thrown. With a small tweak to VMware's canned script, the new and improved hostops-lamw.pl now supports adding an ESX or ESXi host into vCenter with SSL host verification enabled. You'll still be expected to verify the SHA1 thumbprint, but now you can pass this as an additional parameter which will tell vCenter that you have verified the host and add to vCenter management.

Scott Lowe originally wrote an article on how to verify the SHA1 thumbprint for both an ESX and ESXi host.

On ESX you can run the following:

openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout
-fingerprint

On ESXi, the only real way to verify is by looking at the DCUI's "View Support Information":

However, if you truly trust the ESX or ESXi host that you're going to add to vCenter, there is an alternative way of retrieving the SHA1 thumbprint using the vCLI's vifs and the modified hostops-lamw.pl.

By default, you'll be able to point your web browser to https://[hostname]/host/ssl_cert to see actual SSL certificate on your host, assuming this functionality is not disabled. What you can do is download the ssl_cert to vMA or system with vCLI installed and query for the SHA1 hash and provide that as input to hostops-lamw.pl.

Download: hostops-lamw.pl

Step 1. Download hostops-lamw.pl to either vMA or system running vCLI copy it to the following path:

vMA or Linux host /usr/lib/vmware-cli/apps/host
Windows C:\Program Files\VMware\VMware vSphere CLI\Perl\apps\host

Step 2. Download the ssl_cert to vMA:

[vi-admin@scofield ~]$ vifs --server esxi4-1.primp-industries.com --username root --get "/host/ssl_cert" esxi4-1.primp-industries.com-ssl_cert
Enter password:

Downloaded file to esxi4-1.primp-industries.com-ssl_cert successfully.

Step 3. Get the SHA1 thumbprint from the ssl_cert you downloaded:

[vi-admin@scofield ~]$ openssl x509 -sha1 -in esxi4-1.primp-industries.com-ssl_cert -noout -fingerprint
SHA1 Fingerprint=79:BB:39:09:F6:E5:91:BD:B0:C3:F3:09:B4:38:50:FB:ED:9C:53:A5

Step 4. Use the modified hostops-lamw.pl and the new --sslthumbprint providing the SHA1 thumbprint (remember to double quote it) along with the other required input to add the host to vCenter:

[vi-admin@scofield ~]$ ./hostops-lamw.pl --server reflex.primp-industries.com --username primp --operation addhost --target_host esxi4-1.primp-industries.com --target_username root --target_password 'password' --sslthumbprint "79:BB:39:09:F6:E5:91:BD:B0:C3:F3:09:B4:38:50:FB:ED:9C:53:A5" --cluster virtual-cluster
Host 'esxi4-1.primp-industries.com' added successfully