WilliamLam.com

esxcli Part1 - What is esxcli?

by // 9 Comments

esxcli is a new CLI (commandline interface) framework in vSphere that provides a modular architecture for various components called namespaces running in the VMkernel. Some of these namespaces are nmp (Native Multipathing Plugin) for the new VMware Pluggable Storage Architecture, corestorage for claim rules used for masking certain devices to a host, and swiscsi for managing iSCSI interface.

esxcli can be executed within the classic ESX Service Console, the unsupported Busybox console in ESXi or using the vCLI's remote version of esxcli. There are currently 3 namespaces (nmp,corestorage and swiscsi) with the current release of vSphere and we may see others introduced in future releases of vSphere. One important thing to note is that because these modules run within the host, using the vCLI's version of esxcli, you will need to authenticate to the host first to see what modules will be available for access. Currently, esxcli is not vCenter aware, meaning you must connect to a specific ESX or ESXi host when performing an operation.

Here is an example of esxcli executed without connecting to the host first:

Here is an example of esxcli being executed after connecting to the host:

When invoking the esxcli command, you may also notice an esxcli.log is generated. If the command is successfully executed, the log will generally be empty, but if there was an error you may want to take a look at esxcli.log if the command does not provide any output to the screen.

Here is an example of using an auth configuration file and because of the case sensitivity of esxcli, the VI_PROTOCOL entry is failing with HTTPS vs https:

[vi-admin@scofield ~]$ cat esxcli.log

[root CRITICAL] Exception:Unsupported protocol
[root CRITICAL] Traceback (most recent call last):
File "esxcli.py", line 387, in _GetStub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 239, in stub
File "/vmware/esx40-dev/esx40/bora/vim/py/esxcli/Session.py", line 299, in Login
Exception: Unsupported protocol

There is not a whole lot of information available to the public about esxcli. From what I understand after talking to a few VMware engineers, esxcli has an API, but it is currently not exposed to the public for consumption. Not only is there an API, but 3rd party providers or users can potentially create their own modules and install it using the VIB format also known as vSphere Installation Bundle.

Some well known packages that are currently being distributed in the VIB format today are: Cisco Nexus 1000V VEM, HP Insight Manager Agent, EMC PowerPathV/E, Xsigo ESX IB drivers and VMware ESX/ESXi/vMA updates, to name a few. Hopefully in the future, VMware will expose the esxcli API functionality to the developer ecosystem.

Here are a few blog posts with detail examples on using esxcli with the various namespaces:

Stay tuned for Part2 and Part3 where we will look at automating esxcli operations using both the vSphere SDK for Perl and Windows PowerShell!

How to remove stale targets from vMA

by // Leave a Comment

If you have used vMA's vi-fastpass authentication, you will know how easy it is to setup using vifp utility which supports both ESX/ESXi and vCenter targets.

Here's an example of adding ESXi target:

[vi-admin@scofield ~]$ sudo vifp addserver esxi3-1.primp-industries.com
*protected email*'s password:

Here's an example of the listing of the available fastpass targets:

[vi-admin@scofield ~]$ sudo vifp listservers
esxi3-1.primp-industries.com ESXi

During this process, two accounts (vi-userXX & vi-adminXX) are created on the target host with a password that vMA management creates and caches it locally in an obfuscated but not encrypted form. This will allow you to initialize a fastpass target using vifpinit utility and execute commands against the target host without having to manually type in the credentials.

The fastpass targets are stored in 2 configuration files on vMA:

1) The obfuscated cached credentials is stored in /home/vi-admin/.vmware/credstore/vicredentials.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   vi-admin00
   XXXXXXXXXXXXXXXXXXXXXXX

2) A More detailed configuration for each of the targets along is stored in /etc/vmware/viconfig/viconfig.xml

If you cat out the contents, it will look something like this:

   esxi3-1.primp-industries.com
   443
   524d18f6-8bbb-2c5f-a366-6d191813fbe3
   https
   /sdk
   vi-admin00
   vi-user00
   true
   ESX
   1276121961

What happens when you rebuild your host, or the system is no longer available because it has been decommissioned or being used for another purpose? vMA will still think it's managing the host and the fastpass credentials will no longer function as the account is no longer valid the host. If you try to remove the old target, you will see the following error:

[vi-admin@scofield ~]$ sudo vifp removeserver esxi3-1.primp-industries.com
*protected email*'s password:

Error: Failed to connect. Please make sure the server is up and is of supported version.

The reason this occurs is that vMA is unable to login to the host and remove the two accounts that were initially created and fails to remove the target. What you will need to do is actually pass in an additional parameter to vifp command "--force" which will forcefully remove the target from vMA management. This command actually does not require the user to enter the correct password to the host even if it is still reachable by vMA. By specifying this flag and providing some input when prompted for the password, vMA will purge the target from it is system.

[vi-admin@scofield ~]$ sudo vifp removeserver esxi3-1.primp-industries.com --force
*protected email*'s password:

After a target is removed from vMA, it is also removed from the two above files. You do not manually tweak either of these configuration files or it may lead to issues on your vMA host.

Best practice for decommissioning a host that has been added to vMA's management is the following:

  1. Disable vilogger if you've enabled it for the host
  2. Remove target from vMA management
  3. Verify the host is no longer being managed by vMA
  4. Decomission host

Increase Syslog count in ESXi using Busybox

by // 2 Comments

There was an interesting question on the ESXi forum today using syslog and logging to a local VMFS volume and potentially filling up the datastore. By default, the log files are 2MB each before they are rotated out with a total of 9 copies all together. This ensures that the logs will not grow beyond 18MB and potentially fill up your datastore.

Within the unsupported console of ESXi, there is a little tool called "Busybox" which implements a set of familiar command line utilities called applets. Some of these applets are uptime, crond, chroot, md5sum, etc... If you login to the unsupported console and just type "busybox" on the console, you will see the following:

If you look carefully, you will see that one of the applets is "syslogd", which is the syslog daemon running in ESXi. To access a specific applet, you will just type busybox and then the name of the applet along with -h which forces the applet to provide a set of available options. If we do this for syslogd, you will see the following:

The default configurations should ensure that your logs don't go crazy and fill up the volume, but the downside is that your log history will not be kept forever. That is why it is a best practice to setup a remote syslog server to send all your logs for further processing and auditing if necessary.

However, if you wanted to change this defaut, you can. As you can see from the options above, you can configure not only the size (max of 2MB) before rotating out, but also the number of logs to keep (max of 99). While you cannot increase the size of the individual logs, you can specify a larger number of logs to keep. If you want to make the changes live while the system is running, you'll need to perform the following:

1. Find the current pid of syslogd

~ # ps | grep syslog
4313 4313 busybox syslogd

2. Kill the syslog process

~ # kill $(cat /var/run/syslogd.pid)

3. Let's say you want to keep 50 copies instead of 9

~ # busybox syslogd -b 50

Note:

  • The input to -s is in bytes with range (0-2097151)
  • The input to -b has range (1-99)

It's important to note that this change will not persist through a reboot. I have not been able to figure out where this is set; it could just be hardcoded in the binary by default. A way around this is to re-define the configuration upon boot up by adding an entry to /etc/rc.local which will kill the current running syslogd and then start up with the new parameters as shown above.

Add the following lines to the end of /etc/rc.local:

kill $(cat /var/run/syslogd.pid)
busybox syslogd -b 50

Now for this change to fully persist, you need to do one more thing, you'll need to run /sbin/auto-backup.sh which will force a local backup of the ESXi configuration files which includes /etc/rc.local so that it'll survive through the next reboot. Now you'll be able to store additional ESXi logs for a longer period of time if you choose to log locally.