WilliamLam.com

Poor man's traceroute for ESXi

by // 1 Comment

There was a question today on the VMTN community forum about traceroute and ESXi. The user was troubleshooting a networking issue and logging into Tech Support Mode (Busybox Console) of ESXi, had noticed this common networking utility was not available. Since the user was working in a production environment, a re-compilation of a custom Busybox applet was out of the question and most likely unsupported by VMware. The user had to explore other avenues of troubleshooting, or did they have to?

It may not be well known, but within the Busybox Console, a Python interpreter is installed and accessible with a limited set of modules. Armed with this fact, you can actually create your own "poor man's traceroute" which gave you functionality similar to that of the actual traceroute/tracert implementation. This "poor man's" version was dubbed by a blog article I had found online (sorry, no ghetto in this name) by Leonid Grinberg - Learning by doing: Writing your own traceroute in 8 easy steps. The author goes into extreme detail providing step by step instructions on his python implementation of traceroute. Leonid provides a fully functional script at the end of his blog post, but he does caveat the following statement at the end:

This is actually not quite how the real traceroute works. Rather than checking the IP addresses of the hosts and stopping when the destination address matches, it stops when it receives a ICMP “port unreachable” message, which means that the host has been reached. For our purposes, though, this simple address heuristic is good enough.

This was definitely another alternative if one needed functionality similar to that of traceroute. I used the author's polished version of traceroute.py and scp'ed the script onto an ESXi host. It ran just as well as the real traceroute on the classic ESX Service Console.

Though this may not have solved the original network issue, it is important to know what utilities you have at your disposal, even if it is not the actual tool that you are familiar with. There is always more than one way to do something and it is always good to know a few.

VMware does limit the number of "common" UNIX/Linux utilities that are bundled within the Busybox applet and based on demand and feature requests, we may see future utilities added to Tech Support Mode. One example is the release of vSphere 4.1, we saw the addition of nc (netcat) and tcpdump which are invaluable troubleshooting utilities for a vSphere admin. If there are other utilities that you feel should be included in the default Busybox applet, be sure to submit your feedback to either your VMware rep or file a feature request.

Cloning vMA still not supported?

by // 5 Comments

Cloning and/or moving vMA is not supported by VMware, this has been the case since VIMA 1.0. If you take look at the "Known Issues" in the vMA release notes, you will see there is no still workaround. Users may find it useful to be able to clone vMA for backup purposes or to deploy another vMA without having to re-download or re-uploading .OVF from your local desktop/VMware's website.

If you ever tried to clone VIMA 1.0 and boot the system up, you will notice a warning message during the init process when vMA verifies the system's UUID. If it detects that it has been cloned, it will actually disable itself and makes the vMA components unusable until you reinitialize vMA using vimaclean. This is still the case with vMA 4.1, except the vMA components are no longer disabled when you perform a clone.

You will just see the warning message during the init process, but all functionality continues to work. Before the vMA specific components starts up, it makes a call to /opt/vmware/vma/bin/verifyuuid. It verifies the system UUID which you can extract using dmidecode, this is the same UUID found within the .vmx entry called uuid.bios. When you clone a VM, a new UUID will be generated which causes an issue when it tries to verify.

Prior to vMA 4.x, VIMA 1.0 stored all it's configurations within a set of XML files located in /etc/vmware/viconfig, these files held the config for both the vi-fastpass targets and vilogger config and policies. With the release of vMA 4.0, these configurations were migrated to a sqlite3 database which is stored in /var/opt/vmware/vMA/vMA.db. Within this database, it also stored as the old flat file configuration, the system UUID of vMA. This is what verifyuuid was checking against to see if it was cloned or moved, you can actually query the vMA DB to extract this information using a simple SQL query.

To resolve the warning message, you just need to update the new UUID in the vMA DB using an update statement.

Instead of manually typing all that out, I actually created a very simple script that queries for the current system UUID and then updates the vMA DB with the correct UUID. Here is a sample execution:

Download: updatevMAUUID.sh

I suspect that VMware may have allowed cloning of vMA, but just did not update their release notes? The only thing you have to do is manually fix the UUID if you do not want to see the warning message. If you would like to fix this for VIMA 1.0, you will need to do the same thing except you need to modify the UUID found in /etc/vmware/viconfig/viconfig.xml and restart vMA for the changes to effect.

How to automate & cron vi-fastpass scripts on vMA 4.1

by // 3 Comments

vMA's vi-fastpass and vifptarget utility is meant to be used interactively on the command line, this requires the user to set a particular target (ESX,ESXi or vCenter host) and then execute either a vCLI and/or vSphere SDK for Perl script. If you want to stay in the context of a particular server (e.g without having to manually specify --server parameter), you would need to manually change the target. There have been two common set of questions that have re-appeared several times on the VMTN forums:

1) How to automate a particular script and leverage vi-fastpass without having to change the context for each ESX or ESXi host

2) How to automate a particular script and leverage vi-fastpass via cron without having to provide username/password 

    Before we dive into the solutions, the key in using vi-fastpass in a non-interactive fashion is how vifptarget is called from a script. When you call the vifptarget, it is actually an alias to source the following shell script located in /opt/vmware/vma/bin/vifptarget. You can see this by using the "which" utility on a particular command.

    This alias is defined in /etc/bashrc

    You can even redefine the alias name to whatever you like, so long as it's aliasing the actual vifptarget script. Properly sourcing vifptarget script is the key in solving Question 1. If you look at /etc/bashrc, you will also notice that LD_LIBRARY_PATH environmental variable for shared libraries is also defined, this will be the key in solving Question 2 along with properly sourcing vifptarget. You will see these referenced in the example scripts. 

    There are two solutions for Question 1:

    Solution A:

    If you are leveraging vCenter Server and the ESX(i) hosts are being managed by vCenter, then you just need to add vCenter to vi-fastpass and run a simple for loop across the hosts.

    Example:
    2 x ESXi 4.1 hosts being managed by vCenter

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vCenter and reads in a file containing the name of the ESX(i) hosts:

    example1.sh

    Download: example1.sh

    Solution B:

    In this solution, vCenter is not available or you each ESX(i) host being managed by vMA's vi-fastpass. The script will be very similar, but instead of setting vCenter as the target, you will be setting an ESX(i) host. What is interesting that may not be well known is if you are managing more than one ESX(i) vi-fastpass target, you only need to set the target once and you can perform the same operation on all other host. The caveat being, if you want to switch out of the default context of your target, you need to specify --server parameter. This script will actually do it for you behind to scenes.

    Example:
    2 x ESXi 4.1 hosts being managed by vMA

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vMA and reads in a file containing the name of the ESX(i) hosts:

     example2.sh

    Download: example2.sh

    After executing either SolutionA or SolutionB, you now should have the configurations applied to your ESX(i) host as you would if you manually ran it on each host. This is just an example, you can easily substitute multiple esxcfg-* and/or vSphere SDK for Perl scripts. Here is also another example script for a user on the VMTN forums on configuring a vSwitch and adding portgroups.

    Solution for Question 2:

    Now that we know how to automate a particular operation using vi-fastpass in a script, the next logical question is how do we automate it using a cronjob. I will not go into details of how cron works, take a look at the link for more details. One thing I will mention is that cron does not have all the same PATH environmental variables defined as you normally would expect. You should always use full path to scripts and binaries and any shared library modules that is required to execute a particular utility. What this means for leveraging vi-fastpass via a cronjob is that you need to specify the LD_LIBRARY_PATH. In solving Question 1, all we had to do was properly source the vifptarget and the reason we did not have to specify the shared library path is that it was already defined as a default for vi-admin user. This is not the case for cron and you will need to specify that in either the script you are executing or within the crontab.

    Example:

    2 x ESXi 4.1

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute vmware-cmd -l to extract virtual machines on both ESX(i) host and store the output that to a file called /tmp/vms every 5 minutes. Here is what the cron entry should look like for example3.sh

    example3.sh

    Download: example3.sh

    After 5 minutes, you should now see a new file under /tmp/vms that contains all virtual machines registered on your ESX(i) host. Again, you can easily apply this to any other vCLI and/or vSphere SDK for Perl script.

    Here another example that gets asked quite often in scheduling the popular VMware vSphere Health Check Script, here is the crontab entry for automatically running the healthcheck script every hour against your vCenter server.

    example4.sh

    Download: example4.sh

    Again, the key is to make sure you specify the full path to not only the script you would like to run but also if there is any type of output, that you specify the full path to the output.