William Lam

How to backup/restore vMA's config + vi-fastpass DB

02.04.2011 by William Lam // 1 Comment

I recently wrote an article on the process of cloning vMA which provides a way of backing up vMA. Due to the ease of deploying vMA as a virtual appliance (using OVF), there aren't too many reasons you would need to backup this virtual machine. If you lost the system for whatever reason, you can easily re-deploy with just a few clicks.

However, having said that, if you extensively make use of vi-fastpass fpauth and manage a lot of targets whether they are ESX(i) or vCenter hosts, you need to understand it is not simply just re-deploying another vMA host. When you add a target to vMA's vi-fastpass, two accounts are provisioned on the host "vi-adminXX" and "vi-userXX", these accounts are associated with an encrypted cipher located on vMA which allows for "fastpass" access to the host without having to re-type the password to the host each time. If you were to re-deploy a new vMA host and add the targets again, your host will not only contain the old entries but now a new set of accounts for your new vMA host. This can be an issue as you start to have stale accounts on your ESX(i) or vCenter host.

To prevent this issue, you can actually backup both vMA's configurations which is primarily stored in a sqlite database and the vi-fastpass credential store. In the following example, I have two ESXi hosts being managed by my primary vMA and I also have a standby vMA DR host in which I will backup the files to.

First, you want to make sure you do not have any active vifptarget sessions, this is not a requirement but it can ensure you do not copy over any vMA "session cache" files to your DR site. You can check by doing a long listing in /home/vi-admin/.vmware and looking for the directory vmasessioncache which will contain any active cached sessions if you recently initialized a fastpass target.

Note: Again this is not really necessary and you can exclude the vmasessioncache directory as part of your backup

You will first need to "dump" the existing vMA database into a file and provide a name of your choice, you will need to run the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump > vMA.db.backup

Next, you will need to "scp" the following files to your vMA DR:

vi-fastpass encrypted credential store file
- /home/vi-admin/.vmware
vMA's configuration dealing with vi-fastpass targets + vi-logger
- /home/vi-admin/vMA.db.backup
vMA's default logging configs + paths
- /etc/vmware/vMA/vMA.conf

You now should login to your vMA DR host and you should see only two files in the home directory of the vi-admin user: vMA.conf and vMA.db.backup (.vmware directory is a hidden directory in /home/vi-admin)

From here, you will restore vMA.conf and you will need to run the following command:

sudo mv vMA.conf /etc/vmware/vMA/vMA.conf

Next, you will restore vMA.db and you will need to run the following command:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db < vMA.db.backup

At this point, we can verify the database contents by just running the ".dump" command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump

Now, we're not done yet, we need to run one additional tool that will perform some VMware "black magic" which will allow this new vMA to access all your ESX(i) and/or vCenter targets just as you had it before on your primary vMA.

You will need to create a file that provides some dynamic shared libraries for the tool we are going to execute. Create a file under /etc/ld.so.conf.d/vmware-vma using "sudo" and paste the following two lines:

/opt/vmware/vma/lib
/opt/vmware/vma/lib64

Now you will need to run the following to read in the configuration:

sudo ldconfig -f /etc/ld.so.conf.d/vmware-vma

Now you are ready to run the "migratecredstore" utility which is located under /opt/vmware/vma/bin which will perform the "black magic" and make sure you use sudo.

Once you see the successful message on completing the migration of your credential store, you have now fully restored your original vMA configuration. Here we perform a list of the active servers that was once accessible on primary vMA and initializing a target and verifying that it does in fact work.

One thing to note, if you still have access to your primary vMA, both your vMA's are now in an active/active state, with caveat that your primary vMA is the only one allowed to make any changes. What I mean by this, is when you initially add a host to vMA's fastpass, it not only creates two accounts, but it also associates it's system's UUID as part of the unique identifier which is stored on the host with the key VMAID

This means that if you deleted the target off your vMA DR host, it does not actually remove this entry on your ESX(i) and/or vCenter host. Only the primary vMA which has the matching UUID is able to remove the entry all together when you perform a "vifp removeserver" operation.You can see the system UUID by using the dmidecode utility.

We can also see this within the vMA database, when viewing this on the vMA DR host, you will actually see both entries of the primary and the DR vMA UUIDs because we restored the database with the original vMA's config.

If you need your vMA DR host to be able to modify entries or rotate passwords, you will need to shutdown vMA and update it's bios.uuid within the .vmx entry. You must use the "original" vMA's UUID which you should see from the database by running the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db "select * from management_info;"

You will also need to delete the "new" UUID to ensure that only one exists which should be the "original" UUID, you can so by running the following command substituting your UUID:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db "delete from management_info where myUUID='422E4042-63EE-86D1-D22A-79B6ABCA8D68';"

At this point, your vMA DR is now your primary and your old vMA is no longer needed.

Poor man's traceroute for ESXi

02.02.2011 by William Lam // 1 Comment

There was a question today on the VMTN community forum about traceroute and ESXi. The user was troubleshooting a networking issue and logging into Tech Support Mode (Busybox Console) of ESXi, had noticed this common networking utility was not available. Since the user was working in a production environment, a re-compilation of a custom Busybox applet was out of the question and most likely unsupported by VMware. The user had to explore other avenues of troubleshooting, or did they have to?

It may not be well known, but within the Busybox Console, a Python interpreter is installed and accessible with a limited set of modules. Armed with this fact, you can actually create your own "poor man's traceroute" which gave you functionality similar to that of the actual traceroute/tracert implementation. This "poor man's" version was dubbed by a blog article I had found online (sorry, no ghetto in this name) by Leonid Grinberg - Learning by doing: Writing your own traceroute in 8 easy steps. The author goes into extreme detail providing step by step instructions on his python implementation of traceroute. Leonid provides a fully functional script at the end of his blog post, but he does caveat the following statement at the end:

This is actually not quite how the real traceroute works. Rather than checking the IP addresses of the hosts and stopping when the destination address matches, it stops when it receives a ICMP “port unreachable” message, which means that the host has been reached. For our purposes, though, this simple address heuristic is good enough.

This was definitely another alternative if one needed functionality similar to that of traceroute. I used the author's polished version of traceroute.py and scp'ed the script onto an ESXi host. It ran just as well as the real traceroute on the classic ESX Service Console.

Though this may not have solved the original network issue, it is important to know what utilities you have at your disposal, even if it is not the actual tool that you are familiar with. There is always more than one way to do something and it is always good to know a few.

VMware does limit the number of "common" UNIX/Linux utilities that are bundled within the Busybox applet and based on demand and feature requests, we may see future utilities added to Tech Support Mode. One example is the release of vSphere 4.1, we saw the addition of nc (netcat) and tcpdump which are invaluable troubleshooting utilities for a vSphere admin. If there are other utilities that you feel should be included in the default Busybox applet, be sure to submit your feedback to either your VMware rep or file a feature request.