WilliamLam.com

How to backup/restore vMA's config + vi-fastpass DB

by // 1 Comment

I recently wrote an article on the process of cloning vMA which provides a way of backing up vMA. Due to the ease of deploying vMA as a virtual appliance (using OVF), there aren't too many reasons you would need to backup this virtual machine. If you lost the system for whatever reason, you can easily re-deploy with just a few clicks.

However, having said that, if you extensively make use of vi-fastpass fpauth and manage a lot of targets whether they are ESX(i) or vCenter hosts, you need to understand it is not simply just re-deploying another vMA host. When you add a target to vMA's vi-fastpass, two accounts are provisioned on the host "vi-adminXX" and "vi-userXX", these accounts are associated with an encrypted cipher located on vMA which allows for "fastpass" access to the host without having to re-type the password to the host each time. If you were to re-deploy a new vMA host and add the targets again, your host will not only contain the old entries but now a new set of accounts for your new vMA host. This can be an issue as you start to have stale accounts on your ESX(i) or vCenter host.

To prevent this issue, you can actually backup both vMA's configurations which is primarily stored in a sqlite database and the vi-fastpass credential store. In the following example, I have two ESXi hosts being managed by my primary vMA and I also have a standby vMA DR host in which I will backup the files to.

First, you want to make sure you do not have any active vifptarget sessions, this is not a requirement but it can ensure you do not copy over any vMA "session cache" files to your DR site. You can check by doing a long listing in /home/vi-admin/.vmware and looking for the directory vmasessioncache which will contain any active cached sessions if you recently initialized a fastpass target.

Note: Again this is not really necessary and you can exclude the vmasessioncache directory as part of your backup

You will first need to "dump" the existing vMA database into a file and provide a name of your choice, you will need to run the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump > vMA.db.backup

Next, you will need to "scp" the following files to your vMA DR:

  •  vi-fastpass encrypted credential store file
    • /home/vi-admin/.vmware
  •  vMA's configuration dealing with vi-fastpass targets + vi-logger
    • /home/vi-admin/vMA.db.backup
  •  vMA's default logging configs + paths
    • /etc/vmware/vMA/vMA.conf 

You now should login to your vMA DR host and you should see only two files in the home directory of the vi-admin user: vMA.conf and vMA.db.backup (.vmware directory is a hidden directory in /home/vi-admin)

From here, you will restore vMA.conf and you will need to run the following command:

sudo mv vMA.conf /etc/vmware/vMA/vMA.conf

Next, you will restore vMA.db and you will need to run the following command:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db < vMA.db.backup

At this point, we can verify the database contents by just running the ".dump" command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump

Now, we're not done yet, we need to run one additional tool that will perform some VMware "black magic" which will allow this new vMA to access all your ESX(i) and/or vCenter targets just as you had it before on your primary vMA.

You will need to create a file that provides some dynamic shared libraries for the tool we are going to execute. Create a file under /etc/ld.so.conf.d/vmware-vma using "sudo" and paste the following two lines:

  • /opt/vmware/vma/lib
  • /opt/vmware/vma/lib64

Now you will need to run the following to read in the configuration:

sudo ldconfig -f /etc/ld.so.conf.d/vmware-vma

Now you are ready to run the "migratecredstore" utility which is located under /opt/vmware/vma/bin which will perform the "black magic" and make sure you use sudo.

Once you see the successful message on completing the migration of your credential store, you have now fully restored your original vMA configuration. Here we perform a list of the active servers that was once accessible on primary vMA and initializing a target and verifying that it does in fact work.

One thing to note, if you still have access to your primary vMA, both your vMA's are now in an active/active state, with caveat that your primary vMA is the only one allowed to make any changes. What I mean by this, is when you initially add a host to vMA's fastpass, it not only creates two accounts, but it also associates it's system's UUID as part of the unique identifier which is stored on the host with the key VMAID

This means that if you deleted the target off your vMA DR host, it does not actually remove this entry on your ESX(i) and/or vCenter host. Only the primary vMA which has the matching UUID is able to remove the entry all together when you perform a "vifp removeserver" operation.You can see the system UUID by using the dmidecode utility.

We can also see this within the vMA database, when viewing this on the vMA DR host, you will actually see both entries of the primary and the DR vMA UUIDs because we restored the database with the original vMA's config.

If you need your vMA DR host to be able to modify entries or rotate passwords, you will need to shutdown vMA and update it's bios.uuid within the .vmx entry. You must use the "original" vMA's UUID which you should see from the database by running the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db "select * from management_info;"

You will also need to delete the "new" UUID to ensure that only one exists which should be the "original" UUID, you can so by running the following command substituting your UUID:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db "delete from management_info where myUUID='422E4042-63EE-86D1-D22A-79B6ABCA8D68';"

At this point, your vMA DR is now your primary and your old vMA is no longer needed.

How to automate & cron vi-fastpass scripts on vMA 4.1

by // 3 Comments

vMA's vi-fastpass and vifptarget utility is meant to be used interactively on the command line, this requires the user to set a particular target (ESX,ESXi or vCenter host) and then execute either a vCLI and/or vSphere SDK for Perl script. If you want to stay in the context of a particular server (e.g without having to manually specify --server parameter), you would need to manually change the target. There have been two common set of questions that have re-appeared several times on the VMTN forums:

1) How to automate a particular script and leverage vi-fastpass without having to change the context for each ESX or ESXi host

2) How to automate a particular script and leverage vi-fastpass via cron without having to provide username/password 

    Before we dive into the solutions, the key in using vi-fastpass in a non-interactive fashion is how vifptarget is called from a script. When you call the vifptarget, it is actually an alias to source the following shell script located in /opt/vmware/vma/bin/vifptarget. You can see this by using the "which" utility on a particular command.

    This alias is defined in /etc/bashrc

    You can even redefine the alias name to whatever you like, so long as it's aliasing the actual vifptarget script. Properly sourcing vifptarget script is the key in solving Question 1. If you look at /etc/bashrc, you will also notice that LD_LIBRARY_PATH environmental variable for shared libraries is also defined, this will be the key in solving Question 2 along with properly sourcing vifptarget. You will see these referenced in the example scripts. 

    There are two solutions for Question 1:

    Solution A:

    If you are leveraging vCenter Server and the ESX(i) hosts are being managed by vCenter, then you just need to add vCenter to vi-fastpass and run a simple for loop across the hosts.

    Example:
    2 x ESXi 4.1 hosts being managed by vCenter

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vCenter and reads in a file containing the name of the ESX(i) hosts:

    example1.sh

    Download: example1.sh

    Solution B:

    In this solution, vCenter is not available or you each ESX(i) host being managed by vMA's vi-fastpass. The script will be very similar, but instead of setting vCenter as the target, you will be setting an ESX(i) host. What is interesting that may not be well known is if you are managing more than one ESX(i) vi-fastpass target, you only need to set the target once and you can perform the same operation on all other host. The caveat being, if you want to switch out of the default context of your target, you need to specify --server parameter. This script will actually do it for you behind to scenes.

    Example:
    2 x ESXi 4.1 hosts being managed by vMA

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vMA and reads in a file containing the name of the ESX(i) hosts:

     example2.sh

    Download: example2.sh

    After executing either SolutionA or SolutionB, you now should have the configurations applied to your ESX(i) host as you would if you manually ran it on each host. This is just an example, you can easily substitute multiple esxcfg-* and/or vSphere SDK for Perl scripts. Here is also another example script for a user on the VMTN forums on configuring a vSwitch and adding portgroups.

    Solution for Question 2:

    Now that we know how to automate a particular operation using vi-fastpass in a script, the next logical question is how do we automate it using a cronjob. I will not go into details of how cron works, take a look at the link for more details. One thing I will mention is that cron does not have all the same PATH environmental variables defined as you normally would expect. You should always use full path to scripts and binaries and any shared library modules that is required to execute a particular utility. What this means for leveraging vi-fastpass via a cronjob is that you need to specify the LD_LIBRARY_PATH. In solving Question 1, all we had to do was properly source the vifptarget and the reason we did not have to specify the shared library path is that it was already defined as a default for vi-admin user. This is not the case for cron and you will need to specify that in either the script you are executing or within the crontab.

    Example:

    2 x ESXi 4.1

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute vmware-cmd -l to extract virtual machines on both ESX(i) host and store the output that to a file called /tmp/vms every 5 minutes. Here is what the cron entry should look like for example3.sh

    example3.sh

    Download: example3.sh

    After 5 minutes, you should now see a new file under /tmp/vms that contains all virtual machines registered on your ESX(i) host. Again, you can easily apply this to any other vCLI and/or vSphere SDK for Perl script.

    Here another example that gets asked quite often in scheduling the popular VMware vSphere Health Check Script, here is the crontab entry for automatically running the healthcheck script every hour against your vCenter server.

    example4.sh

    Download: example4.sh

    Again, the key is to make sure you specify the full path to not only the script you would like to run but also if there is any type of output, that you specify the full path to the output.

    How to configure and use vMA's vi-fastpass with fpauth and adauth on vSphere 4.1

    by // 7 Comments

    From time to time, I see users posting on the VMTN forums with some questions and confusion around the proper implementation and functionality of vMA's vi-fastpass. The confusion is further enhanced with the new Active Directory functionality and integration with vMA's new vi-fastpass type called adauth.

    The vi-fastpass component found in vMA is a credentials caching mechanism to allow you to connect to your ESX(i) or vCenter servers. Prior to vMA 4.1, vMA 4.0 only supported one type of vi-fastpass which is just called fpauth (fastpass authentication). This fpauth basically allows you to manage an ESX(i) or vCenter server under vMA by creating a vi-adminXX and vi-userXX account. The password for these two accounts are obfuscated using a simple XOR cipher. A user can now initialize one of these managed targets and execute either vCLI or vSphere SDK for Perl scripts without having to specify credentials each and every time, this works because the vi-adminXX credentials are being used to connect to your target. This can make running a simple command across n-number of hosts simple without having to provide the credentials for every host.

    [Read more...]