WilliamLam.com

How to backup/restore vMA's config + vi-fastpass DB

by // 1 Comment

I recently wrote an article on the process of cloning vMA which provides a way of backing up vMA. Due to the ease of deploying vMA as a virtual appliance (using OVF), there aren't too many reasons you would need to backup this virtual machine. If you lost the system for whatever reason, you can easily re-deploy with just a few clicks.

However, having said that, if you extensively make use of vi-fastpass fpauth and manage a lot of targets whether they are ESX(i) or vCenter hosts, you need to understand it is not simply just re-deploying another vMA host. When you add a target to vMA's vi-fastpass, two accounts are provisioned on the host "vi-adminXX" and "vi-userXX", these accounts are associated with an encrypted cipher located on vMA which allows for "fastpass" access to the host without having to re-type the password to the host each time. If you were to re-deploy a new vMA host and add the targets again, your host will not only contain the old entries but now a new set of accounts for your new vMA host. This can be an issue as you start to have stale accounts on your ESX(i) or vCenter host.

To prevent this issue, you can actually backup both vMA's configurations which is primarily stored in a sqlite database and the vi-fastpass credential store. In the following example, I have two ESXi hosts being managed by my primary vMA and I also have a standby vMA DR host in which I will backup the files to.

First, you want to make sure you do not have any active vifptarget sessions, this is not a requirement but it can ensure you do not copy over any vMA "session cache" files to your DR site. You can check by doing a long listing in /home/vi-admin/.vmware and looking for the directory vmasessioncache which will contain any active cached sessions if you recently initialized a fastpass target.

Note: Again this is not really necessary and you can exclude the vmasessioncache directory as part of your backup

You will first need to "dump" the existing vMA database into a file and provide a name of your choice, you will need to run the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump > vMA.db.backup

Next, you will need to "scp" the following files to your vMA DR:

  •  vi-fastpass encrypted credential store file
    • /home/vi-admin/.vmware
  •  vMA's configuration dealing with vi-fastpass targets + vi-logger
    • /home/vi-admin/vMA.db.backup
  •  vMA's default logging configs + paths
    • /etc/vmware/vMA/vMA.conf 

You now should login to your vMA DR host and you should see only two files in the home directory of the vi-admin user: vMA.conf and vMA.db.backup (.vmware directory is a hidden directory in /home/vi-admin)

From here, you will restore vMA.conf and you will need to run the following command:

sudo mv vMA.conf /etc/vmware/vMA/vMA.conf

Next, you will restore vMA.db and you will need to run the following command:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db < vMA.db.backup

At this point, we can verify the database contents by just running the ".dump" command:

sqlite3 /var/opt/vmware/vMA/vMA.db .dump

Now, we're not done yet, we need to run one additional tool that will perform some VMware "black magic" which will allow this new vMA to access all your ESX(i) and/or vCenter targets just as you had it before on your primary vMA.

You will need to create a file that provides some dynamic shared libraries for the tool we are going to execute. Create a file under /etc/ld.so.conf.d/vmware-vma using "sudo" and paste the following two lines:

  • /opt/vmware/vma/lib
  • /opt/vmware/vma/lib64

Now you will need to run the following to read in the configuration:

sudo ldconfig -f /etc/ld.so.conf.d/vmware-vma

Now you are ready to run the "migratecredstore" utility which is located under /opt/vmware/vma/bin which will perform the "black magic" and make sure you use sudo.

Once you see the successful message on completing the migration of your credential store, you have now fully restored your original vMA configuration. Here we perform a list of the active servers that was once accessible on primary vMA and initializing a target and verifying that it does in fact work.

One thing to note, if you still have access to your primary vMA, both your vMA's are now in an active/active state, with caveat that your primary vMA is the only one allowed to make any changes. What I mean by this, is when you initially add a host to vMA's fastpass, it not only creates two accounts, but it also associates it's system's UUID as part of the unique identifier which is stored on the host with the key VMAID

This means that if you deleted the target off your vMA DR host, it does not actually remove this entry on your ESX(i) and/or vCenter host. Only the primary vMA which has the matching UUID is able to remove the entry all together when you perform a "vifp removeserver" operation.You can see the system UUID by using the dmidecode utility.

We can also see this within the vMA database, when viewing this on the vMA DR host, you will actually see both entries of the primary and the DR vMA UUIDs because we restored the database with the original vMA's config.

If you need your vMA DR host to be able to modify entries or rotate passwords, you will need to shutdown vMA and update it's bios.uuid within the .vmx entry. You must use the "original" vMA's UUID which you should see from the database by running the following command:

sqlite3 /var/opt/vmware/vMA/vMA.db "select * from management_info;"

You will also need to delete the "new" UUID to ensure that only one exists which should be the "original" UUID, you can so by running the following command substituting your UUID:

sudo sqlite3 /var/opt/vmware/vMA/vMA.db "delete from management_info where myUUID='422E4042-63EE-86D1-D22A-79B6ABCA8D68';"

At this point, your vMA DR is now your primary and your old vMA is no longer needed.

How to automate & cron vi-fastpass scripts on vMA 4.1

by // 3 Comments

vMA's vi-fastpass and vifptarget utility is meant to be used interactively on the command line, this requires the user to set a particular target (ESX,ESXi or vCenter host) and then execute either a vCLI and/or vSphere SDK for Perl script. If you want to stay in the context of a particular server (e.g without having to manually specify --server parameter), you would need to manually change the target. There have been two common set of questions that have re-appeared several times on the VMTN forums:

1) How to automate a particular script and leverage vi-fastpass without having to change the context for each ESX or ESXi host

2) How to automate a particular script and leverage vi-fastpass via cron without having to provide username/password 

    Before we dive into the solutions, the key in using vi-fastpass in a non-interactive fashion is how vifptarget is called from a script. When you call the vifptarget, it is actually an alias to source the following shell script located in /opt/vmware/vma/bin/vifptarget. You can see this by using the "which" utility on a particular command.

    This alias is defined in /etc/bashrc

    You can even redefine the alias name to whatever you like, so long as it's aliasing the actual vifptarget script. Properly sourcing vifptarget script is the key in solving Question 1. If you look at /etc/bashrc, you will also notice that LD_LIBRARY_PATH environmental variable for shared libraries is also defined, this will be the key in solving Question 2 along with properly sourcing vifptarget. You will see these referenced in the example scripts. 

    There are two solutions for Question 1:

    Solution A:

    If you are leveraging vCenter Server and the ESX(i) hosts are being managed by vCenter, then you just need to add vCenter to vi-fastpass and run a simple for loop across the hosts.

    Example:
    2 x ESXi 4.1 hosts being managed by vCenter

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vCenter and reads in a file containing the name of the ESX(i) hosts:

    example1.sh

    Download: example1.sh

    Solution B:

    In this solution, vCenter is not available or you each ESX(i) host being managed by vMA's vi-fastpass. The script will be very similar, but instead of setting vCenter as the target, you will be setting an ESX(i) host. What is interesting that may not be well known is if you are managing more than one ESX(i) vi-fastpass target, you only need to set the target once and you can perform the same operation on all other host. The caveat being, if you want to switch out of the default context of your target, you need to specify --server parameter. This script will actually do it for you behind to scenes.

    Example:
    2 x ESXi 4.1 hosts being managed by vMA

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vMA and reads in a file containing the name of the ESX(i) hosts:

     example2.sh

    Download: example2.sh

    After executing either SolutionA or SolutionB, you now should have the configurations applied to your ESX(i) host as you would if you manually ran it on each host. This is just an example, you can easily substitute multiple esxcfg-* and/or vSphere SDK for Perl scripts. Here is also another example script for a user on the VMTN forums on configuring a vSwitch and adding portgroups.

    Solution for Question 2:

    Now that we know how to automate a particular operation using vi-fastpass in a script, the next logical question is how do we automate it using a cronjob. I will not go into details of how cron works, take a look at the link for more details. One thing I will mention is that cron does not have all the same PATH environmental variables defined as you normally would expect. You should always use full path to scripts and binaries and any shared library modules that is required to execute a particular utility. What this means for leveraging vi-fastpass via a cronjob is that you need to specify the LD_LIBRARY_PATH. In solving Question 1, all we had to do was properly source the vifptarget and the reason we did not have to specify the shared library path is that it was already defined as a default for vi-admin user. This is not the case for cron and you will need to specify that in either the script you are executing or within the crontab.

    Example:

    2 x ESXi 4.1

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute vmware-cmd -l to extract virtual machines on both ESX(i) host and store the output that to a file called /tmp/vms every 5 minutes. Here is what the cron entry should look like for example3.sh

    example3.sh

    Download: example3.sh

    After 5 minutes, you should now see a new file under /tmp/vms that contains all virtual machines registered on your ESX(i) host. Again, you can easily apply this to any other vCLI and/or vSphere SDK for Perl script.

    Here another example that gets asked quite often in scheduling the popular VMware vSphere Health Check Script, here is the crontab entry for automatically running the healthcheck script every hour against your vCenter server.

    example4.sh

    Download: example4.sh

    Again, the key is to make sure you specify the full path to not only the script you would like to run but also if there is any type of output, that you specify the full path to the output.

    How to configure and use vMA's vi-fastpass with fpauth and adauth on vSphere 4.1

    by // 7 Comments

    From time to time, I see users posting on the VMTN forums with some questions and confusion around the proper implementation and functionality of vMA's vi-fastpass. The confusion is further enhanced with the new Active Directory functionality and integration with vMA's new vi-fastpass type called adauth.

    The vi-fastpass component found in vMA is a credentials caching mechanism to allow you to connect to your ESX(i) or vCenter servers. Prior to vMA 4.1, vMA 4.0 only supported one type of vi-fastpass which is just called fpauth (fastpass authentication). This fpauth basically allows you to manage an ESX(i) or vCenter server under vMA by creating a vi-adminXX and vi-userXX account. The password for these two accounts are obfuscated using a simple XOR cipher. A user can now initialize one of these managed targets and execute either vCLI or vSphere SDK for Perl scripts without having to specify credentials each and every time, this works because the vi-adminXX credentials are being used to connect to your target. This can make running a simple command across n-number of hosts simple without having to provide the credentials for every host.

    In vMA 4.1, a new vi-fastpass type has been introduced called adauth, in which you can leverage your Windows Active Directory credentials to authenticate against a managed target without having to store an insecure cipher in vMA. The concept works exactly the same as fpauth, but there are some additional pre-requisites that must be met before you take advantage of adauth. You need to ensure that your ESX(i) or vCenter server is joined to your Active Directory and that your vMA host is also joined to that same Active Directory domain. Once this is all configured, you will now be able to login to vMA using your AD credentials and execute a vCLI command or vSphere SDK for Perl script against an ESX(i) or vCenter host and your AD credentials will be used to authenticate against your target without using any stored cached credentials. This helps to centralize your user management and is much more secure than using the standard fpauth where the credentials are stored in a simple XOR cipher which can easily be decrypted.

    The following will describe instructions on setting up and verifying both standard fpauth and adauth using vMA 4.1 and vSphere 4.1 (ESXi and vCenter server). Before continuing, it is highly recommended that you take a look at both the vSphere 4.1 and vMA 4.1 documentation with regards to Active Directory integration.

    Configuring vMA vi-fastpass using fpauth (fastpass authentication):

    1. Login to vMA using vi-admin credentials, note that your user context is with the vi-admin account.

    2. Use the vifp addserver command to add an ESX or ESXi host, you will be using the root account to do so and provide the credentials to that account. If you do not receive any errors, you have successfully added the target.

    3. Using vifp addserver again, we now add a vCenter server and we will be using a local account found on the host. In this example, we will be using the administrator account and provide the credentials to that account.

    Note: You do not need to add both your ESX(i) and vCenter server to utilize vi-fastpass, this is just an example of adding both types of supported targets.

    4. Next, we use vifp listservers to display the managed targets and the type of vi-fastpass we are using denoted by fpauth or adauth.

    5. Earlier we mentioned that when using fpauth, the stored credentials are encrypted using a simple XOR cipher. You can view these entries by taking a look at /home/vi-admin/.vmware/credstore/vmacredentials.xml.

    6. To use a target, you need to first initialize the target by using the vifptarget -s command. This will change your command prompt in which it will display the current target that has been initialized.

    7. As you can see from the screenshot, we now have initialized the host esxi4-1.primp-industries.com and now we can run a vCLI command without having to specify the credentials to the host. In this example, we will just list the number of vmnics using esxcfg-nics -l command.

    8. We can also perform the same operation against the ESXi host by going through vCenter. To do so, you will need to initialize your target to your vCenter server and specify the --vihost parameter to the name of a managed ESX or ESXi host under vCenter.

    As you can see, you perform this vi-fastpass using fpauth against an ESX(i) or vCenter server. When you initialize a target, the command prompt tells you which host context you are currently in. If you decide you want to switch context, you can issue the vifptarget -c command which clears the target.

    Configuring vMA vi-fastpass using adauth (Active Directory authentication):

    1. Login to your ESX or ESXi host using the vSphere Client, you will need to first join your ESX(i) host to your AD domain. Click on Configuration tab and on the lower left hand side, click on Authentication Services and click on Properties. You will need to select Active Directory as the type and specify the Domain and click Join Domain. You will now be prompted to provide credentials that have the permissions to add a new host to the domain.

    2. If you joined the host to your AD domain successfully, you should see the following displaying the domain.

    3. When you join an ESX or ESXi host to AD, it actually does a group lookup for "ESX Admins", if it exists, it will assign the Administrator role to it. If it does not exist, it will periodically query your AD until one is created. In my lab environment, it was not created automatically and I had to manually create this group. For more details, take a look at this blog post by Maish - http://technodrone.blogspot.com/2010/07/esxi-41-active-directory-integration.html

    4. Next, you will need to either add the individual users or group that will utilize the vi-fastpass with adauth. You need to do this for each and every ESX or ESXi host under the Permissions tab. If you have more than one user, I would suggest creating and Active Directory group and adding all the users into that group so you add only the group. You will right click and Add Permission and specify your AD domain and select either the individual users or groups and make sure you select Administrator role. Once you have added the appropriate users, you should see the following.

     

    Note: At this point you should be able to login to your ESX(i) host using AD credentials using the vSphere Client and remember to use DOMAIN\username. If you are unable to login using your AD credentials, you will not be able to proceed with the next step as this is required for vi-fastpass adauth to function.

    5. You will login to your vMA host using the vi-admin account. You will now join your vMA host to your Active Directory domain using the domainjoin-cli join command. You will need to use sudo and specify the name of your domain and the username which as the permissions to join a host to your AD domain.

    6. To confirm you have successfully joined vMA to your AD domain, you can run the domainjoin-cli query command to list the domain. You can also verify the new computer name in your AD server.

    7. Now we add an ESXi host to vi-fastpass but instead of using fpauth, we will leverage adauth. You will need to use the vifp addserver command and specify --authpolicy adauth parameter. You will need to specify an AD account that will be utilizing the vi-fastpass, make sure you specify the DOMAIN_NAME\username. If you decide to use the --username parameter, make sure you escape the forward slash with another slash (e.g. --username PRIMP-IND\\primp). You would repeat this if you plan on adding a vCenter Server that is joined to the same domain.

    8. We can also verify that the vi-fastpass targets have been added using adauth by using the vifp listservers -l command which displays the managed targets and the type of vi-fastpass they have been added with.

    9. Since we are using adauth, we can confirm that no cached entries should be stored locally within the vMA host by looking at /home/vi-admin/.vmware/credstore/vmacredentials.xml which should be blank now.

    10. Now, we are ready to login to vMA using our AD credentials. Depending if you implemented this vMA AD Intergration Tip, you may not need to specify the full domain when logging in. If you did not, you would to need to specify the full DOMAIN\[email protected] when logging in.

    11. You will now use the vifptarget -s command to initialize an ESX(i) server just like in our fpauth use case and you now should be able to run esxcfg-nics -l without having to provide any additional credentials.

    12. We can also use the vifptarget -s command to initialize a vCenter server just like in our fpauth use case and specifying --vihost parameter to either your ESX(i) host and now should be able to run esxcfg-nics -l without having to provide any additional credentials.

    Hopefully by outlining the entire process for both vi-fastpass fpauth and adauth, implementing this functionality will be a breeze now.

    Additional Caveats: There have been some users who have not been able to get full vi-fastpass with adauth working with ESXi host. There have been a few threads on the VMTN forums including this one which refers to a potential VMware bug. I have not been able to reproduce the issue noted by the user, but I have seen several cases in which the above workflow is not the behavior that users have seen. If you run into any issues, I would recommend before trying again to unjoin both your ESX(i) host and vMA from your AD domain. You will also want to make sure you go into your AD server and delete these entries before trying again.