WilliamLam.com

How to Automate the Deployment & Configuration of vShield Manager 5

by // 8 Comments

If you have ever worked with VMware vShield Manager, you know that deployment and configuration of the virtual appliance is pretty much a manual process. You can automate the deployment of the vShield Manager OVA using the various vSphere SDK's or the ovftool, but the initial IP address configuration for vSM still needs to be configured manually using the remote console for the very first time.

An easy solution to this problem would be for VMware to create the vSM OVA to support IP address configuration out of the box as part of the deployment options (but why make things easy). In any case, I will demonstrate how you can easily automate both the deployment and the initial configuration of vShield Manager to your vSphere environment.

Before I begin, I can not take credit for coming up with the idea of automating vShield Manager deployment, the credit goes to Alan Renouf. Alan recently contacted me and ask if it was possible to automate the IP configuration. The answer is yes and here is a solution.

One of the main challenges in figuring out how to automate the IP address configuration of vShield Manager was due to the vtysh integrated shell daemon for Zebra that launches by default as part of the "admin" user account. This interface is used to manage the kernel routing and management table and made it very difficult to interface with for any type of automation. I decided manually go through a vSM configuration and then using Knoppix LIVE-CD, I was able to mount up the vSM filesystem and look around to get a better understanding of what was going on. After some investigating, it looks like IP address configuration is stored in /common/configs/cli/zebra.conf, here is an example of what the configuration looks like:

Armed with this knowledge, it was pretty straight forward in developing an automated way of deploying and configuring vShield Manager. I created a script called deployvShieldManager.sh which utilizes guestOpsManagement.pl, vCLI and ovftool. It's recommended that you use vMA and install ovftool to quickly get started. At a high level, the script is doing the following:

  1. Deploy vShield Manager OVA using ovftool
  2. PowerOn vSM and wait 2 minutes for VMware Tools to be ready on the system
  3. Create new zebra.conf, backup the old zebra.conf and upload new zebra.conf using the new vSphere 5 VIX integration
  4. PowerOff vSM to force the configurations to be read in upon next bootup
  5. PowerOn vSM and it is now ready for use

At the top of the script, there are several configuration variables that need to be edited by the user to specify the vSM configuration, including vCenter and ESX(i) host to deploy vSM.

Here is a list of variables that need to be configured at the top of the deployvShieldManager.sh script:

Once you are done updating the variables, you are now ready to execute the script. Before the script performs any changes, it will first prompt the summary of configurations you have specified in the script. Once you are satisfiy, you may than proceed by typing "y" or "yes" to start, or if you would like to cancel, type "n" or "no".

Note: The script will perform some basic validation such as existence of the vShield Manager OVA, ovftool, etc. else you will get an error message and the script will exit.

Next, the script will perform the deployment of vSM using ovftool and proceed with the configuration of vSM once it has been deployed.

Note: If it takes longer to poweron vSM in your environment to get it into a ready state, you may want to tweak the sleep period from 120 seconds (2minutes) to something longer.

At this point, you now should see a new vShield Manager VM deployed and if you take a look at the summary page, you should see the new IP address and hostname configurations.

Now all that is left is to point your browser to the vSM address and you should be prompted to login to vShield Manager management interface.

Instead of manually deploying vShield Manager in your environment, you can now automate the initial deployment and configuration for general use or with VMware vCloud Director. For further automation and configuration of vShield manager, once vSM is online and accessible, you can leverage the vShield REST API.

How to Create and Modify vgz (vmtar) Files on ESXi 3.x/4.x

by // 6 Comments

There were several questions today on the VMTN community forums with regards to manipulating .vgz files in ESXi, also known as vmtar files. Due to the sparse amount of information on the web, I wanted to document some of the common operations that can be performed on the vmtar files. I will not be going over the use cases for manipulating or creating custom vmtar files, but here is one use case.

UPDATE (10/16/18) - For ESXi 6.5+, please use the following commands and the example below is using the s.v00 file:

Decompress file:

gunzip < s.v00 > s.v00.xz
xz --single-stream --decompress < s.v00.xz > s.v00.vtar
vmtar -v -x s.v00.vtar -o s.v00.tar
tar -xvf s.v00.tar

Compress file:

tar -cvf s.v00-new.tar bin/ etc/ lib/ lib64/ opt/ usr/ var/
vmtar -v -c s.v00.tar -o s.v00.vtar
xz --single-stream --compress < s.v00.vtar > s.v00.xz
xz --single-stream --compress < s.v00.vtar > s.v00

You can find some of these vmtar files with .vgz extension in the ESXi installation iso, here are a few highlighted in red:

To operate on existing vmtar files, you will need access to an ESXi host via ESXi Shell and using the /sbin/vmtar utility.

Usage: vmtar {[-x vtar/vgz-file] [-c tar/tgz-file] [-v] -o destination} | -t < vtar/vgz-file

In this example, we will copy the install.vgz to an ESXi host to perform some operations.

To list the contents of a vmtar file, you will need to use the -t option:

To extract the contents of vmtar file, you will need to use the -x and -o option:

vmtar -x install.vgz -o install.tar

Note: The output will be a standard tar file which will then need to be extracted before getting to the actual contents

To extract the tar file, we will be using the tar utility:

Let's say we made a change to one of the files and now we would like to re-create the vmtar file, we will first need to tar up the contents by using the tar utility again:

To verify the contents were all tarred up, we can view the contents by using the following command:

tar -tf install.tar

Now we will create the vmtar file using the vmtar utility:

vmtar -c install.tar -o install.vgz

We can confirm the contents by using vmtar -t option once again:

vmtar -t < install.vgz

If you decide to create your own custom vmtar files and want to verify the file layout, you can use vmkramdisk to assist you. Using vdf command, make note of the number of tardisks that have been mounted up.

Also make note of the filesystem layout by performing an "ls" on / (slash):

Now let's say you wanted to create a directory called virtuallyGhetto with a file in that directory called foobar and you wanted it to be mounted up under /

Here are the steps to perform the above:

Do you notice anything different? How about performing an "ls" on / (slash) again?

To umount the vmtar disk, you would use the following command:

vmkramdisk -u virtuallyGhetto.vgz

Automating Storage DRS & Datastore Cluster Management in vSphere 5

by // 1 Comment

Storage DRS is probably one of, if not the coolest feature in vSphere 5. Storage DRS allows you to cluster your datastores into what is known as a datastore cluster (storage pod) and automatically balances both your storage I/O and capacity just like DRS does with your compute. The user interface is extremely easy to use but as always, if you need to click through several screens to get to the outcome, some automation can never hurt 🙂

I decided to create a vSphere SDK for Perl script called datastoreClusterManagement.pl which allows you to automate all aspects of creating and managing your storage pod/cluster. You will need a system that has the vCLI installed or you can use VMware vMA 5 to run the script. You will also need to connect to a vCenter Server 5 for all SDRS operations.

The script supports 8 different types of operations and are describe below:

Operation Description
List List all available datastore clusters
Query  Query details for a specific datastore cluster
Create  Create a datastore cluster
Delete  Delete a datastore cluster (Datastores are left intact)
Add Datastore  Add datastore(s) to an existing datastore cluster
Remove Datastore  Remove datastore(s) from an existing datastore cluster
Enter Maintenance Mode  Put a datastore into maintenance mode
Exit Maintenance Mode  Take a datastore out of maintenance mode
Here is an example of performing the "list" operation: 

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation list

Here is an example of performing the "query" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation query --pod homer-NFS-pod

Here is an example of performing the "create" operation w/single datastore:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation create --datacenter MN-physical --enable_sdrs true --enable_sdrs_iometric true --pod moe-NFS-pod --sdrs_automation automated --sdrs_evaluate_period 480 --sdrs_imbal_thres 30 --sdrs_latency 15 --sdrs_util_diff 20 --sdrs_util_space 60 --datastore himalaya-NFS-moe-primp-1

Here is an example of performing the "create" operation w/datastore file:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation create --datacenter MN-physical --enable_sdrs true --enable_sdrs_iometric true --pod moe-NFS-pod --sdrs_automation automated --sdrs_evaluate_period 480 --sdrs_imbal_thres 30 --sdrs_latency 15 --sdrs_util_diff 20 --sdrs_util_space 60 --datastore_file dsfile

Here is an example of performing the "delete" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation delete --pod moe-NFS-pod

Here is an example of performing the "add_datastore" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation add_datastore --pod moe-NFS-pod --datastore himalaya-NFS-moe-primp-2

Here is an example of performing the "remove_datastore" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation remove_datastore --pod moe-NFS-pod --datastore himalaya-NFS-moe-primp-1

Note: Both "add_datastore" and "remove_datastore" operation support single datastore and/or datastore file

Here is an example of performing the "ent_maint" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation ent_maint --pod homer-NFS-pod --datastore himalaya-NFS-moe-primp-5

Here is an example of performing the "ext_maint" operation:

./datastoreClusterManagement.pl --server vcenter50-1 --username root --operation exi_maint --pod homer-NFS-pod --datastore himalaya-NFS-moe-primp-5

There is also complete perl docs for this script which can be called using the following command:

perldoc datastoreClusterManagement.pl