WilliamLam.com

Automating VCSA Network Configurations For Greenfield Deployments

by // 6 Comments

If you deploy the VCSA (vCenter Server Appliance) or other virtual appliances directly onto an ESXi host,
you will notice the network configuration wizard for the virtual appliance is not available as you would expect when deploying to a vCenter Server.

The reason for this is that ESXi does not support some of the advanced OVF/OVA properties such as the Networking section and you will need to deploy the OVF/OVA to a vCenter Server to be able to configure these advanced options. This poses a problem if you need to deploy the VCSA in a greenfield environment where you will not have an existing vCenter Server running and you will be deploying directly to the ESXi host. Unless you have a DHCP enabled network, you will most likely need to manually go into the vSphere C# Client to change the network configuration as it was unable to obtain an IP Address.

Though this is a one time configuration, it is still not ideal and would require the use of a Windows system to access the vSphere C# Client. You can actually get around this by leveraging the GuestOperations API (previously known as VIX API) which allows you to perform operations within the guestOS that is running VMware Tools. The other nice thing about the GuestOperations API is that it does not require any network connectivity from the virtual machine.

Note: The GuestOperations API can be accessed in variety of ways and in this article I am demonstrating just two methods and does not require a Windows system. You can also access the GuestOperations API using PowerCLI if you are more comfortable with Windows and do not wish to use the vSphere C# Client to manually configure the network settings for the VCSA. I would also like to stress that though this article is about the VCSA, you can easily apply this to any VMware based virtual appliance or virtual appliance running VMware Tools.

The most important thing to identity before using the GuestOperations API is the specific command or program you wish to invoke and the argument it accepts. To configure the network configuration for the VCSA or any other VMware based virtual appliance, you would use /opt/vmware/share/vami/vami_set_network If you just run this command by itself, there are variety of options from IPv4 to IPv6, static or dhcp configuration. In our example, we will be configuring a Static IPv4 address for our VCSA and the command we would run is the following:

/opt/vmware/share/vami/vami_set_network "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1"

Method 1 - Using RVC (Ruby vSphere Console)

 

RVC is a nice open-source tool for interactively managing and configuring your vSphere infrastructure. RVC can be installed on any platform, in this example, I am running RVC on my Apple OS X laptop.

Step 1 - We first need to deploy the VCSA OVA and we can do so by using the ovftool via the command-line which can also be installed on Mac OS X system.

Step 2 - We then login to our ESXi host using RVC.

Step 3 - Next we will need to "change directory" to the location of our VM, in this example my VCSA is called VCSA-5.1. We can then run the "info ." command to view the summary of our VM. We can see that our VM is powered on from our initial deployment and we are ready to apply our network configurations in the next step.

Step 4 - To be able to run the above command, we will need to first authenticate into the guestOS. To do so, we will run the "vm_guest.authenticate ." and we will be prompted for the VCSA password. By default, the command assumes the username is root but that can also be specified on the command-line. If you are successful, you should not see any errors and then we can run the "vm_guest.start_program" command. Run the following to set a IPv4 static IP Address:

vm_guest.start_program . --program-path /opt/vmware/share/vami/vami_set_network --arguments "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1"

Note: All commands in RVC can be tabbed out with auto-completion.

If the command was successful, you can quit RVC and you should be able to ping the IP Address that you have just configured.

Method 2 - Using vSphere SDK for Perl Script

 

Awhile back I wrote a script called guestOperations.pl which is a vSphere SDK for Perl script that implements the new GuestOperations API. This is a generic script which can be used to remotely connect to either a vCenter Server or ESXi host and perform operations within a guestOS as long as VMware Tools is installed and running. In this example, I also have the vSphere SDK for Perl installed on my Mac OS X laptop, but you can also install this SDK on any platform as well.

Step 1 - We will first use the "validate" operation to ensure our credentials to the guestOS is correct, but more importantly ensure that VMware Tools is up and running.

If the operation was successful, we should see our guest credentials validated. If not, you may need to wait a minute or two while VMware Tools is still loading up.

Step 2 - To invoke the command to configure the network configuration, we will use the "startprog" operation and run the following:

./guestOpsManagement.pl --server mini --username root --guestusername root --vm VCSA-5.1 --operation startprog --program_path /opt/vmware/share/vami/vami_set_network --program_args "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1" --working_dir /

If the command was successful, then you should now be able to ping the IP Address that you have just configured.

As you can see, with the use of the GuestOperations API, you can do more than just setup the network configuration for a VM, you can run pretty much any command within the guestOS as you normally would if you were to RDP or SSH in. This is a very powerful interface that you can leverage to help you automate your virtual machine deployment and configurations!

Monitoring vCenter SSO User Account Expiration

by // 2 Comments

Did you know that user accounts created in the vCenter SSO Server automatically expire by default after 365 days? If you do not update your password prior to the expiration date, in about a years time you could potentially be locked out of your vCenter SSO Server which also applies to the admin@system-domain account.

You can change the default password expiration policy by logging into the vSphere Web Client with an SSO Administrator account. Under the configuration section of "Sign-On and Discovery", there is a Password Policies tab that allows you can modify password policies. By default, this is set to 365 days. I would also recommend that after you have installed and setup your vCenter SSO Server, you add at least one user or group from your directory service such as Active Directory and assign it to the SSO Administrator group. This will ensure that you can still log in to the SSO configuration in the event the local SSO user accounts are locked out.

Even though you can change the password expiration policy, there is still no automated notification or alerting built-in for user accounts that are going to expire. The best you can do is to create a calendar event to remind you update your passwords prior to the expiration date. I am sure that many of you are anxious to add another color event to your already busy schedule 🙂

While investigating alternative options a few weeks back, the only method that I have found to retrieve the status for each SSO user is to directly connect to the vCenter SSO Database. There are two specific tables of interest, one which provides the current password policy and the other providing the last password changed date for each SSO user:

  • ims_authn_password_policy
  • ims_principal_data

Disclaimer: This "may" not be officially supported by VMware.

Instead of having you manually dig around in the SSO Database, I have created a Perl script called getSSOUserExpiration.pl which can connect to either a MSSQL or vPostgress backend SSO database. The script which will automatically list out the current password policy as well as user accounts that will be expiring in N days, where N is input provided by the user. You also have the ability to configure the script to automatically email you the results which is nice for a daily or weekly report and can be setup using a cronjob or a scheduled task. There are several configuration variables that will need to be adjusted based on your environment and these are all located within the script itself. For more details on how to setup and use the script, please take a look at the Setup and Configuration section below.

Note: To reduce any negative impact to the vCenter SSO Database, you should add or ask your DBA's to create a limited read-only account and limit access to the following tables above. You may even be able to have your DBA's create a scheduled routine for the specific queries and have that emailed to you internally.

Here is a screenshot of connecting to a vPostgres backend Database:

Here is a screenshot of connecting to a MSSQL backend database:

Here is a screenshot of what the email report looks like:

Note: The email body should contain the specific vCenter SSO Database, but I am not sure why it is not showing up in Gmail, but it does work for other email clients.

Setup and Configuration

vPostgres

To connect to a vPostgres DB, you will need to install the following two perl packages: perl-DBI and perl-DBD-Pg. In this example, I am using the vMA appliance and the zypper package installer. For more details on how to add a SLES repo, please take a look at the following article. I also assume if you are connecting to a vPostgres DB, then you are using the VCSA (vCenter Server Appliance) and by default it does not accept remote connections. We will need to also make two configuration changes to the VCSA for our script to be able to connect to the database.

Step 1 - Run the following two commands to install both perl packages:

sudo zypper in perl-DBI
sudo zypper in perl-DBD-Pg

Step 2 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/pg_hba.conf you will need to add the network in which you will be connecting from:

host    all             all             172.30.0.0/24           trust

Step 3 - SSH into your VCSA and in the following configuration file /storage/db/vpostgres/postgresql.conf you will need to add the IP Address(s) that you want vPostgres to listen for remote connection. If you use "*", it will allow all addressees:

listen_addresses = '*'

Step 4 - For the changes to go into effect, you will need to restart the vPostgres DB by running the following command:

service vmware-vpostgres restart

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB. If you need help in identifying the vCenter SSO DB credentials, please refer to this article for the details.

MSSQL

To connect to an MSSQL DB, there are a few additional steps and packages that will be required. We will be using FreeTDS which provides libraries to connect to an MSSQL DB for UNIX/Linux platforms. There was a bit of trial and error in getting the MSSQL solution working and I would like to thank Reuben Stump for his assistance. The following article was used as a reference for the setup below.

Step 1 - Run the following two commands to install the required packages:

sudo zypper in perl-DBI
sudo zypper in gcc

Step 2 - Download and extract the contents of the FreeTDS package:

wget ftp://ftp.astron.com/pub/freetds/stable/freetds-stable.tgz
tar -zxvf freetds-stable.tgz
cd freetds-0.91

Step 3 - Compile and install FreeTDS under /usr/local/freetds:

export SYBASE=/usr/local/freetds/
./configure --prefix=/usr/local/freetds
make
sudo make install

Step 4 - Add your vCenter SSO Server details into the FreeTDS configuration file located in /usr/local/freetds/etc/freetds.conf

[sso]
host = 172.30.0.239
port = 1433
tds version = 7.0

In the example above, I named my database entry "sso" but you can use any name and this will be referenced when editing the script in step 5.

Step 5 - Modify the getSSOUserExpiration.pl script and provide the credentials to your vCenter SSO DB.

Retrieving vscsiStats Using the vSphere 5.1 API

by // Leave a Comment

In my previous article, I talked about the new Service Manager API that was introduced in vSphere 5.1 and how you can retrieve ESXTOP performance data using this new vSphere API. In this article I will show you how to collect vscsiStats data using this same interface. If you are not familiar or have not used vscsiStats before, I would highly recommend you take a look at the Using vscsiStats for Storage Performance Analysis as it goes over some of the basics of vscsiStats and how it works.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.

Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

The first step is to get a reference to the vscsiStats service via the Service Manager (must connect directly to an ESXi 5. host, this is not supported when connecting to vCenter Server) and to invoke an operation for vscsiStats, you will need to use the ExecuteSimpleCommand. For vscsiStats, there are four valid operations:

  • StartVscsiStats
  • FetchAllHistograms
  • ResetVscsiStats
  • StopVscsiStats

To demonstrate the vscsiStats interface, I have written a sample vSphere SDK for Perl script called getVscsiStats.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

StartVscsiStats

This operation starts the vscsiStats collection for ALL virtual machines residing on your ESXi hosts. This is exactly the same operation if you were to only specify the -s option to the vscsiStats command-line. Here is a screenshot of the "start" operation implemented in the script:

You should see a response of OK from the output and this would indicate the vscsiStats collection has started.

FetchAllHistograms

This operation fetches ALL the vscsiStats histogram data similar to specifying the -p All option in the vscsiStats command-line. The output contains the following:

The tag denotes the details about each Virtual Machine:

  • VM Display Name
  • VM VMX Configuration Path
  • VM BIOS UUID
  • vCenter Server UUID

This is then followed by the tag which provides the VMDK name in the format of scsi:X:Y and within each virtual disk section it will contain 13 tags which represents each of the statistics type and their associated values:

  1. VSCSIVsi_DistanceHistogram: Histogram: distance (in LBNs) between successive commands
  2. VSCSIVsi_DistanceLast16Histogram: Histogram: distance (in LBNs) between each command from the closest of previous 16
  3. VSCSIVsi_DistanceReadsHistogram: Histogram: distance (in LBNs) between successive Read commands
  4. VSCSIVsi_DistanceWritesHistogram: Histogram: distance (in LBNs) between successive Write commands
  5. VSCSIVsi_IoLatencyHistogram: Histogram: latency of IOs in Microseconds (us)
  6. VSCSIVsi_IoLatencyReadsHistogram: Histogram: latency of Read IOs in Microseconds (us)
  7. VSCSIVsi_IoLatencyWritesHistogram: Histogram: latency of Write IOs in Microseconds (us)
  8. VSCSIVsi_IoLengthHistogram: Histogram: IO lengths of commands
  9. VSCSIVsi_IoLengthReadsHistogram: Histogram: IO lengths of Read commands
  10. VSCSIVsi_IoLengthWritesHistogram: Histogram: IO lengths of Write commands
  11. VSCSIVsi_OutstandingIOsHistogram: Histogram: number of outstanding IOs when a new IO is issued
  12. VSCSIVsi_OutstandingIOsReadsHistogram: Histogram: number of outstanding Read IOs when a new Read IO is issued
  13. VSCSIVsi_OutstandingIOsWritesHistogram: Histogram: number of outstanding Write IOs when a new Write IO is issued

Here is a screenshot of the "getstats" operation implemented in the script:

Note: In comparing the output between the vscsiStats command-line and this interface, I found the following three statistics are not available:

  • Histogram: latency of IO interarrival time in Microseconds (us)
  • Histogram: latency of IO interarrival time for Reads in Microseconds (us)
  • Histogram: latency of IO interarrival time for Writes in Microseconds (us)

 

ResetVscsiStats

This operation will reset the vscsiStats collection similar to the -r option in the vscsiStats command-line. Here is a screenshot of the "reset" operation implemented in the script:

StopVscsiStats 

This operation will stop the vscsiStats collection similar to the -x option in the vscsiStats command-line. Make sure you perform this operation once you are done retrieving your vscsiStats data. Here is a screenshot of the "stop" operation implemented in the script:

In addition to the four operations, you can also save the output to a file by specifying the --output option along with the name of the file. vscsiStats is an extremely useful tool to help vSphere administrators profile their virtual machine's IO workload and now you can easily collect this information using the vSphere API. Some really cool things you can do with this data is to create some nifty graphs such as the ones here and here.