vsphere sdk for perl

Automating VCSA Network Configurations For Greenfield Deployments

02.17.2013 by William Lam // 6 Comments

If you deploy the VCSA (vCenter Server Appliance) or other virtual appliances directly onto an ESXi host,
you will notice the network configuration wizard for the virtual appliance is not available as you would expect when deploying to a vCenter Server.

The reason for this is that ESXi does not support some of the advanced OVF/OVA properties such as the Networking section and you will need to deploy the OVF/OVA to a vCenter Server to be able to configure these advanced options. This poses a problem if you need to deploy the VCSA in a greenfield environment where you will not have an existing vCenter Server running and you will be deploying directly to the ESXi host. Unless you have a DHCP enabled network, you will most likely need to manually go into the vSphere C# Client to change the network configuration as it was unable to obtain an IP Address.

Though this is a one time configuration, it is still not ideal and would require the use of a Windows system to access the vSphere C# Client. You can actually get around this by leveraging the GuestOperations API (previously known as VIX API) which allows you to perform operations within the guestOS that is running VMware Tools. The other nice thing about the GuestOperations API is that it does not require any network connectivity from the virtual machine.

Note: The GuestOperations API can be accessed in variety of ways and in this article I am demonstrating just two methods and does not require a Windows system. You can also access the GuestOperations API using PowerCLI if you are more comfortable with Windows and do not wish to use the vSphere C# Client to manually configure the network settings for the VCSA. I would also like to stress that though this article is about the VCSA, you can easily apply this to any VMware based virtual appliance or virtual appliance running VMware Tools.

The most important thing to identity before using the GuestOperations API is the specific command or program you wish to invoke and the argument it accepts. To configure the network configuration for the VCSA or any other VMware based virtual appliance, you would use /opt/vmware/share/vami/vami_set_network If you just run this command by itself, there are variety of options from IPv4 to IPv6, static or dhcp configuration. In our example, we will be configuring a Static IPv4 address for our VCSA and the command we would run is the following:

/opt/vmware/share/vami/vami_set_network "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1"

Method 1 - Using RVC (Ruby vSphere Console)

RVC is a nice open-source tool for interactively managing and configuring your vSphere infrastructure. RVC can be installed on any platform, in this example, I am running RVC on my Apple OS X laptop.

Step 1 - We first need to deploy the VCSA OVA and we can do so by using the ovftool via the command-line which can also be installed on Mac OS X system.

Step 2 - We then login to our ESXi host using RVC.

Step 3 - Next we will need to "change directory" to the location of our VM, in this example my VCSA is called VCSA-5.1. We can then run the "info ." command to view the summary of our VM. We can see that our VM is powered on from our initial deployment and we are ready to apply our network configurations in the next step.

Step 4 - To be able to run the above command, we will need to first authenticate into the guestOS. To do so, we will run the "vm_guest.authenticate ." and we will be prompted for the VCSA password. By default, the command assumes the username is root but that can also be specified on the command-line. If you are successful, you should not see any errors and then we can run the "vm_guest.start_program" command. Run the following to set a IPv4 static IP Address:

vm_guest.start_program . --program-path /opt/vmware/share/vami/vami_set_network --arguments "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1"

Note: All commands in RVC can be tabbed out with auto-completion.

If the command was successful, you can quit RVC and you should be able to ping the IP Address that you have just configured.

Method 2 - Using vSphere SDK for Perl Script

Awhile back I wrote a script called guestOperations.pl which is a vSphere SDK for Perl script that implements the new GuestOperations API. This is a generic script which can be used to remotely connect to either a vCenter Server or ESXi host and perform operations within a guestOS as long as VMware Tools is installed and running. In this example, I also have the vSphere SDK for Perl installed on my Mac OS X laptop, but you can also install this SDK on any platform as well.

Step 1 - We will first use the "validate" operation to ensure our credentials to the guestOS is correct, but more importantly ensure that VMware Tools is up and running.

If the operation was successful, we should see our guest credentials validated. If not, you may need to wait a minute or two while VMware Tools is still loading up.

Step 2 - To invoke the command to configure the network configuration, we will use the "startprog" operation and run the following:

./guestOpsManagement.pl --server mini --username root --guestusername root --vm VCSA-5.1 --operation startprog --program_path /opt/vmware/share/vami/vami_set_network --program_args "eth0 STATICV4 192.168.1.150 255.255.255.0 192.168.1.1" --working_dir /

If the command was successful, then you should now be able to ping the IP Address that you have just configured.

As you can see, with the use of the GuestOperations API, you can do more than just setup the network configuration for a VM, you can run pretty much any command within the guestOS as you normally would if you were to RDP or SSH in. This is a very powerful interface that you can leverage to help you automate your virtual machine deployment and configurations!

Retrieving vscsiStats Using the vSphere 5.1 API

01.17.2013 by William Lam // Leave a Comment

In my previous article, I talked about the new Service Manager API that was introduced in vSphere 5.1 and how you can retrieve ESXTOP performance data using this new vSphere API. In this article I will show you how to collect vscsiStats data using this same interface. If you are not familiar or have not used vscsiStats before, I would highly recommend you take a look at the Using vscsiStats for Storage Performance Analysis as it goes over some of the basics of vscsiStats and how it works.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.

Disclaimer: You should try to limit the use of these interfaces for statistics collection or debugging/troubleshooting purposes as there is a certain amount of overhead when running these commands. It is also important to note that since the output is based on the implementer of the service, there is no guarantee that output would not change from one release to the other.

The first step is to get a reference to the vscsiStats service via the Service Manager (must connect directly to an ESXi 5. host, this is not supported when connecting to vCenter Server) and to invoke an operation for vscsiStats, you will need to use the ExecuteSimpleCommand. For vscsiStats, there are four valid operations:

StartVscsiStats
FetchAllHistograms
ResetVscsiStats
StopVscsiStats

To demonstrate the vscsiStats interface, I have written a sample vSphere SDK for Perl script called getVscsiStats.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

StartVscsiStats

This operation starts the vscsiStats collection for ALL virtual machines residing on your ESXi hosts. This is exactly the same operation if you were to only specify the -s option to the vscsiStats command-line. Here is a screenshot of the "start" operation implemented in the script:

You should see a response of OK from the output and this would indicate the vscsiStats collection has started.

FetchAllHistograms

This operation fetches ALL the vscsiStats histogram data similar to specifying the -p All option in the vscsiStats command-line. The output contains the following:

The tag denotes the details about each Virtual Machine:

VM Display Name
VM VMX Configuration Path
VM BIOS UUID
vCenter Server UUID

This is then followed by the tag which provides the VMDK name in the format of scsi:X:Y and within each virtual disk section it will contain 13 tags which represents each of the statistics type and their associated values:

VSCSIVsi_DistanceHistogram: Histogram: distance (in LBNs) between successive commands
VSCSIVsi_DistanceLast16Histogram: Histogram: distance (in LBNs) between each command from the closest of previous 16
VSCSIVsi_DistanceReadsHistogram: Histogram: distance (in LBNs) between successive Read commands
VSCSIVsi_DistanceWritesHistogram: Histogram: distance (in LBNs) between successive Write commands
VSCSIVsi_IoLatencyHistogram: Histogram: latency of IOs in Microseconds (us)
VSCSIVsi_IoLatencyReadsHistogram: Histogram: latency of Read IOs in Microseconds (us)
VSCSIVsi_IoLatencyWritesHistogram: Histogram: latency of Write IOs in Microseconds (us)
VSCSIVsi_IoLengthHistogram: Histogram: IO lengths of commands
VSCSIVsi_IoLengthReadsHistogram: Histogram: IO lengths of Read commands
VSCSIVsi_IoLengthWritesHistogram: Histogram: IO lengths of Write commands
VSCSIVsi_OutstandingIOsHistogram: Histogram: number of outstanding IOs when a new IO is issued
VSCSIVsi_OutstandingIOsReadsHistogram: Histogram: number of outstanding Read IOs when a new Read IO is issued
VSCSIVsi_OutstandingIOsWritesHistogram: Histogram: number of outstanding Write IOs when a new Write IO is issued

Here is a screenshot of the "getstats" operation implemented in the script:

Note: In comparing the output between the vscsiStats command-line and this interface, I found the following three statistics are not available:

Histogram: latency of IO interarrival time in Microseconds (us)
Histogram: latency of IO interarrival time for Reads in Microseconds (us)
Histogram: latency of IO interarrival time for Writes in Microseconds (us)

ResetVscsiStats

This operation will reset the vscsiStats collection similar to the -r option in the vscsiStats command-line. Here is a screenshot of the "reset" operation implemented in the script:

StopVscsiStats

This operation will stop the vscsiStats collection similar to the -x option in the vscsiStats command-line. Make sure you perform this operation once you are done retrieving your vscsiStats data. Here is a screenshot of the "stop" operation implemented in the script:

In addition to the four operations, you can also save the output to a file by specifying the --output option along with the name of the file. vscsiStats is an extremely useful tool to help vSphere administrators profile their virtual machine's IO workload and now you can easily collect this information using the vSphere API. Some really cool things you can do with this data is to create some nifty graphs such as the ones here and here.

Retrieving ESXTOP Performance Data Using the vSphere 5.1 API

01.15.2013 by William Lam // 5 Comments

In vSphere 5.1, VMware introduced a new managed object called the Service Manager. The Service Manager is a generic object that wraps the execution of a single command and it requires a specific set of inputs to invoke a particular service command. This is particularly interesting as it allows users to access both the ESXTOP and vscsiStatsinterface using the vSphere API. Prior to vSphere 5.1, to use ESXTOP you would need to either login to the ESXi Shell to run the local ESXTOP command or connect remotely using the RESXTOP utility which is only available on a Linux system. For vScsiStats, you would need to login to the ESXi Shell as a remote version of this tool does not exist. The Service Manager used to be a private interface, an interesting tidbit is that some of you may have already interacted with this interface without even realizing it if you have used PowerCLI's Get-Esxtop cmdlet. In this article I will show you how to programmatically access ESXTOP using the vSphere API.

UPDATE 02/15/2017 - Please see this updated article here for how to access this API using vCenter Server.

Both the ESXTOP and vscsiStats services are only available when connecting directly to an ESXi 5.1 host, it is not available when connecting to a vCenter Server. If we browse over to the vSphere MOB, we can clearly see the two services:

The first step is to get a reference to the ESXTOP service via the Service Manager and to invoke an operation for ESXTOP, you will need to use the ExecuteSimpleCommand. For ESXTOP, there are three valid operations:

CounterInfo
FetchStats
FreeStats

To demonstrate the ESXTOP interface, I have written a sample vSphere SDK for Perl script called getEsxtop.pl which I will use to explain each operation. Please note the data set that is retrieved is in it's raw data form and requires a bit of data processing.

CounterInfo

This operation only needs to be invoked once and it will provide you with the list of available counters and their associated properties and data types for a given ESXi host. Here is an example of this using the "getcounters" operation implemented in the script:

Each line represents a specific counter type followed by each property name and their data type. For example, the first line is for the Server counter and has the following properties and types:

Property Name	Type
MinFetchIntervalInUsec	U64
IsVMVisor	B
TimeStampInUsec	U64
Time	S64

Here is a quick diagram to help you visualize the hierarchy of all the ESXTOP counters and their relationships with one another:

Note: This diagram was created using yuml.me and here is the raw text in JSON format if you are interested.

FetchStats

This operation fetches a single snapshot of ALL the ESXTOP statistics which contains two pieces of information:

The topology of the counter instances
The actual counter instances values

The first section is denoted by ==NUM-OF-OBJECTS== which contains either inventory data that does not change or counter instance structure which describes the relationship between the different counter instances. Here is an example of the first section using the "getstats" operation implemented in the script:

If we take a look at the second line as an example |PCPU|LCPU,24|Core,12|Package,2| we can see that PCPU counter contains 24 LCPU that you would need to then enumerate as well as inventory information describing the CPU's logical cores and physical socket.

To view the enumerated counter instances and their instance values, we need to look in the second portion of the data which is denoted by ==COUNTER-VALUE== within the output. Here is a screenshot of this section and we can see the enumerated LCPU's (24 in total as denoted earlier) and their associated instance values:

Remember you will need to correlate with the counter definitions that was extracted earlier from the "getcounters" and this will help you build up the data. I do have to say it can be a bit confusing when you first look at the raw data, but as you start to play with it a bit more, it will start to make sense. Two useful references that can help with parsing the data is the ESXTOP bible and an article that Luc Dekens wrote awhile back exploring the Get-Esxtop cmdlet which I mentioned earlier leverages this exact interface.

FreeStats

Lastly, once you are done collecting the ESXTOP data, you will need to run the "freestats" operation and this will release any server side resources used during the collection. When this command is invoked, it will free up all resources even for past collections where you might have forgotten to perform this last step. There is no output from this operation as you can see from the example screenshot below:

Even though it is nice to see the ESXTOP interface be accessible via the vSphere API, it is not the easiest interface to use and is definitely geared more towards a developer. For extracting general performance data, I would still recommend using the Performance Manager managed object or one of the above mentioned command-line interfaces. In the next article, I go into more detail about the vscsiStats interface and how to consume it using the vSphere API.