vSphere 4.1

Did you know that VMware Host Profile is extensible by 3rd Parties?

07.24.2013 by William Lam // 1 Comment

Managing ESXi host configurations can be challenging and the potential risk for configuration drift between the running environment and the set of configuration scripts or worse, manual configuration is quite high. On top of that, how do you ensure proper compliance of all your ESXi host configurations in your environment and easily prove that in an internal or security audit?

This is where VMware Host Profile can help which allows administrators to capture the running configurations of an ESXi host and automatically creating a template (Host Profile) that can then be applied across new or existing ESXi hosts. By leveraging Host Profile, administrators can ensure that all their ESXi host configurations are always consistent and configuration drifts can easily be prevented through automatic compliance checks.

Recently, while searching for something on VMware's HCL website, I accidentally stumbled onto what appears to be 3rd party Host Profiles? There were only two listed, one from Brocade for managing and configuring Brocade storage adapters and the other from Dell for managing and configuring Dell's EqualLogic MEM (Multipathing Extension Module). I was actually quite surprise to learn about these custom 3rd party Host Profiles. In doing a bit of digging and research it turns out that VMware Host Profile are in fact extensible by design, which was something new to me.

Note: For a technical overview of Host Profile, you can take a look at this whitepaper here.

Host Profile Architecture

Host Profile was first introduced with the release of vSphere 4.1 and the brain of the system is known as the Host Profile Engine which was part of the vCenter Server. In vSphere 5.0, Host Profile was re-architected and the Host Profile Engine was moved into the ESXi host which allowed for Host Profile Plugins to be added to an ESXi Image and expose new Host Profiles through the Host Profile Engine.

A Host Profile is actually a hierarchical composition of multiple sub-profiles and policies. Each policy defines a set of parameters that a user can select from and apply to an ESXi host. For instance, the default VMware Host Profile is composed up of 12 individual sub-profiles: authentication, datetime, firewall, memory, network, option, security, service, storage, userAccount and userGroupAccount.

With this new re-architecutre, Host Profile can be extended by 3rd party partners/vendors to create custom Host Profile Plugins to expose vendor specific hardware or software configurations and made available through a common Host Profile API/UI for customers to consume.

Host Profile Extensibility Options

To build a Host Profile Plugin, you will need to use the Host Profile SDK which is only available as part of VMware TAP (Technology Alliance Partner) Program. A Host Profile Plugin basically wraps the actual configuration work and can be backed by one of three ways:

CIM Provider using the CIM SDK
ESXCLI plugins
Userworld binaries

As you can see, creating a Host Profile Plugin is quite flexible and can be exposed through several mechanisms. The most shocking discovery that I found was the lack of 3rd party vendor Host Profiles that exists today, especially from the big server hardware vendors. Coming from a Systems Administrator background, I would loved to have been able to configure and manage my server's firmware, BIOS, out-of-band management (iLO/DRAC), etc. through either a custom ESXCLI plugin or Host Profile Plugin. This would really benefit customers from having to manage configurations using multiple tools and allowing them centralize their management including compliance capabilities all from a single interface.

Hopefully this was an educational post for everyone and if you are a customer and would like to see certain functionality exposed by our 3rd party partners, feel free to leave a message and perhaps one of them may consider adding a custom Host Profile Plugin 🙂

VMware officially releases vibddi for vSphere 4.1

09.18.2011 by William Lam // Leave a Comment

There were several product releases last week that got a lot of buzz on the inter-tube:

However, VMware actually released an additional product last week which snuck under the radar, vibddi.

I actually wrote about this unsupported and undocumented utility last year: How to inject custom drivers into an ESXi 4.1 image using vibddi? vibddi (pronounced vib d-d-i) stands for VIB (vSphere Installation Bundle) Disk Dump Image and it is a utility to help users easily customize ESXi images with custom drivers. This utility first appeared in the vSphere Auto Deploy appliance and it looks like VMware has finally released it as an official tool to support vSphere 4.1 image customization. You also may have heard about the new Image Builder tool with the release of vSphere 5, the origins of that utility actually came from vibddi.

If you are still using vSphere 4.1 and need to inject or modify drivers, I would highly recommend you take a look at the tool as it is extremely simple to use. For more details, please check out the new VMware KB article 2003316 documenting the details of the utility or my blog post. If you are using vSphere 5, you will need to use Image Builder as vSphere 4.1 is not supported and vice-a-versa with ESXi 5 with vibddi.

Note: There are some changes in the latest vibddi utility compared to the one found in the vSphere Auto Deploy such as injecting custom kickstart configuration file or license file. If you rely on these features, you may want to use the older version or manually update these after the system build.

New Application Awareness API in vSphere 5

08.25.2011 by William Lam // 12 Comments

Application Awareness HA is not a new feature in vSphere 5, it has actually has been around since vSphere 4.1. With this feature, vSphere HA can monitor heartbeats generated from an application monitor within the guestOS and reboot the virtual machine.

What is actually new in vSphere 5 is the availability of the Application Awareness API for anyone to consume and integrate into their own application and/or script. Prior to this, the API was only exposed to ISV and 3rd party vendors with solutions such as Symantec's ApplicationHA and Neverfail's vAppHA.

The Application Awareness API (will be shorthanded as AAA, going forward) is supported in both Linux and Windows (32/64bit) and can be accessed by installing a package within the guestOS. This package includes the necessary AAA libraries to create your own program/scripts in C, C++, Java and Perl. In addition, the package also includes a pre-compiled binary (vmware-appmonitor) that implements all the AAA methods that can easily be called from within a script or program. AAA uses the VMware Tools as communication channel to the ESX(i) host and you will need to ensure VMware Tools is installed and running. Since the communication is between VMware Tools and the ESX(i) host, there is no reliance on a TCP/IP network for this communication channel.

UPDATE: You can download GuestAppMonitor SDK here.

There are currently 6 AAA methods:

VMGuestAppMonitor_Enable()
- Enables Monitoring

VMGuestAppMonitor_MarkActive()
- Call every 30 seconds to mark application as active

VMGuestAppMonitor_Disable()
- Disable Monitoring

VMGuestAppMonitor_IsEnabled()
- Returns status of Monitoring

VMGuestAppMonitor_GetAppStatus()
- Returns the current application status recorded for the application, three possible values:
  - green = Virtual machine infrastructure acknowledges that the application is being monitored.
  - red = Virtual machine infrastructure does not think the application is being monitored. The
    HA monitoring agent will initialize an asynchronous reset on the virtual machine if the status is Red
  - gray = Application should send VMGuestAppMonitor_Enable again, followed
    by VMGuestAppMonitor_MarkActive, because either application monitoring failed, or the virtual machine was vMotioned to a different location

VMGuestAppMonitor_Free()
- Frees the result of the *_GetAppStatus() call (only required when writing your own program)

Here is the basic workflow for using AAA within your application:

Check_if_enabled();

If not enabled, set enabled

Monitor application

If application is good, send heartbeat

Wait 15 seconds

Loop

To start using AAA functionality, you will first need to have a vSphere HA enabled cluster and enable the "VM and Application Monitoring" under VM Monitoring.

You have the ability to configure the sensitivity of AAA from Low, Medium and High which correlates to the heartbeat interval and frequency of virtual machine reboots. You also have the option of configuring your own custom policy.

Lastly, you can choose which virtual machines will be included in VM Monitoring and/or Application Monitoring.

Note: It is important to note, that HA will expect an application heartbeat to be generated every 30secs. If HA fails to receive a heartbeat within 30secs, it will transition the appHeartbeatStatus state from green to red. Depending on the configured sensitivity policy, once the heartbeat interval has been violated, HA will then restart the virtual machine. For example, if you have the sensitivity configured to medium and a heartbeat was not received within 30secs, it will change to a red state. If HA still has not received a heartbeat within 60secs of that time, then it will reboot the virtual machine.

Here is an example of installing AAA on a Linux system and compiling the C sample program:

Step 1 - Copy the AAA package to Linux system and extract the contents

Step 2 - Change into VMware-GuestAppMonitorSDK/docs/samples/C/ and ensure you have gcc compiler. You may have to change the makefile if you are on a 64bit platform as it is configured by default to point to the 32bit library. When you are ready, just type "make" and you should get compiled binary called "sample" which is the sample C application

Before you run the application, you need to ensure that your shared library path variable LD_LIBRARY_PATH has been updated to include the libappmonitorlib.so. To update the variable, you will run the following command:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/full/path/to/VMware-GuestAppMonitorSDK/lib64

Step 3 - You can now run the "sample" application which runs in a continuous loop and automatically enables AAA within the virtual machine and sends heartbeats to ESX(i) host. You can press control+C which then brings up three options: stop (s), disable (d), continue (c). The last two options should be pretty straight forward, but if you decide to stop the heartbeating and you don't resume, you will see HA restart the virtual machine based on your cluster configuration.

As you can see, once the heartbeats have not been received within the specified interval, HA will take action and reboot the virtual machine as expected. Here is a detail of the events as seen by vCenter and the HA cluster:

Here is an example of installing AAA on a Windows system and using the pre-compiled vmware-appmonitor binary:

Step1 - Copy the AAA package to Windows system and extract the contents

Step 2 - Launch a cmd prompt and change into C:\Users\Administrator\Desktop\VMware-GuestAppMonitorSDK-457765\VMware-GuestAppMonitorSDK\bin\win64 directory. Depending if you are on a 32/64bit OS, you will need to modify the win{32,64}

Step 3 - Run the vmware-appmonitor.exe which will then provide you with options: enable, disable, markActive, isEnabled, getAppStatus

Note: The options in vmware-appmonitor for both Linux and Windows are exactly the same, this is very nice for consistency and scripting purposes. Just like with the direct use of the API, you need to first run the enable command to turn on Application monitoring and then run the markActive command which sends the heartbeats. You can always check the current heartbeat status by running getAppStatus or check whether AAA is enabled by running isEnabled command

As a reference, here are the paths to the vmware-appmonitor for both Linux and Windows:

VMware-GuestAppMonitorSDK/bin/bin{32,64}/vmware-appmonitor
VMware-GuestAppMonitorSDK-457765\VMware-GuestAppMonitorSDK\bin\win{32,64}\vmware-appmonitor.exe

For those of you who are not into programming languages such as C,C++ and Java, here is an example using Perl. In the example, the script simulates the monitoring of an application by checking whether or not a file exists. The script starts off by creating a file that will monitored and then loops for 5 minutes and checks for the existence of the file. Once the 5 minutes are up, the script then disables Application monitoring and exits.

Note: You will need to set the two variables at the top which define the path to the shared library and the vmware-appmonitor binary.

So far we have demonstrated on how to setup AAA within the guestOS and provided a variety of programming/scripting interfaces such as C,C++, Java and Perl to integrate with your own application/script. Now what if we wanted to extract the heartbeat status for all virtual machines that have AAA implemented going through vCenter? You can easily do so by using the vSphere API and querying the appHeartbeatStatus property of your virtual machine.

I wrote a very simple vSphere SDK for Perl script getVMAppStatus.pl that allows you to query a virtual machine connecting to either vCenter or directly to an ESX(i) host to extract the heartbeat status.

Download the getVMAppStatus.pl script here.

The script can return three different status: gray, green or red and the definition for each is defined above.

Now before you jump right in and start leveraging this awesome API in either a custom application or script, you need to understand your application and various ways on detecting that it has failed and when you would like vSphere HA to reboot the virtual machine. Simply checking whether the process is running may or may not be enough.

To get more details on some of the best practices around using the Application Awareness API, I would highly recommend you check out Tom Stephens upcoming VMworld 2011 presentation TEX1928 Implementing Application Awareness in the Web Client and The Uptime Blog for more details about implementing AAA . For now, if you would like to learn more about Application Awareness API, check out last year's VMworld presentation.