Application Awareness HA is not a new feature in vSphere 5, it has actually has been around since vSphere 4.1. With this feature, vSphere HA can monitor heartbeats generated from an application monitor within the guestOS and reboot the virtual machine.
What is actually new in vSphere 5 is the availability of the Application Awareness API for anyone to consume and integrate into their own application and/or script. Prior to this, the API was only exposed to ISV and 3rd party vendors with solutions such as Symantec's ApplicationHA and Neverfail's vAppHA.
The Application Awareness API (will be shorthanded as AAA, going forward) is supported in both Linux and Windows (32/64bit) and can be accessed by installing a package within the guestOS. This package includes the necessary AAA libraries to create your own program/scripts in C, C++, Java and Perl. In addition, the package also includes a pre-compiled binary (vmware-appmonitor) that implements all the AAA methods that can easily be called from within a script or program. AAA uses the VMware Tools as communication channel to the ESX(i) host and you will need to ensure VMware Tools is installed and running. Since the communication is between VMware Tools and the ESX(i) host, there is no reliance on a TCP/IP network for this communication channel.
UPDATE: You can download GuestAppMonitor SDK here.
There are currently 6 AAA methods:
- VMGuestAppMonitor_Enable()
- Enables Monitoring
- VMGuestAppMonitor_MarkActive()
- Call every 30 seconds to mark application as active
- VMGuestAppMonitor_Disable()
- Disable Monitoring
- VMGuestAppMonitor_IsEnabled()
- Returns status of Monitoring
- VMGuestAppMonitor_GetAppStatus()
- Returns the current application status recorded for the application, three possible values:
- green = Virtual machine infrastructure acknowledges that the application is being monitored.
- red = Virtual machine infrastructure does not think the application is being monitored. The
HA monitoring agent will initialize an asynchronous reset on the virtual machine if the status is Red - gray = Application should send VMGuestAppMonitor_Enable again, followed
by VMGuestAppMonitor_MarkActive, because either application monitoring failed, or the virtual machine was vMotioned to a different location
- Returns the current application status recorded for the application, three possible values:
- VMGuestAppMonitor_Free()
- Frees the result of the *_GetAppStatus() call (only required when writing your own program)
Here is the basic workflow for using AAA within your application:
To start using AAA functionality, you will first need to have a vSphere HA enabled cluster and enable the "VM and Application Monitoring" under VM Monitoring.
You have the ability to configure the sensitivity of AAA from Low, Medium and High which correlates to the heartbeat interval and frequency of virtual machine reboots. You also have the option of configuring your own custom policy.
Lastly, you can choose which virtual machines will be included in VM Monitoring and/or Application Monitoring.
Note: It is important to note, that HA will expect an application heartbeat to be generated every 30secs. If HA fails to receive a heartbeat within 30secs, it will transition the appHeartbeatStatus state from green to red. Depending on the configured sensitivity policy, once the heartbeat interval has been violated, HA will then restart the virtual machine. For example, if you have the sensitivity configured to medium and a heartbeat was not received within 30secs, it will change to a red state. If HA still has not received a heartbeat within 60secs of that time, then it will reboot the virtual machine.
Step 1 - Copy the AAA package to Linux system and extract the contents
Step 2 - Change into VMware-GuestAppMonitorSDK/docs/samples/C/ and ensure you have gcc compiler. You may have to change the makefile if you are on a 64bit platform as it is configured by default to point to the 32bit library. When you are ready, just type "make" and you should get compiled binary called "sample" which is the sample C application
Before you run the application, you need to ensure that your shared library path variable LD_LIBRARY_PATH has been updated to include the libappmonitorlib.so. To update the variable, you will run the following command:
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/full/path/to/VMware-GuestAppMonitorSDK/lib64
Step 3 - You can now run the "sample" application which runs in a continuous loop and automatically enables AAA within the virtual machine and sends heartbeats to ESX(i) host. You can press control+C which then brings up three options: stop (s), disable (d), continue (c). The last two options should be pretty straight forward, but if you decide to stop the heartbeating and you don't resume, you will see HA restart the virtual machine based on your cluster configuration.
As you can see, once the heartbeats have not been received within the specified interval, HA will take action and reboot the virtual machine as expected. Here is a detail of the events as seen by vCenter and the HA cluster:
Here is an example of installing AAA on a Windows system and using the pre-compiled vmware-appmonitor binary:
Step1 - Copy the AAA package to Windows system and extract the contents
Step 2 - Launch a cmd prompt and change into C:\Users\Administrator\Desktop\VMware-GuestAppMonitorSDK-457765\VMware-GuestAppMonitorSDK\bin\win64 directory. Depending if you are on a 32/64bit OS, you will need to modify the win{32,64}
Step 3 - Run the vmware-appmonitor.exe which will then provide you with options: enable, disable, markActive, isEnabled, getAppStatus
Note: The options in vmware-appmonitor for both Linux and Windows are exactly the same, this is very nice for consistency and scripting purposes. Just like with the direct use of the API, you need to first run the enable command to turn on Application monitoring and then run the markActive command which sends the heartbeats. You can always check the current heartbeat status by running getAppStatus or check whether AAA is enabled by running isEnabled command
As a reference, here are the paths to the vmware-appmonitor for both Linux and Windows:
- VMware-GuestAppMonitorSDK/bin/bin{32,64}/vmware-appmonitor
- VMware-GuestAppMonitorSDK-457765\VMware-GuestAppMonitorSDK\bin\win{32,64}\vmware-appmonitor.exe
For those of you who are not into programming languages such as C,C++ and Java, here is an example using Perl. In the example, the script simulates the monitoring of an application by checking whether or not a file exists. The script starts off by creating a file that will monitored and then loops for 5 minutes and checks for the existence of the file. Once the 5 minutes are up, the script then disables Application monitoring and exits.
Note: You will need to set the two variables at the top which define the path to the shared library and the vmware-appmonitor binary.
So far we have demonstrated on how to setup AAA within the guestOS and provided a variety of programming/scripting interfaces such as C,C++, Java and Perl to integrate with your own application/script. Now what if we wanted to extract the heartbeat status for all virtual machines that have AAA implemented going through vCenter? You can easily do so by using the vSphere API and querying the appHeartbeatStatus property of your virtual machine.
I wrote a very simple vSphere SDK for Perl script getVMAppStatus.pl that allows you to query a virtual machine connecting to either vCenter or directly to an ESX(i) host to extract the heartbeat status.
Download the getVMAppStatus.pl script here.
The script can return three different status: gray, green or red and the definition for each is defined above.
Now before you jump right in and start leveraging this awesome API in either a custom application or script, you need to understand your application and various ways on detecting that it has failed and when you would like vSphere HA to reboot the virtual machine. Simply checking whether the process is running may or may not be enough.
To get more details on some of the best practices around using the Application Awareness API, I would highly recommend you check out Tom Stephens upcoming VMworld 2011 presentation TEX1928 Implementing Application Awareness in the Web Client and The Uptime Blog for more details about implementing AAA . For now, if you would like to learn more about Application Awareness API, check out last year's VMworld presentation.
New Hidden CBRC (Content-Based Read Cache) Feature in vSphere 5 & for VMware View 5?
CBRC (Content-Based Read Cache) is another new/hidden feature of vSphere 5, not to be confused with the new Host Cache feature (swap to host cache). I initially thought CBRC was related to host cache and that it might have been an internal name for the feature. In a recent discussion on the VMTN community forums, a reader pointed out that CBRC is different and there is not whole lot of information on how it works. I decided to perform a quick Google search to see if anyone has written about this feature and one site had an interesting quote from a VMware sales rep on how CBRC works:
Content-Based Read Cache. A content-based read cache (CBRC) has been delivered for specific use with View (VDI) workloads. With this option configured in ESX, a read cache is constructed in memory optimized for recognizing, handling, and deduplicating VDI client images. The cache is managed from within the View Composer and delivers a significant reduction, as high as 90% by early estimates, in IOPS from each ESX host to the storage platform holding client images. This reduction in IOPS enables large scaling of the number of clients in case multiple I/O storms, typical in large VDI deployments, occur.
It looks like CBRC is implemented within hypervisor but it will be leveraged by VMware View 5 around provisioning Linked Clones? One question I had was whether or not this constructed cache in memory could be used without VMware View?
There are two new sections under the Advanced Settings of an ESXi 5 host, CBRC and Digest.
There are 4 configurable options to enable CBRC, to configure the cache size, to configure the cache size reservation and interval for digest journal.
I believe the Digest section is related with respect to the algorithms used for CBRC
CBRC looks to only support a maximum of 2GB of memory and default of 400MB reservation. In this example, I have a brand new ESXi 5 host with 8GB configured and without any running virtual machines, here is how much memory it is using.
Now let's go ahead and change the CBRC reserved memory from 400MB to 2048GB (max) and enable CBRC.
If we go back to ESXi summary page, we'll see that an additional 2GB is now being reserved by the ESXi host.
You can also enable and configure CBRC using the CLI, but the ability to enable is only available when using vim-cmd interface in ESXi Shell, the other three options can be configured using legacy esxcfg-advcfg, esxcli or vsish.
Here's an example of enabling CBRC and changing the CBRC memory reservation to 1GB:
vim-cmd hostsvc/advopt/update CBRC.Enable bool true
vim-cmd hostsvc/advopt/update CBRC.DCacheMemReserved long 1024
Ifyou are interested in other advanced settings for CBRC that are not publicly exposed, be sure to check out this post here.
Note: Something I noticed about CBRC is there is an admission control check when you initially enable the feature. If you do not have sufficient memory to pass the admission control check, you will get a very generic error. To see if it is related to admission control by looking in vmkernel.log, you may see a message such as the following: WARNING: cbrc_filter: CBRC_MemSetMemAllocation:1420:Failed to set memory resource parameters for CBRC (Admission check failed for memory resource)
So now that CBRC is enabled, how do we actually use the feature? Well, as I mentioned earlier in the post, this is both a new and hidden feature. Hidden in the sense, that it's not meant to be used directly on an ESXi or vCenter Server but by actually by View Composer. Though you can still access the hidden APIs using the vSphere MOB connected to either vCenter Server or ESXi host.
To get to the new CBRC managed object manager, you can point your browser over to following URL:
https://[hostname]/mob/?moid=ServiceInstance&method=retrieveInternalContent
You will notice a new managed object manager "cbrcManager", go ahead and click on it.
There are 4 methods associated with configuring CBRC:
Note: A shortcut to getting to the CBRC managed object manager is using the following URL:
ESX(i) - https://[hostname]/mob/?moid=ha-cbrc-manager
vCenter - https://[hostname]/mob/?moid=CbrcManager
The ConfigureDigest_Task is what is needed to configure CBRC for a given virtual machine and specifically a virtual disk. The parameters that are needed is both the managed object reference ID of the virtual machine and the virtual disk device ID.
If we browse over to https://[hostname]/mob/?moid=ha-host, we'll be able to identify the virtual machine's MoRef ID and in this example, it's 1
Next we'll need to identify the virtual disk device ID, by traversing the virtual hardware array until we find the virtual disk we're interested in.
The ID for this virtual disk is 3000 and you will also notice a new property called digestEnabled, currently it is disabled.
Now we have everything we need to construct ConfigureDigest_Task method, click on the method name and it will open a new page for you to specify the virtual machine ID and the deviceKey for the virtual disk. You will see the next option is to enable CBRC, I'm not exactly sure what the latter two options but I'll go ahead and enable them anyways.
Once the task has completed, you can view the results and you will see that CBRC has successfully been configured for the virtual machine:
You can also see that the task has been kicked off within the vSphere Client depending if you're connecting to ESXi 5 directly or to vCenter Server:
If we go back to the virtual disk, we will see that digestEnable property is now enabled:
One interesting observation I found was when you enable CBRC, if you browse the datastore of the virtual machine, you will see a new VMDK created with name "digest", this is probably what keeps track of the blocks from the virtual machine and what is loaded into memory.
We can also use the QueryDigestRuntimeStatus to check whether a virtual machine is enabled with CBRC, again specifying the virtual machine's MoRef ID and virtual disk device ID:
We can use QueryDigestInfo on an offline virtual machine to get the details of the digest information for a virtual machine with CBRC enabled:
The last method RecomputeDigest_Task should be self explanatory and it allows you to compute partial or full digest.
By enabling CBRC, the CBRC kernel module is loaded and with that, there are some interesting statistics that can be viewed for a given virtual machine. We'll be leveraging the VMware vsish interface to access the CBRC statistics. To get started, just type vsish and then you will need to change into vmkModules/cbrc_filter path.
One interesting property is the dcacheStats, you can just cat this entry and it provides an enormous amount of statistics about the cache including the number of virtual machines that are using the cache and various IO counters.
You will see that all counters are currently zero, once we start to spin up some Linked Clone virtual machines, you will want to pay close attention to some of these counters.
To answer the question on whether or not CBRC would work outside of VMware View 5, I decided to perform a functional test of the feature. I generated my own VMware Linked Clones using the following vSphere SDK for Perl script vGhettoLinkedClone.pl. You will need a vCenter Server, but you will NOT need VMware View.
Step 1 - Create an offline snapshot for the virtual machine that will be used as the base/golden image to create Linked Clones, in this example, the snapshot will be called "base"
Step 2 - Create several Linked Clones based off of this base/golden virtual machine, in this example, I will be creating three Linked Clone virtual machine named: ALinkedCloneVM1,ALinkedCloneVM2,ALinkedCloneVM3
If we go over to your vCenter Server, you should see the following inventory:
Step 3 - Let's power on the first two virtual machine "ALinkedCloneVM1" and "ALinkedCloneVM2" and let's check out the dcacheStats from vsish and see what has changed.
As you can see we now have counters incrementing on the number of VMs using the cache that was created with CBRC and also counters regarding the backend IO. It looks like the new Linked Clones that was generated can in fact leverage CBRC without VMware View 5. Now this was purely a functional test, these VMs were basically dummy shells, I would be very interested to see if someone is able to get this really working and actually leveraging CBRC with real base images and seeing the reduction of IOPS during a VDI boot storm.
How to Send vCenter Alarm Notification to Growl
This tweet from Jason Nash and @PunchingClouds says it all and here it is!
I did some research this afternoon and stumbled upon this article Nagios: Notifications via Growl and leveraging the Net::Growl Perl module, I was able to forward alarms generated from a vCenter server to a system that was running Growl.
Software Requirements:
- Growl for Windows or Mac OSX installed on a system to receive notifications
- vSphere SDK for Perl installed on vCenter Server
Step 1 - Install Grow and configure a password under the "Security" tab and ensure you "Allow network notification"
Step 2 - Install vSphere SDK for Perl on your vCenter server. You may also need to update the PATH variable with Perl bin directory (e.g. C:\Program Files\VMware\VMware vSphere CLI\Perl\bin)
Step 3 - Install Net::Growl Perl module using ppm (Perl Package Manager) which is part of ActiveState Perl with the installation of vSphere SDK for Perl. This will require your vCenter server have internet access to ActiveState Perl site, if you can not get this access, you can install this locally on your system and extract the Growl.pm and copy it to your vCenter server C:\Program Files\VMware\VMware vSphere CLI\Perl\site\lib\Net
Step 4 - Copy the Perl script from here and store it somewhere on your vCenter server, make sure it has the .pl extension. In this example, I named it growl.pl
Step 5 - To verify that Growl Perl script works and can communicate to the system with Growl install, you can manually test it by running the following command:
growl.pl -H william.primp-industries.com -p vmware -a custom -t Alert -m "hello william" -s 1
You will need to change -H to the hostname or IP Address of the system with Growl installed and of course the password you had setup. You should see a notification of the message you had just sent.
Step 6 - Create a batch script which will call the growl.pl script and store it somewhere on your vCenter server. Here is what the script (sendGrowl.bat) looks like, you can modify it to fit your requirements.
:: http://www.virtuallyghetto.com/ :: Custom vCenter Alarm script to generate growl notifications set GROWL_SERVER=william.primp-industries.com set GROWL_PASSWORD=vmware set GROWL_SCRIPT_PATH="C:\Documents and Settings\primp.PRIMP-IND\Desktop\growl.pl" set PATH="%PATH%;C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\site\bin" %GROWL_SCRIPT_PATH% -H %GROWL_SERVER% -p %GROWL_PASSWORD% -a %COMPUTERNAME% -t Alert -m "%VMWARE_ALARM_EVENTDESCRIPTION%" -s 1
Note: If you would like to get a list of other default VMware alarm variables, run the "SET" command and output it to a file to get more details on various variables that can be accessed.
Step 7 - Create a new or update an existing vCenter alarm and under "Actions", specify "Run a command" option and provide the full path to the sendGrowl.bat
Step 8 - For testing purposes, I created a new alarm that would trigger upon an ESX(i) host going in/out of maintenance mode and you can see from the "Tasks and Events", our script is triggered on the vCenter server
and now for the finale, you should see a notification from Growl on your system and since we enable the "sticky" parameter, the notification will stay on your screen until you click on it. You can see that in the script example, I set the message to the event and application is registered as the name of the vCenter server, which allows you to have multiple vCenter forward you notifications.
So there you have it, forwarding vCenter alarms to Growl.
Note: Once a vCenter alarm has been triggered, the script will not fire off again until the original alarm has been reset to green. This behavior probably is okay for majority of the events one would want to monitor, but if you want it to continuously alert you, you will need to fiddle with a way to reset the alarm on the vCenter server.
UPDATE: Thanks to Richard Cardona for reminding me, but this can also be implemented on the new VCVA (vCenter Server Virtual Appliance) in vSphere 5. Here are the instructions on setting it up
Step 1 - Install Grow and configure a password under the "Security" tab and ensure you "Allow network notification" on the system that is receiving the Growl notifications
Step 2 - To install Net::Growl, we'll be using cpan which requires 2 modules that are not installed by default on the SLES VCVA. Using the Tips and Tricks for vMA 5 (running SLES as well), we'll go ahead and setup zypper package manager for VCVA to install the two required packages: make and yaml
zypper --gpg-auto-import-keys ar http://download.opensuse.org/distribution/11.1/repo/oss/ 11.1
zypper --gpg-auto-import-keys ar http://download.opensuse.org/update/11.1/ Update-11.1
zypper refresh
zypper in make
zypper in perl-YAML
Step 3 - You will use cpan to install Net::Growl
perl -MCPAN -e shell
Step 4 - Once you are inside the cpan shell, type the following to install Net::Growl
install Net::Growl
Step 5 - Copy the Perl script from here and store it somewhere on your vCenter server (e.g. /root), make sure it has the .pl extension and has execute permission. In this example, I named it growl.pl
Step 6 - To verify that Growl Perl script works and can communicate to the system with Growl install, you can manually test it by running the following command:
vcenter50-2:~ # ./growl.pl -H william.primp-industries.com -p vmware -a custom -t Alert -m "hello william" -s 1
Step 7 - Create a shell script which will call the growl.pl script and store it somewhere on your vCenter server (e.g. /root). Here is what the script (sendGrowl.sh) looks like, you can modify it to fit your requirements.
Step 8 - Create a new or update an existing vCenter alarm and under "Actions", specify "Run a command" option and provide the full path to the sendGrowl.sh
- « Previous Page
- 1
- …
- 45
- 46
- 47
- 48
- 49
- …
- 69
- Next Page »