WilliamLam.com

Creating vCenter Alarms based on Task Events such as Folder creation

by // 13 Comments

The vCenter Server Events sub-system is an incredibly rich and powerful interface that enables customers to monitor, alert and even trigger additional actions based on a particular event. One such example that I have written about before is to key off of a VM provisioned event and automatically apply security hardening settings when the VM is created or cloned. This can be useful if customers are not taking advantage of VM Templates or if a VI Admins manually creates a VM from scratch, you can still ensure you have a compliant VM deployment through the use of Automation. You can either poll for the VM created event and then execute a script as shown in this example or you can automatically trigger a remote action by generating an SNMP trap when the event actually occurs.

The possibilities are truly endless on what you can do with vCenter Events and for the complete list of all Event types, you can refer to the vSphere API documentation here. One thing to be aware of is that not every operation within vCenter Server generates an Event, one example of this is when a Folder object is created or deleted. You can use vCenter Server Tasks sub-system to query for this info but there is not a respective vCenter Event that you can key off of to generate an Alarm for example. This was something I had noticed myself and assumed it was a limitation of the platform or feature teams that publish VC Events.

Recently, this question came up again from a customer who was looking for a way to trigger an alarm every time a VM Folder was created. I took another look at this and came to learn about a more generic type of Event that can be used to create an Alarm for such use cases where a native VC Event may not exists called a Task Event.

[Read more...]

Create vCenter Alarms for monitoring HCX migration events

by // 2 Comments

With the Hybrid Cloud Extension (HCX) solution, customers can migrate workloads immediately or have them scheduled at a later date and time of their choosing. Most customers will most likely not migrate all their workloads at once, but instead migrate them in different phases or waves based on their own internal schedules. The HCX "Migration" tab in the vSphere Client is a great place to get an overview of all the active, failed, planned and completed migrations and you can get additional details for each migration such status, percentage, amount of data transferred, etc.

Having said that, you probably also do not want to just sit there and constantly watch for progress, especially if something fails. The good news is that HCX already includes some default vCenter Alarms that are generated when a migration fails. There is one for vMotion/Cloud Motion, Cold Migration as well as Bulk Migration and currently these alarms only trigger a UI indication that something is wrong using the red critical icon on the VM in question.

Since these are vCenter Alarms, you also have the option of adding additional actions such as sending an email alert to a particular user or group, an SNMP trap which can then be monitored by your operations team or even run a command within the vCenter Server. Simply updating the default alarm which is defined at the vCenter Server level, you can proactively get notified when an HCX migration fails without having to constantly watch the UI. You may have also noticed the alarm definition is using a numeric ID which is highlighted in the screenshot above. This maybe useful if you wish to leave the default alarms alone and create a brand new alarm, you just need to use those IDs.

Another useful scenario to consider is getting notifications for a successful migration which can then help with transitioning users to the new VM running on VMC or even trigger automated VM and application verification before end users are even notified. Along these similar lines, you can imagine another scenario to consider is with using Bulk Migration and/or Cloud Motion with vSphere Replication. Both methods initially start off by replicating the VM's data from the source vCenter Server to the destination vCenter Server and then either a Cold Migration or vMotion is performed based on the configured scheduled.

[Read more...]

How to run a script from a vCenter Alarm action in the VCSA?

by // 9 Comments

As more and more customers transition from the vCenter Server for Windows to the vCenter Server Appliance (VCSA), a question that I get from time to time is how do I run a script using a vCenter Alarm from the VCSA? Traditionally, customers would create and run Windows specific scripts such as a simple batch script or a more powerful Powershell/PowerCLI script. However, since the VCSA is a Linux based operating system, these Windows specific scripts would no longer function, at least out of the box.

There are a couple of options:

  1. If you have quite a few scripts and you prefer not to re-write them to run on the VCSA, then you can instead have them executed on a remote system.
  2. Re-write or create new scripts that would run in the VCSA. You can leverage pyvmomi (vSphere SDK for Python) which is already included in the VCSA. This will allow you to access the vSphere API just like you would with PowerCLI on a vCenter Server for Windows if you had that installed.

Given there are already multiple ways of achieving #1 and I am sure there are other options if you just did a quick Google search, I am going to focus on #2 which will demonstrate how to create a pyvmomi script that can be triggered by a vCenter Alarm action running on the VCSA.

Disclaimer: It is generally recommended that you do not install or run additional tools or scripts on the vCenter Server (Windows or VCSA) itself as it may potentially impact the system. If it is possible, a remote system using programmatic APIs to provide centralize management and configuration is preferred and it also limits the number of users that may have access directly to the vCenter Server.

The following example is based off of a real use case from one of our customers. The requirement was to automatically generate and collect an ESXi performance bundle (part of the support bundle) for all hosts in a given vSphere Cluster when a specific vCenter Alarm is triggered. In this example, we will create a simple vCenter Alarm on our vSphere Cluster that triggers our script when a particular VM is modified as this is probably one of the easiest way to test the alarm itself.

UPDATE (06/14/16) - For generating an ESXI performance support bundle, please have a look at this article which can then be applied to this article in terms of running the script in the VCSA.

When a vCenter Alarm is triggered, there are a bunch of metadata that is passed from the vCenter Alarm itself and stored in a set of VMware specific environmental variables which you can find some more details in the documentation here. These environmental variables can then be accessed and used directly from within the script itself. Here is an example of what the environmental variables looks like for my vCenter Alarm which triggers off of the VM reconfigured event:

VMWARE_ALARM_EVENT_DVS=
VMWARE_ALARM_TRIGGERINGSUMMARY=Event: VM reconfigured (224045)
VMWARE_ALARM_OLDSTATUS=Gray
VMWARE_ALARM_ID=alarm-302
VMWARE_ALARM_EVENT_DATACENTER=NUC-Datacenter
VMWARE_ALARM_EVENT_NETWORK=
VMWARE_ALARM_DECLARINGSUMMARY=([Event alarm expression: VM reconfigured; Status = Red])
VMWARE_ALARM_EVENT_DATASTORE=
VMWARE_ALARM_TARGET_ID=vm-606
VMWARE_ALARM_NAME=00 - Collect all ESXi Host Logs
VMWARE_ALARM_EVENT_USERNAME=VGHETTO.LOCAL\Administrator
VMWARE_ALARM_EVENTDESCRIPTION=Reconfigured DummyVM on 192.168.1.190 in NUC-Datacenter
VMWARE_ALARM_TARGET_NAME=DummyVM
VMWARE_ALARM_EVENT_COMPUTERESOURCE=Non-VSAN-Cluster
VMWARE_ALARM_EVENT_HOST=192.168.1.190
VMWARE_ALARM_NEWSTATUS=Red
VMWARE_ALARM_EVENT_VM=DummyVM
VMWARE_ALARM_ALARMVALUE=Event details

The variable that we care about in this particular example is the VMWARE_ALARM_EVENT_COMPUTERESOURCE as this provides us with the name of the vSphere Cluster that we will need to iterate all the ESXi hosts to then generate the ESXi support bundles. I have created the following pyvmomi script called generate_esxi_support_bundle_from_vsphere_cluster.py which you will need to download and upload to your VCSA. The script has been modified so that the credentials are stored in (line 44 and 50) within the script versus prompting for it on the command-line (you can easily tweak it for testing purposes). By default, the script will store the logs under /storage/log but if you wish to place it else where, you can modify line 55.

Note: If you wish to manually test this script before implementing the vCenter Alarm, you will probably want to comment line 105 and manually specify the vSphere Cluster in line 106.

Below are the instructions on configuring the vCenter Alarm and the python script to run:

Step 1 - Upload the generate_esxi_support_bundle_from_vsphere_cluster.py script to your VCSA (I stored it in /root) and set it to an executable by running the following command:

chmod +x generate_esxi_support_bundle_from_vsphere_cluster.py

Step 2 - Create a new vCenter Alarm under a vSphere Cluster which has some ESXi hosts attached. The trigger event will be VM reconfigured and the alarm action will be the path to the script such as /root/generate_esxi_support_bundle_from_vsphere_cluster.py as shown in the screenshot below.

run-script-from-vcenter-alarm-vcsa-0
Step 3 - Go ahead and modify the VM to trigger the alarm. I found the easiest way with the least amount of clicks is to just update the VM's annotation 😉 If everything was configured successfully, you should see the vCenter Alarm activate as well as a task to start the log bundle collection. Since the script is being call non-interactively, if you want to see the progress of the script itself, you can login to the VCSA and all details from the script is logged to its own log file in /var/log/vcenter_alarms.log.

Here is a snippet of what you would see

2016-06-05 17:37:54;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-05 17:37:54;INFO;Generating support bundle
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.190:443/downloads/vmsupport-527c584f-f4cf-cd6d-fc3e-963a7a375bc9.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.191:443/downloads/vmsupport-521774a8-4fba-7ce4-42a5-e2f0a244f237.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

Hopefully this gives you an idea of how you might run custom scripts triggered from a vCenter Alarm from within the VCSA. With a bit of vSphere API knowledge and the fact that pyvmomi is installed on the VCSA by default, you can create some pretty powerful workflows triggered from various events using a vCenter Alarm in the VCSA!