As more and more customers transition from the vCenter Server for Windows to the vCenter Server Appliance (VCSA), a question that I get from time to time is how do I run a script using a vCenter Alarm from the VCSA? Traditionally, customers would create and run Windows specific scripts such as a simple batch script or a more powerful Powershell/PowerCLI script. However, since the VCSA is a Linux based operating system, these Windows specific scripts would no longer function, at least out of the box.
There are a couple of options:
- If you have quite a few scripts and you prefer not to re-write them to run on the VCSA, then you can instead have them executed on a remote system.
- Instead of running the Windows scripts on the VCSA itself, a simplified script could be created to remotely call into another system which would then run the actual scripts. An example of this could be creating a Python script and using WMI to remotely run a script on a Windows system.
- Similar to previous option, instead of a local script you could send an SNMP trap to a remote system would then run the actual Windows scripts. Here is an example of using vCenter Orchestrator which has a PowerCLI plugin that can be triggered by receiving an SNMP trap.
- Re-write or create new scripts that would run in the VCSA. You can leverage pyvmomi (vSphere SDK for Python) which is already included in the VCSA. This will allow you to access the vSphere API just like you would with PowerCLI on a vCenter Server for Windows if you had that installed.
Given there are already multiple ways of achieving #1 and I am sure there are other options if you just did a quick Google search, I am going to focus on #2 which will demonstrate how to create a pyvmomi script that can be triggered by a vCenter Alarm action running on the VCSA.
Disclaimer: It is generally recommended that you do not install or run additional tools or scripts on the vCenter Server (Windows or VCSA) itself as it may potentially impact the system. If it is possible, a remote system using programmatic APIs to provide centralize management and configuration is preferred and it also limits the number of users that may have access directly to the vCenter Server.
The following example is based off of a real use case from one of our customers. The requirement was to automatically generate and collect an ESXi performance bundle (part of the support bundle) for all hosts in a given vSphere Cluster when a specific vCenter Alarm is triggered. In this example, we will create a simple vCenter Alarm on our vSphere Cluster that triggers our script when a particular VM is modified as this is probably one of the easiest way to test the alarm itself.
UPDATE (06/14/16) - For generating an ESXI performance support bundle, please have a look at this article which can then be applied to this article in terms of running the script in the VCSA.
When a vCenter Alarm is triggered, there are a bunch of metadata that is passed from the vCenter Alarm itself and stored in a set of VMware specific environmental variables which you can find some more details in the documentation here. These environmental variables can then be accessed and used directly from within the script itself. Here is an example of what the environmental variables looks like for my vCenter Alarm which triggers off of the VM reconfigured event:
VMWARE_ALARM_EVENT_DVS= VMWARE_ALARM_TRIGGERINGSUMMARY=Event: VM reconfigured (224045) VMWARE_ALARM_OLDSTATUS=Gray VMWARE_ALARM_ID=alarm-302 VMWARE_ALARM_EVENT_DATACENTER=NUC-Datacenter VMWARE_ALARM_EVENT_NETWORK= VMWARE_ALARM_DECLARINGSUMMARY=([Event alarm expression: VM reconfigured; Status = Red]) VMWARE_ALARM_EVENT_DATASTORE= VMWARE_ALARM_TARGET_ID=vm-606 VMWARE_ALARM_NAME=00 - Collect all ESXi Host Logs VMWARE_ALARM_EVENT_USERNAME=VGHETTO.LOCAL\Administrator VMWARE_ALARM_EVENTDESCRIPTION=Reconfigured DummyVM on 192.168.1.190 in NUC-Datacenter VMWARE_ALARM_TARGET_NAME=DummyVM VMWARE_ALARM_EVENT_COMPUTERESOURCE=Non-VSAN-Cluster VMWARE_ALARM_EVENT_HOST=192.168.1.190 VMWARE_ALARM_NEWSTATUS=Red VMWARE_ALARM_EVENT_VM=DummyVM VMWARE_ALARM_ALARMVALUE=Event details
The variable that we care about in this particular example is the VMWARE_ALARM_EVENT_COMPUTERESOURCE as this provides us with the name of the vSphere Cluster that we will need to iterate all the ESXi hosts to then generate the ESXi support bundles. I have created the following pyvmomi script called generate_esxi_support_bundle_from_vsphere_cluster.py which you will need to download and upload to your VCSA. The script has been modified so that the credentials are stored in (line 44 and 50) within the script versus prompting for it on the command-line (you can easily tweak it for testing purposes). By default, the script will store the logs under /storage/log but if you wish to place it else where, you can modify line 55.
Note: If you wish to manually test this script before implementing the vCenter Alarm, you will probably want to comment line 105 and manually specify the vSphere Cluster in line 106.
Below are the instructions on configuring the vCenter Alarm and the python script to run:
Step 1 - Upload the generate_esxi_support_bundle_from_vsphere_cluster.py script to your VCSA (I stored it in /root) and set it to an executable by running the following command:
chmod +x generate_esxi_support_bundle_from_vsphere_cluster.py
Step 2 - Create a new vCenter Alarm under a vSphere Cluster which has some ESXi hosts attached. The trigger event will be VM reconfigured and the alarm action will be the path to the script such as /root/generate_esxi_support_bundle_from_vsphere_cluster.py as shown in the screenshot below.
Step 3 - Go ahead and modify the VM to trigger the alarm. I found the easiest way with the least amount of clicks is to just update the VM's annotation 😉 If everything was configured successfully, you should see the vCenter Alarm activate as well as a task to start the log bundle collection. Since the script is being call non-interactively, if you want to see the progress of the script itself, you can login to the VCSA and all details from the script is logged to its own log file in /var/log/vcenter_alarms.log.
Here is a snippet of what you would see
2016-06-05 17:37:54;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-05 17:37:54;INFO;Generating support bundle
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.190:443/downloads/vmsupport-527c584f-f4cf-cd6d-fc3e-963a7a375bc9.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.191:443/downloads/vmsupport-521774a8-4fba-7ce4-42a5-e2f0a244f237.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz
Hopefully this gives you an idea of how you might run custom scripts triggered from a vCenter Alarm from within the VCSA. With a bit of vSphere API knowledge and the fact that pyvmomi is installed on the VCSA by default, you can create some pretty powerful workflows triggered from various events using a vCenter Alarm in the VCSA!
How did you generate the list of environment variables showing the VMWARE_ALARMS_ variables? I'm using a vCenter 6.0 server on Server 2012 R2 and if i run CMD>>SET or Powershell>Get-ChildItem Env: i do not see any VMWARE_ALARM_ variables anywhere, even when i generate an alarm within vCenter. I do see other VMware variables like VMWARE_CFG_DIR but i dont see any _ALARM_ variables at all.
I am attempting to add an attribute value to a VM when it is created. I have found a python script that is supposed to add the values of creator and time created to the attributes of the VM and it completes without error, but does not populate the attribute value. Can you tell me where this is going wrong?
Here is the script:
#!/usr/bin/python
##
## To be ran on the VCSA and called via alarm rule
##
import sys
from getpass import getpass
from datetime import datetime
import ssl
import os
sys.path.extend(os.environ['VMWARE_PYTHON_PATH'].split(';'))
from pyVim import connect
from pyVim.connect import SmartConnect
from pyVmomi import vim
alarm_name = os.getenv('VMWARE_ALARM_NAME', 'debug_VMWARE_ALARM_NAME')
alarm_target_name = os.getenv('VMWARE_ALARM_TARGET_NAME', 'debug_VMWARE_ALARM_TARGET_NAME')
event_decscription = os.getenv('VMWARE_ALARM_EVENTDESCRIPTION', 'debug_VMWARE_ALARM_EVENTDESCRIPTION')
alarm_value = os.getenv('VMWARE_ALARM_ALARMVALUE', 'debug_VMWARE_ALARM_EVENTDESCRIPTION')
alarm_vm = os.getenv('VMWARE_ALARM_EVENT_VM', 'debug_VMWARE_ALARM_EVENT_VM')
alarm_user = os.getenv('VMWARE_ALARM_EVENT_USERNAME', 'debug_VMWARE_ALARM_EVENT_USERNAME')
if alarm_vm != 'debug_VMWARE_ALARM_EVENT_VM':
s=ssl.SSLContext(ssl.PROTOCOL_SSLv23) # For VC 6.5/6.0 s=ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode=ssl.CERT_NONE
# The pwd variable will need to be to a real password.
# Perhaps using a lookup from a vault.
# This is not built into this script
# Of course "vcenter" and "user" will need to be updated as well
si = SmartConnect(host="vcenter", user="user", pwd="password", sslContext=s)
content=si.content
def find_vm_obj(content, vimtype, name):
obj = {}
container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
for c in container.view:
if name:
if c.name == name:
obj = c
break
else:
obj = c
break
return obj
vm = find_vm_obj(content, [vim.VirtualMachine], alarm_vm)
## These attributes must exist in vcenter before writing
## Feel free to add, remove or change the ones below
if vm:
vm.setCustomValue('vm.owner', alarm_user)
vm.setCustomValue('vm.provisioned', str(datetime.now().strftime('%Y-%m-%d %H:%M:%S')))
Nevermind, it looks to be working great!
This article needs some serious update. In vSphere 8, it unfortunately does not work like that anymore...
First issue is, the VCSA log file to look for problems is no longer /var/log/vcenter_alarms.log, but /var/log/vmware/vpxd/vpxd.log - along with tons of unrelated messages, so no dedicated alarms log anymore.
Next problem is, alarm scripts are executed now under the vpxd user context, and it does not have permissions to run the script. There's a KB article about it:
vCenter Server 8.0 Alarm actions which run scripts fail with “Command must exist/be executable”
https://kb.vmware.com/s/article/87918
Despite applying the chown vpxd command, I can't get it working. Unfortuntely the KB article is only talking about a shell script and not about a Python script, but I tried everything (calling the script with /usr/bin/python and without, having both variants allowed in sudoers, calling the python script via shell script, etc.), but nothing works...
I know there's VEBA as an alternative, but setting it up has also been just a desaster so far... the services inside the appliance are simply not running.
"needs some serious update" - I'm sorry the article that I wrote many years ago isn't magically forwards compatible with some of the new changes in the product. Your response/tone also comes across an expectation that as the author, I have made some guarantee that the content will always reflect the latest versions of the product? If that is your expectation, you may want to look else where or provide your feedback to the official documentation and consider how you actually provide feedback to get better outcome.
Regarding VEBA, that is definitely the recommended approach as running scripts on VCSA is not only not supported but you're limited in what you can do and can even take down your VC with poorly written scripts. I suspect your deployment of VEBA was most likely done incorrectly, either bad/incorrect user input as we've got many customers who've successfully deployed without any issues. Not sure if you're the person who commented on the Fling site, but I suggest you take a look at all deployment options and ensure you're providing the correct input based on your environment