Search Results for: vsphere events

How to tell if your vCenter Server Appliance (VCSA) was migrated from a Windows vCenter Server?

09.19.2016 by William Lam // 2 Comments

In case you had not heard, last week VMware had officially released the VCSA Migration Tool which is included in the new vSphere 6.0 Update 2m release. Customers can now easily migrate from a Windows based vCenter Server over to the vCenter Server Appliance (VCSA) all while preserving their existing vCenter Server configurations and integrations. For more details, I highly recommend you check out all the links and resources here related to the VCSA Migration Tool.

One interesting question that came up over the weekend from a troubleshooting standpoint was how do you tell if your VCSA was migrated from a Windows vCenter Server? Besides remembering 😉 there is actually a pretty simple way to check by looking at the install parameters as I have previously written about here. To do so, you will need to SSH to your VCSA and enable the Bash Shell first. Once that has been done, go ahead and run the following command:

install-parameter upgrade.source.platform

If your VCSA was migrated from a Windows based vCenter Server using the new VCSA Migration Tool, you should see a value of windows. If you do not get any results, then it means the VCSA was not migrated and it was freshly deployed as an appliance.

In addition, you can also check whether or not you had migrated over the original vCenter Server's Stats, Events and Tasks (SET) data. To do so, run the following command:

install-parameter upgrade.user.options

You should get back a value of either yes or no for migrating over the SET data.

Lastly, if your VCSA was migrated from a Windows based vCenter Server, you can even tell if the migration was done so using the UI or CLI. To do so, run the following command:

install-parameter upgrade.silent

You should get back a value of either True for a CLI-based migration or False for a UI-based migration.

Here is a quick screenshot of running the three commands on a VCSA that was migrated.

How to easily disable vMotion & Cross vCenter vMotion for a particular Virtual Machine?

07.20.2016 by William Lam // 6 Comments

The question of disabling vMotion for a specific set of Virtual Machine(s) is not a new one. In fact, this topic comes up on some what of a frequent basis and usually driven by arcane change management processes or worse licensing restrictions. Do not get me wrong, there are definitely some valid use cases where you would not want a particular VM to be migrated off. The classic example is a 3rd Party VM solution that provides Anti-Malware, Intrusion Detection & Firewall capabilities for your workload VMs. For this particular use case, VMware provides our partners with an integration hook into the vSphere platform called ESX Agent Manager (EAM) that ensures these "Service VMs" are not allowed to be powered off or migrated to another ESXi host, even in the case of a Maintenance Mode operation. This solutuion even allows you to configure custom icons for your Service VMs!

For all other use cases outside of the "Service VMs", there really is not an easy way of disabling vMotion for a particular VM. There have been many solutions that have been suggested in the past ranging from disabling DRS for a specific VM, DRS Affinity Rules, VM miss-configurations to break vMotion compatibility to using vSphere Permissions to prevent vMotion operations. However, many of these solutions do not work very well or is very difficult to manage at scale. I actually like the idea of using vSphere Permissions to prevent a vMotion, however, I have seen some customers push back on this because the vSphere Administrator still has the ability to perform this operation. For these cases, customers just want to be able to completely disable vMotion for a given VM and prevent anyone from migrating the VM, including the vSphere Administrators.

Given that this topic had recently come up again, I was wondering if there was an easier way in which this could be achieved and made more manageable for our customers. After thinking about about how EAM handles "disabling" certain operations for a VM and recalling an article I wrote last year which leveraged this exact capability to resolve an NSX Controller issue, I thought why not apply it to this use case here?

UPDATE (09/27/18) - As of vSphere 6.5, the MigrateVM_Task() method has been deprecated in favor of the RelocateVM_Task() which is used to handle BOTH vMotion, Storage vMotion and other variants. This means that as of 6.5, you no longer have the ability to disable a specific migration type and when you apply the settings, both migration types will be disabled for the specific VM.

Disclaimer: The use of internal APIs are not officially supported by VMware and can change at any time. Please use at your own risk.

Each VM has a property called DisableMethod which lists the specific vSphere API methods that are currently disabled. These are not governed by vSphere Permissions but rather the runtime state of the VM. For example, if you have a VM that is currently powered on, then the PowerOnVM_Task API would not be available and would show up in the disabled list.

Here is a quick PowerCLI snippet on how to retrieve the current set of disable methods for a VM:

$vm = Get-Vm -Name TestVM-1
$vm.ExtensionData.DisabledMethod

As mentioned in this article, the ability to enable and disable these methods are only available as an internal vCenter Server API. However, it is possible to access these APIs using the vSphere MOB, but it is not very user friendly nor intuitive. Below is a screenshot of invoking the disableMethods API using the vSphere MOB.

A couple of weeks back I started to investigate on how we might be able to automate against the vSphere MOB. The result of that investigation lead to the creation of a simple PowerCLI script that allows you to automate operations using the vSphere MOB which I had published here. That work became the foundation for the new PowerCLI script that I had created for disabling and enabling the vMotion capability for a particular VM.

You can download the PowerCLI script here called enable-disable-vsphere-api-method.ps1 which includes two functions Enable-vSphereMethod and Disable-vSphereMethod. You will need to edit the script to provide a couple of pieces of information.

Credentials to your vCenter Server
Name of the VM you wish to either disable or enable vMotion capability on
Name of the vSphere API method you wish to disable (by default this is MigrateVM_Task which maps to the vMotion capability)

By default, I have commented out both functions usage, you will need to manually uncomment one of the lines based on the operation you wish to perform.

To Disable the vMotion capability, run the following:

Disable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -disable_method $method_name

To Enable the vMotion capability if you had disabled it, run the following:

Enable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -enable_method $method_name

After the script has completed, you can now re-run the command that we ran earlier to see which methods have been disabled and you should see that the MigrateVM_Task is now part of the disable methods.

If we now login to either the vSphere Web/C# Client and right click on the VM that we had disabled vMotion on, you should also see that the Migrate option is now grayed out and unavailable. This behavior will be true for ALL users including those in the vSphere Administrators group.

It is important to note that vMotion is not only disabled from the UI, but it is also disabled from the vSphere API standpoint which the UI is built on top of. Here is an example of trying to perform a vMotion using the PowerCLI Move-VM cmdlet and you can see that an error is thrown immediately stating that the method has been disabled.

Note: The "self" text output from the PowerCLI command is actually something that you can specify as part of disabling the vMotion capability. This might be useful to specify a change control ID or some string to signal to the user who might be trying to perform the operation. Please refer to the script and search for the "self" keyword if you wish to change it.

The really nice thing about this solution is not only is it really easy to enable or disable, but it can also be managed at scale which many of the other solutions mentioned earlier start to break down. The last thing anyone would want is additional operational overhead to manage manage complex DRS rules (which can still be overridden through manual migrations) or additional vSphere Permissions which also runs into the same problem where a vSphere Administrator can still override by performing a manual migration. This solution does prevent both standard vMotion as well as the new Cross vCenter vMotion capability (both between same/different SSO Domain) that was introduced in vSphere 6.0. You do not need to be running vSphere 6.0 to be able to leverage this solution, this should actually work for almost all versions of vSphere. Lastly, enabling or disabling the functionality does not require any type of system restart or impact to your VM other than the ability to vMotion.

Limitations

Beyond artificially limiting what vSphere DRS and HA can do, I did observe an interesting behavior when a Maintenance Mode operation is performed. If you leave the "Move powered-off and suspended virtual machines to other hosts in the cluster" uncheck, then all VMs will be migrated off and the VMs that have vMotion disabled will reside on the ESXi host while it goes into Maintenance Mode. However, if you do check the box, I did find that the system would override setting and actually move the VM to another ESXi host. This is something to be aware of and may not be a bad thing depending on your requirements.

Disabling Storage vMotion

You may have noticed that if the VM is in a powered off state, that the Migrate option is still available in the UI. The reason for this is that we only disabled vMotion but you are still allowed to perform a Storage vMotion. If you wish to also disable the Storage vMotion capability, then you will need to disable RelocateVM_Task vSphere API method as well.

Auditing vMotion and Storage vMotion Operations

With or without this solution, you may still want another level of confidence that a VM has either not migrated or migrated to authorized set of ESXi hosts. We can easily do so by auditing the VM's Event system and looking for migration events. The name of the vMotion event is called VmMigratedEvent and the name of the Storage vMotion event is called VmRelocatedEvent. Here is a sample script using the vSphere SDK for Perl that exercises this specific vSphere API and provides you with all the ESXi hosts a given VM might have migrated to. For those that rather consume the vSphere API using something like PowerCLI, here is a quick one-liner to extract vMotion events:

$vm = Get-VM TestVM-2
Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmMigratedEvent"} | Select CreatedTime, UserName, FullFormattedMessage | ft -wrap -AutoSize

How to run a script from a vCenter Alarm action in the VCSA?

06.07.2016 by William Lam // 9 Comments

As more and more customers transition from the vCenter Server for Windows to the vCenter Server Appliance (VCSA), a question that I get from time to time is how do I run a script using a vCenter Alarm from the VCSA? Traditionally, customers would create and run Windows specific scripts such as a simple batch script or a more powerful Powershell/PowerCLI script. However, since the VCSA is a Linux based operating system, these Windows specific scripts would no longer function, at least out of the box.

There are a couple of options:

If you have quite a few scripts and you prefer not to re-write them to run on the VCSA, then you can instead have them executed on a remote system.
- Instead of running the Windows scripts on the VCSA itself, a simplified script could be created to remotely call into another system which would then run the actual scripts. An example of this could be creating a Python script and using WMI to remotely run a script on a Windows system.
- Similar to previous option, instead of a local script you could send an SNMP trap to a remote system would then run the actual Windows scripts. Here is an example of using vCenter Orchestrator which has a PowerCLI plugin that can be triggered by receiving an SNMP trap.
Re-write or create new scripts that would run in the VCSA. You can leverage pyvmomi (vSphere SDK for Python) which is already included in the VCSA. This will allow you to access the vSphere API just like you would with PowerCLI on a vCenter Server for Windows if you had that installed.

Given there are already multiple ways of achieving #1 and I am sure there are other options if you just did a quick Google search, I am going to focus on #2 which will demonstrate how to create a pyvmomi script that can be triggered by a vCenter Alarm action running on the VCSA.

Disclaimer: It is generally recommended that you do not install or run additional tools or scripts on the vCenter Server (Windows or VCSA) itself as it may potentially impact the system. If it is possible, a remote system using programmatic APIs to provide centralize management and configuration is preferred and it also limits the number of users that may have access directly to the vCenter Server.

The following example is based off of a real use case from one of our customers. The requirement was to automatically generate and collect an ESXi performance bundle (part of the support bundle) for all hosts in a given vSphere Cluster when a specific vCenter Alarm is triggered. In this example, we will create a simple vCenter Alarm on our vSphere Cluster that triggers our script when a particular VM is modified as this is probably one of the easiest way to test the alarm itself.

UPDATE (06/14/16) - For generating an ESXI performance support bundle, please have a look at this article which can then be applied to this article in terms of running the script in the VCSA.

When a vCenter Alarm is triggered, there are a bunch of metadata that is passed from the vCenter Alarm itself and stored in a set of VMware specific environmental variables which you can find some more details in the documentation here. These environmental variables can then be accessed and used directly from within the script itself. Here is an example of what the environmental variables looks like for my vCenter Alarm which triggers off of the VM reconfigured event:

VMWARE_ALARM_EVENT_DVS=
VMWARE_ALARM_TRIGGERINGSUMMARY=Event: VM reconfigured (224045)
VMWARE_ALARM_OLDSTATUS=Gray
VMWARE_ALARM_ID=alarm-302
VMWARE_ALARM_EVENT_DATACENTER=NUC-Datacenter
VMWARE_ALARM_EVENT_NETWORK=
VMWARE_ALARM_DECLARINGSUMMARY=([Event alarm expression: VM reconfigured; Status = Red])
VMWARE_ALARM_EVENT_DATASTORE=
VMWARE_ALARM_TARGET_ID=vm-606
VMWARE_ALARM_NAME=00 - Collect all ESXi Host Logs
VMWARE_ALARM_EVENT_USERNAME=VGHETTO.LOCAL\Administrator
VMWARE_ALARM_EVENTDESCRIPTION=Reconfigured DummyVM on 192.168.1.190 in NUC-Datacenter
VMWARE_ALARM_TARGET_NAME=DummyVM
VMWARE_ALARM_EVENT_COMPUTERESOURCE=Non-VSAN-Cluster
VMWARE_ALARM_EVENT_HOST=192.168.1.190
VMWARE_ALARM_NEWSTATUS=Red
VMWARE_ALARM_EVENT_VM=DummyVM
VMWARE_ALARM_ALARMVALUE=Event details

The variable that we care about in this particular example is the VMWARE_ALARM_EVENT_COMPUTERESOURCE as this provides us with the name of the vSphere Cluster that we will need to iterate all the ESXi hosts to then generate the ESXi support bundles. I have created the following pyvmomi script called generate_esxi_support_bundle_from_vsphere_cluster.py which you will need to download and upload to your VCSA. The script has been modified so that the credentials are stored in (line 44 and 50) within the script versus prompting for it on the command-line (you can easily tweak it for testing purposes). By default, the script will store the logs under /storage/log but if you wish to place it else where, you can modify line 55.

Note: If you wish to manually test this script before implementing the vCenter Alarm, you will probably want to comment line 105 and manually specify the vSphere Cluster in line 106.

Below are the instructions on configuring the vCenter Alarm and the python script to run:

Step 1 - Upload the generate_esxi_support_bundle_from_vsphere_cluster.py script to your VCSA (I stored it in /root) and set it to an executable by running the following command:

chmod +x generate_esxi_support_bundle_from_vsphere_cluster.py

Step 2 - Create a new vCenter Alarm under a vSphere Cluster which has some ESXi hosts attached. The trigger event will be VM reconfigured and the alarm action will be the path to the script such as /root/generate_esxi_support_bundle_from_vsphere_cluster.py as shown in the screenshot below.

Step 3 - Go ahead and modify the VM to trigger the alarm. I found the easiest way with the least amount of clicks is to just update the VM's annotation 😉 If everything was configured successfully, you should see the vCenter Alarm activate as well as a task to start the log bundle collection. Since the script is being call non-interactively, if you want to see the progress of the script itself, you can login to the VCSA and all details from the script is logged to its own log file in /var/log/vcenter_alarms.log.

Here is a snippet of what you would see

2016-06-05 17:37:54;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-05 17:37:54;INFO;Generating support bundle
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.190:443/downloads/vmsupport-527c584f-f4cf-cd6d-fc3e-963a7a375bc9.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.191:443/downloads/vmsupport-521774a8-4fba-7ce4-42a5-e2f0a244f237.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

Hopefully this gives you an idea of how you might run custom scripts triggered from a vCenter Alarm from within the VCSA. With a bit of vSphere API knowledge and the fact that pyvmomi is installed on the VCSA by default, you can create some pretty powerful workflows triggered from various events using a vCenter Alarm in the VCSA!