vSphere – Page 73

How to automatically log all VM configuration changes using a vCenter Server Alarm?

08.18.2015 by William Lam // 9 Comments

If you followed my previous blog post on How to audit VM reconfigurations and see what exactly changed, you may have concluded that historical VM configuration events may potentially be unavailable by the time you need to perform an audit. The reason for this is that the VM Events tables may have rotated out depending on the retention policy of your vCenter Server Database. This is where we can take advantage of the powerful vCenter Server Alarm feature which would allows us to capture every single VM reconfiguration and store this information outside of the vCenter Server. This not only allows you to reduce the amount of data stored in the vCenter Server Database but it also allows you to efficiently archive this data into a data warehouse or Big Data platform that provides more advanced analytics and reporting capabilities.

Building on top of our previous script, I have created a slightly modified version called Get-VMConfigChangesFromAlarm.ps1. The main difference is instead of passing in a VM object to look for past configuration changes, the script can now pull in information from a triggered VmReconfigureEvent and log the specific changes associated with that VM reconfiguration. The way in which it does this is by querying the following two environmental variables which are set when the vCenter Server alarm is triggered and below is an example of their values:

Alarm Environmental Variable	Value
VMWARE_ALARM_TRIGGERINGSUMMARY	Event: VM reconfigured (9322)
VMWARE_ALARM_TARGET_ID	vm-125

In the first variable, what we care about is the eventId which is associated with the VM reconfiguration. What I have found is you have to take this ID and subtract 1 to get the event which actually contains the reconfiguration information that we want. The second variable provides us with the MoRef ID of the VM that was configured. Using these two pieces of information, we can then perform an event lookup to pull out the configuration changes that were made to that particular VM.

Here are the steps for creating the vCenter Server Alarm:

Step 1 - Ensure that the latest PowerCLI 6.0 Release 1 is installed on your vCenter Server. I will assume that this is how you wish to execute the PowerCLI script. If not, other options can include sending an SNMP trap to vRealize Orchestrator and have it perform the execution of the script.

Step 2 - Create a new Alarm and fill in the general settings as shown in the screenshots below.

Step 3 - Add the "VM reconfigured" trigger with the status set to "Unset", this will ensure you do not have a notification icon when the alarm is activated.

Step 4 - Select "Run a command" as the action and then paste the full path to where a "wrapper.bat" script will be located in the vCenter Server.

Step 5 - Create the "wrapper.bat" script on the vCenter Server system with the following (adjust the paths to fit your environment):

C:\Windows\System32\cmd.exe /c powershell.exe -noninteractive -noprofile -file C:\Users\primp\Desktop\Get-VMConfigChangesFromAlarm.ps1 >> C:\Users\primp\Desktop\alarm.log

The following snippet will execute the PowerCLI script and any console output will be re-directed to the alarm.log file (which can be helpful in troubleshooting errors in the script itself).

Step 6 - Download the Get-VMConfigChangesFromAlarm.ps1 script and place it on the vCenter Server system and ensure it aligns with the path of the wrapper.bat script. You will also need to edit the script to update the vCenter Server credentials as well as the log file in which the VM configurations will be stored. Currently it will just append to a file called alarm.txt.

If everything was configured successfully, you can now test the alarm by simply editing one of your VMs. If you click under Monitor->Events for the VM, you should see the alarm being triggered and that the script was executed.

If we take a look at our alarm.txt file, we should hopefully see the VM reconfiguration details logged like the following:

ChangeVersion : 2015-08-13T21:10:58.248345Z
DeviceChange : {VMware.Vim.VirtualDeviceConfigSpec}
Files : VMware.Vim.VirtualMachineFileInfo
MemoryMB : 8192
NumCPUs : 2
VMName : Test-VM
Start : 8/13/2015 9:11:17 PM
End : 8/13/2015 9:11:19 PM
State : success
User : VGHETTO.LOCAL\Administrator
Device : VirtualDisk
Operation : edit

Logging the output to a file is just one way of storing this data. I am sure some of you may want to modify the script to forward to a remote syslog collector like vRealize Log Insight for example or directly storing it into a Big Data platform. I will leave that as an exercise for my readers but hopefully this give you idea on how you can automatically archive all VM reconfigurations along with the what, when and who details for auditing and/or reporting purposes in the future.

Lastly, I wanted to give a big thanks to Jonathan Medd who helped me with a problem I was running into when trying to get a PowerCLI script to execute when using a vCenter Server Alarm. It turned out that I had the vCenter Server service configure to run as a local admin and switching it over to a Domain Account resolved my problem.

How to audit VM reconfigurations and see what exactly changed?

08.13.2015 by William Lam // 27 Comments

A question that I almost always see come up on a regular basis is around the topic of auditing or understanding what configuration changes were made to a given Virtual Machine. Today, the process of identifying this information is actually quite difficult at least without resorting to a custom built solution which allows you to compare the configuration changes over time. This is definitely an area that VMware is investing heavily in and although I can not go into specific details, believe me when I say they are taking this very seriously both from a data completeness standpoint as well as simplifying the user experience.

Having said that, what options exists for customers today?

Custom Solution - A system that could periodically snapshot your VM configurations into some type of data warehouse or CMDB platform. There are many challenges here but the biggest one is efficiently capturing the changes and ensuring you do not impact the overall performance of your vCenter Server, especially with larger inventories.
vRealize Air Compliance - This is a new VMware SaaS offering which provides both compliance and remediation for your VM Configurations. I had a chance to preview this awhile back and I have to say it is a pretty slick solution. You can easily step back in time to see exactly what has changed for a given set of VMs, very intuitive UI. You can also add additional configurations to monitor and be alerted on when you are out of compliance. Definitely something worth checking out, especially for customers who must go through regular audit reviews.
vCenter Configuration Manager - I have not personally used this tool before, but I have been told it would also be possible to detect configuration changes for your VMs.
Enabling "Trivia" Logging in vCenter Server - Though this is an option, it is not one that I recommend for variety of reasons. The "Trivia" logging mode is very verbose and will generate huge amounts of data which will causes your logs to quickly rotate out if you are not forwarding to a remote syslog server. There's also additional overhead cost for this type of logging and more importantly, it may not capture all of the required data. This is an approach that some customers have tried but is not really a practical solution.

This topic has always been interesting to me and with several recent inquiries from the field, it got me thinking about this area again. While working on a completely different project, I ended up on Luc Dekens awesome blog and came across his Events Part 3: Auditing VM Device Changes article. If you take a look at the article, you will see that Luc shows you how you can easily audit changes to a VM's devices (e.g. Virtual Disk, CD-ROM, etc). What Luc demonstrated in his script is just a specific type of configuration, but the point is that this type of information has always been available, just not easily consumable.

The secret is to key off of the VmReconfiguredEvent which includes a configSpec property that captures the exact set of configuration changes for a given VM. Below is an example of the configSpec dump of one of these events. We can clearly see that this VM had its vCPUs modified to 4 and its vMEM modified to 20GB.

With this information, we can now easily query the configuration changes for a given VM by looking through its past events. Leveraging the awesome work that Lud has already done with his script, I slightly enhanced it to cover more than just device changes but overall VM configuration changes. With that, here is a PowerCLI script/function that I created called Get-VMConfigChanges.ps1

Note: The amount of historical events that you will be able to search through will purely depend on your Center Server DB's retention period of Tasks/Events. For VMs which have been deleted, you will not be able to retrieve any events as they must be associated with an object in the database.

Below is an example of how to use this function which accepts a VM object and the number of hours (default to 8) to search through the VM's events:

$vcserver = "192.168.1.150"
$vcusername = "*protected email*"
$vcpassword = "VMware1!"

Connect-VIServer -Server $vcserver -User $vcusername -Password $vcpassword

$vm = Get-VM "Test-VM"

Get-VMConfigChanges -vm $vm -hours 12

Disconnect-VIServer -Server $vcserver -Confirm:$false

From the output below, we can clearly see the following configuration changes have been applied:

Change vCPU to 2
Change vMEM to 4GB
Change vMEM to 5GB and Edited Virtual Disk (you can of course get further details by dumping more information)

Although this solution is not as clean as the vRealize Air Compliance offering, it does allow anyone to quickly pull out the relevant configuration changes for a given VM along with the user and time the configurations was performed. Hopefully this goes to show how powerful the vSphere Platform APIs really are and it is definitely worth while in learning how they work.

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

07.14.2015 by William Lam // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI: