William Lam

How to automatically log all VM configuration changes using a vCenter Server Alarm?

08.18.2015 by William Lam // 9 Comments

If you followed my previous blog post on How to audit VM reconfigurations and see what exactly changed, you may have concluded that historical VM configuration events may potentially be unavailable by the time you need to perform an audit. The reason for this is that the VM Events tables may have rotated out depending on the retention policy of your vCenter Server Database. This is where we can take advantage of the powerful vCenter Server Alarm feature which would allows us to capture every single VM reconfiguration and store this information outside of the vCenter Server. This not only allows you to reduce the amount of data stored in the vCenter Server Database but it also allows you to efficiently archive this data into a data warehouse or Big Data platform that provides more advanced analytics and reporting capabilities.

Building on top of our previous script, I have created a slightly modified version called Get-VMConfigChangesFromAlarm.ps1. The main difference is instead of passing in a VM object to look for past configuration changes, the script can now pull in information from a triggered VmReconfigureEvent and log the specific changes associated with that VM reconfiguration. The way in which it does this is by querying the following two environmental variables which are set when the vCenter Server alarm is triggered and below is an example of their values:

Alarm Environmental Variable	Value
VMWARE_ALARM_TRIGGERINGSUMMARY	Event: VM reconfigured (9322)
VMWARE_ALARM_TARGET_ID	vm-125

In the first variable, what we care about is the eventId which is associated with the VM reconfiguration. What I have found is you have to take this ID and subtract 1 to get the event which actually contains the reconfiguration information that we want. The second variable provides us with the MoRef ID of the VM that was configured. Using these two pieces of information, we can then perform an event lookup to pull out the configuration changes that were made to that particular VM.

Here are the steps for creating the vCenter Server Alarm:

Step 1 - Ensure that the latest PowerCLI 6.0 Release 1 is installed on your vCenter Server. I will assume that this is how you wish to execute the PowerCLI script. If not, other options can include sending an SNMP trap to vRealize Orchestrator and have it perform the execution of the script.

Step 2 - Create a new Alarm and fill in the general settings as shown in the screenshots below.

Step 3 - Add the "VM reconfigured" trigger with the status set to "Unset", this will ensure you do not have a notification icon when the alarm is activated.

Step 4 - Select "Run a command" as the action and then paste the full path to where a "wrapper.bat" script will be located in the vCenter Server.

Step 5 - Create the "wrapper.bat" script on the vCenter Server system with the following (adjust the paths to fit your environment):

C:\Windows\System32\cmd.exe /c powershell.exe -noninteractive -noprofile -file C:\Users\primp\Desktop\Get-VMConfigChangesFromAlarm.ps1 >> C:\Users\primp\Desktop\alarm.log

The following snippet will execute the PowerCLI script and any console output will be re-directed to the alarm.log file (which can be helpful in troubleshooting errors in the script itself).

Step 6 - Download the Get-VMConfigChangesFromAlarm.ps1 script and place it on the vCenter Server system and ensure it aligns with the path of the wrapper.bat script. You will also need to edit the script to update the vCenter Server credentials as well as the log file in which the VM configurations will be stored. Currently it will just append to a file called alarm.txt.

If everything was configured successfully, you can now test the alarm by simply editing one of your VMs. If you click under Monitor->Events for the VM, you should see the alarm being triggered and that the script was executed.

If we take a look at our alarm.txt file, we should hopefully see the VM reconfiguration details logged like the following:

ChangeVersion : 2015-08-13T21:10:58.248345Z
DeviceChange : {VMware.Vim.VirtualDeviceConfigSpec}
Files : VMware.Vim.VirtualMachineFileInfo
MemoryMB : 8192
NumCPUs : 2
VMName : Test-VM
Start : 8/13/2015 9:11:17 PM
End : 8/13/2015 9:11:19 PM
State : success
User : VGHETTO.LOCAL\Administrator
Device : VirtualDisk
Operation : edit

Logging the output to a file is just one way of storing this data. I am sure some of you may want to modify the script to forward to a remote syslog collector like vRealize Log Insight for example or directly storing it into a Big Data platform. I will leave that as an exercise for my readers but hopefully this give you idea on how you can automatically archive all VM reconfigurations along with the what, when and who details for auditing and/or reporting purposes in the future.

Lastly, I wanted to give a big thanks to Jonathan Medd who helped me with a problem I was running into when trying to get a PowerCLI script to execute when using a vCenter Server Alarm. It turned out that I had the vCenter Server service configure to run as a local admin and switching it over to a Domain Account resolved my problem.

How to audit VM reconfigurations and see what exactly changed?

08.13.2015 by William Lam // 27 Comments

A question that I almost always see come up on a regular basis is around the topic of auditing or understanding what configuration changes were made to a given Virtual Machine. Today, the process of identifying this information is actually quite difficult at least without resorting to a custom built solution which allows you to compare the configuration changes over time. This is definitely an area that VMware is investing heavily in and although I can not go into specific details, believe me when I say they are taking this very seriously both from a data completeness standpoint as well as simplifying the user experience.

Having said that, what options exists for customers today?

Custom Solution - A system that could periodically snapshot your VM configurations into some type of data warehouse or CMDB platform. There are many challenges here but the biggest one is efficiently capturing the changes and ensuring you do not impact the overall performance of your vCenter Server, especially with larger inventories.
vRealize Air Compliance - This is a new VMware SaaS offering which provides both compliance and remediation for your VM Configurations. I had a chance to preview this awhile back and I have to say it is a pretty slick solution. You can easily step back in time to see exactly what has changed for a given set of VMs, very intuitive UI. You can also add additional configurations to monitor and be alerted on when you are out of compliance. Definitely something worth checking out, especially for customers who must go through regular audit reviews.
vCenter Configuration Manager - I have not personally used this tool before, but I have been told it would also be possible to detect configuration changes for your VMs.
Enabling "Trivia" Logging in vCenter Server - Though this is an option, it is not one that I recommend for variety of reasons. The "Trivia" logging mode is very verbose and will generate huge amounts of data which will causes your logs to quickly rotate out if you are not forwarding to a remote syslog server. There's also additional overhead cost for this type of logging and more importantly, it may not capture all of the required data. This is an approach that some customers have tried but is not really a practical solution.

This topic has always been interesting to me and with several recent inquiries from the field, it got me thinking about this area again. While working on a completely different project, I ended up on Luc Dekens awesome blog and came across his Events Part 3: Auditing VM Device Changes article. If you take a look at the article, you will see that Luc shows you how you can easily audit changes to a VM's devices (e.g. Virtual Disk, CD-ROM, etc). What Luc demonstrated in his script is just a specific type of configuration, but the point is that this type of information has always been available, just not easily consumable.

The secret is to key off of the VmReconfiguredEvent which includes a configSpec property that captures the exact set of configuration changes for a given VM. Below is an example of the configSpec dump of one of these events. We can clearly see that this VM had its vCPUs modified to 4 and its vMEM modified to 20GB.

With this information, we can now easily query the configuration changes for a given VM by looking through its past events. Leveraging the awesome work that Lud has already done with his script, I slightly enhanced it to cover more than just device changes but overall VM configuration changes. With that, here is a PowerCLI script/function that I created called Get-VMConfigChanges.ps1

Note: The amount of historical events that you will be able to search through will purely depend on your Center Server DB's retention period of Tasks/Events. For VMs which have been deleted, you will not be able to retrieve any events as they must be associated with an object in the database.

Below is an example of how to use this function which accepts a VM object and the number of hours (default to 8) to search through the VM's events:

$vcserver = "192.168.1.150"
$vcusername = "*protected email*"
$vcpassword = "VMware1!"

Connect-VIServer -Server $vcserver -User $vcusername -Password $vcpassword

$vm = Get-VM "Test-VM"

Get-VMConfigChanges -vm $vm -hours 12

Disconnect-VIServer -Server $vcserver -Confirm:$false

From the output below, we can clearly see the following configuration changes have been applied:

Change vCPU to 2
Change vMEM to 4GB
Change vMEM to 5GB and Edited Virtual Disk (you can of course get further details by dumping more information)

Although this solution is not as clean as the vRealize Air Compliance offering, it does allow anyone to quickly pull out the relevant configuration changes for a given VM along with the user and time the configurations was performed. Hopefully this goes to show how powerful the vSphere Platform APIs really are and it is definitely worth while in learning how they work.

New HTML5 Embedded Host Client for ESXi

08.11.2015 by William Lam // 61 Comments

How many of you remember the old WebAccess interface that came with classic ESX 2.0? I know I sure do! It was a very useful interface that allowed administrators to easily manage their ESX hosts through a web browser that provided basic VM creation, troubleshooting and most importantly initial bootstrap configurations for greenfield deployments where a vCenter Server may not be available yet.

I have always felt that having a simple web interface that customers can just point their browser to an ESX(i) host after an installation would be the most intuitive from a user experience standpoint. About a year ago, I was involved in a project with two VMware Engineers, Kevin Christopher and Jehad Affoneh, when we worked on an early prototype of an optimized Embedded Host Client for ESXi. We socialized the prototype across many different teams at VMware and the overwhelming feedback was super positive. Earlier this year, with the help of the ESXi PM and the Engineering team, they decided to take the concept of the prototype to the next level and really turn it into something that could really be used by customers.

UPDATE: (08/25/15) - v2 has just been released, be sure to take a look at this blog post for more details.

Today, I am very happy to announce the release of a new VMware Fling: HTML5 Embedded Host Client for ESXi which is currently distributed as a simple installable VIB. The Fling currently only supports ESXi 6.0 (however, there is a workaround ESXi 5.x which you can find below). In this first release, the Free ESXi Hypervisor is not supported, but the plan is to add support for this in a future update. Here is a list of some of the supported operations in this first release:

VM operations (power on, off, reset, suspend, etc.)
Creating a new VM, from scratch or from OVF/OVA (limited OVA support)
Configuring NTP on a host
Displaying summaries, events, tasks and notifications/alerts
Providing a console to VMs
Configuring host networking
Configuring host advanced settings
Configuring host services
License management

Once the VIB is installed, which does not require a system reboot, you simply just point your web browser to the following URL: https://[ESXI-HOST]/ui and login with an administrator account.

One of my favorite features of the Embedded Host Client is the integrated VM Console which is accessible directly within the browser and does not require any additional plugins or installers. I do want to mention that the VM Console feature is currently only available when installed onto an ESXi 6.0 host. One other caveat that I would like to point out is that for customers who wish to try the Embedded Host Client on ESXi 5.5 or on ESXi 6.0 host which have been upgraded from an ESXi 5.x environment, the following workaround is required:

ESXi 5.x or ESXi 6.0 (upgraded from ESXi 5.x) workaround:

Step 1 - SSH to your ESXi host and open the following configuration file:

/etc/vmware/rhttpproxy/endpoints.conf

Step 2 - Remove the following line and save the changes

/ui local 8308 redirect allow

Step 3 - Restart the reverse proxy by running the following command:

/etc/init.d/rhttpproxy restart

Step 4 - Open a browser to the following URL: https://[ESXI-HOST]/ui/ and note the additional trailing "slash" at the end of the URL. This is mandatory else the Embedded Host Client page will not properly load.

Note: This only applies to ESXi 6.0 hosts which have been upgraded from ESXi 5.x or pre-ESXi 5.5 Update 3 systems. You may also noticed translation errors when running the Fling on pre-ESXi 6.0, this is expected. For the best experience, we recommend using ESXi 6.0 or waiting until ESXi 5.5 Update 3 is released which will resolve this problem.

Having spent the last couple of months working with both of the Engineers: Etienne Le Sueur and George Estebe, I was able to help provide feedback and usability enhancements and I think you will really like what is there in this first release. I do want to stress that this is in no way, shape or form a comprehensive replacement for the existing vSphere C# Client functionality, but I think it could be a good start if the feedback from customers is positive. I really hope you do give this a try in your lab environment and please let us know what features and capabilities we should add next. There is a variety of way you can provide feedback by leaving a comment on this blog post, leaving a comment on the Flings page.

Big thanks goes out to both George and Etienne who took on the challenge and did a fantastic job on the Fling!