A question that I almost always see come up on a regular basis is around the topic of auditing or understanding what configuration changes were made to a given Virtual Machine. Today, the process of identifying this information is actually quite difficult at least without resorting to a custom built solution which allows you to compare the configuration changes over time. This is definitely an area that VMware is investing heavily in and although I can not go into specific details, believe me when I say they are taking this very seriously both from a data completeness standpoint as well as simplifying the user experience.
Having said that, what options exists for customers today?
- Custom Solution - A system that could periodically snapshot your VM configurations into some type of data warehouse or CMDB platform. There are many challenges here but the biggest one is efficiently capturing the changes and ensuring you do not impact the overall performance of your vCenter Server, especially with larger inventories.
- vRealize Air Compliance - This is a new VMware SaaS offering which provides both compliance and remediation for your VM Configurations. I had a chance to preview this awhile back and I have to say it is a pretty slick solution. You can easily step back in time to see exactly what has changed for a given set of VMs, very intuitive UI. You can also add additional configurations to monitor and be alerted on when you are out of compliance. Definitely something worth checking out, especially for customers who must go through regular audit reviews.
- vCenter Configuration Manager - I have not personally used this tool before, but I have been told it would also be possible to detect configuration changes for your VMs.
- Enabling "Trivia" Logging in vCenter Server - Though this is an option, it is not one that I recommend for variety of reasons. The "Trivia" logging mode is very verbose and will generate huge amounts of data which will causes your logs to quickly rotate out if you are not forwarding to a remote syslog server. There's also additional overhead cost for this type of logging and more importantly, it may not capture all of the required data. This is an approach that some customers have tried but is not really a practical solution.
This topic has always been interesting to me and with several recent inquiries from the field, it got me thinking about this area again. While working on a completely different project, I ended up on Luc Dekens awesome blog and came across his Events Part 3: Auditing VM Device Changes article. If you take a look at the article, you will see that Luc shows you how you can easily audit changes to a VM's devices (e.g. Virtual Disk, CD-ROM, etc). What Luc demonstrated in his script is just a specific type of configuration, but the point is that this type of information has always been available, just not easily consumable.
The secret is to key off of the VmReconfiguredEvent which includes a configSpec property that captures the exact set of configuration changes for a given VM. Below is an example of the configSpec dump of one of these events. We can clearly see that this VM had its vCPUs modified to 4 and its vMEM modified to 20GB.
With this information, we can now easily query the configuration changes for a given VM by looking through its past events. Leveraging the awesome work that Lud has already done with his script, I slightly enhanced it to cover more than just device changes but overall VM configuration changes. With that, here is a PowerCLI script/function that I created called Get-VMConfigChanges.ps1
Note: The amount of historical events that you will be able to search through will purely depend on your Center Server DB's retention period of Tasks/Events. For VMs which have been deleted, you will not be able to retrieve any events as they must be associated with an object in the database.
Below is an example of how to use this function which accepts a VM object and the number of hours (default to 8) to search through the VM's events:
$vcserver = "192.168.1.150" $vcusername = "*protected email*" $vcpassword = "VMware1!" Connect-VIServer -Server $vcserver -User $vcusername -Password $vcpassword $vm = Get-VM "Test-VM" Get-VMConfigChanges -vm $vm -hours 12 Disconnect-VIServer -Server $vcserver -Confirm:$false
From the output below, we can clearly see the following configuration changes have been applied:
- Change vCPU to 2
- Change vMEM to 4GB
- Change vMEM to 5GB and Edited Virtual Disk (you can of course get further details by dumping more information)
Although this solution is not as clean as the vRealize Air Compliance offering, it does allow anyone to quickly pull out the relevant configuration changes for a given VM along with the user and time the configurations was performed. Hopefully this goes to show how powerful the vSphere Platform APIs really are and it is definitely worth while in learning how they work.
Dell FogLight has, or used to have, an option for tracking changes.
You could properly also use the vmware.log files and to ease the process send them to a syslog, then there should be potential for some datacenter wide analytics on changes. Using log insight there could also be created events and send til vRops and there from reporting should be easy.
VMware log files are not rich enough.
to what extent ? The how vmx config is saved in there - but only on boot or why is it not rich enough ? care to elaborate ?
In William's example the VM had the number of CPU's and memory changed. That data is in the VCDB and William has done a great job showing how to get that information out.
The issue is that that same information is not expressed in the logs. All you'll get is "William reconfigured the VM ". That's it. Logging systems like Log Insight and others are only as good as the data they get. To make the data actionable, you'd be looking for keywords. For example, if the logging was "William changed the VM memory on VM from 2048 to 4096" then we have something we can work with on a logging system. I can parse for "changed the VM memory" and the values and create an alert accordingly.
My favorite example is I have a VM on the PCI network. An admin changes that to the non-PCI network. The log is "*protected email* reconfigured . Doesn't tell me what from/to. If the log was "*protected email* reconfigured network from "PCI Network" to "External Network" THEN I have some actionable items.
From an operational standpoint this is far easier to react to than going in after the fact to retrieve the information or setting up alerts and scripts. (I'm not dismissing the great work William has done at all. I think he'd agree with me. Mostly :))
Here's hoping it gets better in the future (of which we cannot speak about)
mike
@Mike - I think @michael was asking about vmware.log (Virtual Machine's log file that's generated when its powered up) and not general VC/ESXi logging which I agree doesn't contain the details.
The vmware.log DOES contain all VM configurations, however you would have to periodically dump the file and/or forward to a remote syslog server and then compare the deltas which can be quite difficult. So though you *could* use this method to retrieve the properties, its probably not very efficient
Very Nice!!! Thanks!!!
Hey William, so I have this up and running on an automation VM and it emails me a report every 12 hours. So far it looks ok, but what the information I'm really looking for is auditing of disk expansions. Like, who added 50GB to a drive on server X and when was it done? So far this script is spotty at best at even reporting that a change is made, but it doesn't include information as to what drive and how much space was added. Hoping this is including in something like vROps soon.
Adam,
I'm not sure what you mean by "spotty", assuming you've specified the right time interval, any "Reconfigure" operation should be caught by the script. Below is an example of expanding a VMDK for a VM called "Base-ESXi", you can see the details below on the specific user, start/stop time and also that this was an Edit operation on the VMDK. It doesn't go to the next level of detail which I agree would be useful and something Engr is quite aware of and will be improving in the future.
ChangeVersion : 2016-06-08T20:20:21.077253Z
DeviceChange : {VMware.Vim.VirtualDeviceConfigSpec}
Files : VMware.Vim.VirtualMachineFileInfo
VMName : Base-ESXi
Start : 6/17/2016 7:03:39 PM
End : 6/17/2016 7:03:39 PM
State : success
User : VGHETTO.LOCAL\Administrator
ChainID : 276545
Device : VirtualDisk
Operation : edit
Spotty meaning the reporting isn't very consistent. I have this running every 12 hours as a scheduled task and the script pulls 12 hours of data. Last week during testing I would make disk changes (add, delete or expand) and only 6 out of 10 changes showed up in the report. Today I tried changes on 2 VMs and neither showed up in the report. Not really looking for a solution now (although it would be nice), but hoping that something like this will be included in the rumored vROps update later this year.
Finding more...the script as is will report on a disk expansion, but not when a new disk is added to a VM.
Replying to myself again. I'm using Datastore clusters and SDRS is enabled, so adding a disk is a vCenter event "Apply Storage DRS recommendations" and not a VM level reconfigure machine vent. At least there's an explanation for it!
Hi. Just love this script. 🙂
Any chance you can get the same information about changes in vCenter environment? What I'm looking for is host configuration changes or vCenter cluster settings changes and so on?
Hi William, Thanks for the script. Im looking for a way to track vcpu and memory changes. Is there a way to get what the setting was changed from as well? E.g. the VM was 8GB and is now 10GB. Thanks, Geoff.
Neat script. Doesn't seem to be helpful for vlan changes tho
William,
Fantastic. Do you know if is possibile audit changes when add/remove vm in DRS Group manager?
great, thx
William, any update on this for vSphere 6 - Get-VMConfigChanges is no longer recognized.
I'm not sure what you mean ... it still works for me. Remember, this looks at the Events which is maintained by your retention period. I recommend making a tiny edit to a dummy VM and try the query to ensure that works (which is what I just did)
Using PowerCLI 6.5.0.234
PowerCLI C:\> $vm = Get-VM "VMNAME"
PowerCLI C:\> Get-VMConfigChanges -vm $vm -
Get-VMConfigChanges : The term 'Get-VMConfigChanges' is not recognized as the
name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is
correct and try again.
At line:1 char:1
+ Get-VMConfigChanges -vm $vm -
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Get-VMConfigChanges:String) [],
CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Did you actually load the Get-VMConfigChanges function? That's what the error is indicating 🙂
Hello William.
I'm having quite another issue with your script that I hope you can help me with.
When I try to run the script i get following error(s).
Exception setting "Entity": "Cannot convert the "VirtualMachine-vm-910" value of type "VMware.Vim.ManagedObjectReference" to type "VMware.Vim.ManagedObjectReference"."
At line:68 char:5
+ $tFilter.Entity.Entity = $vm.ExtensionData.MoRef
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], SetValueInvocationException
+ FullyQualifiedErrorId : ExceptionWhenSetting
Cannot convert argument "filter", with value: "VMware.Vim.TaskFilterSpec", for "CreateCollectorForTasks" to type "VMware.Vim.TaskFilterSpec": "Cannot convert the "VMware.Vim.TaskFilterSpec"
value of type "VMware.Vim.TaskFilterSpec" to type "VMware.Vim.TaskFilterSpec"."
At line:72 char:5
+ $tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument
You cannot call a method on a null-valued expression.
At line:75 char:5
+ $tasks = $tCollector.ReadNextTasks($tasknumber)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
You cannot call a method on a null-valued expression.
At line:107 char:5
+ $tCollector.DestroyCollector()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
Thanks!
Did you find out what is causing this? I am also getting same error when I run this script:
Exception calling "CreateCollectorForTasks" with "1" argument(s): "
Required property entity is missing from data object of type TaskFilterSpecByEntity
while parsing serialized DataObject of type vim.TaskFilterSpec.ByEntity
at line 1, column 267
while parsing property "entity" of static type TaskFilterSpecByEntity
while parsing serialized DataObject of type vim.TaskFilterSpec
at line 1, column 259
while parsing call information for method CreateCollectorForTasks
at line 1, column 171
while parsing SOAP body
at line 1, column 64
while parsing SOAP envelope
at line 1, column 0
while parsing HTTP request for method createCollector
on object of type vim.TaskManager
at line 1, column 0"
At line:48 char:5
+ $tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : VimException
You cannot call a method on a null-valued expression.
At line:50 char:5
+ $tasks = $tCollector.ReadNextTasks($tasknumber)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
You cannot call a method on a null-valued expression.
At line:79 char:5
+ $tCollector.DestroyCollector()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
Great script, i have been lookig for something like this for a while. I have a 6.5 vcentre installation managin around 300VMs, and it works great.
I also export the VIEvent logs each day, but cannot seem to find the event details from the script in the VIEvent logs..
Or am i doing something wrong here..
Hi, would this work for Vcentre /Vsphere 5.5 or only 6.0 and above.
Also anyway to know for how long has the data been logged. Example if I put the hours as 1000 hrs how would it react . Would it give spurious data or no data if no information is available on the system.
Is there any way to identify who enabled the option to Synchronize time with the host on a VM ?
Hi William, What is the current solution offered by VMware to detect VM and Host configuration?