PowerCLI – Page 38

How Wrecking Crew Inc. leveraged vSphere's Instant Clone to instantly provision hundreds of VMs

09.18.2015 by William Lam // 9 Comments

Okay, Wrecking Crew Inc. is not a real company but a fictitious one that was used during Alan Renouf and Li Zheng's VMworld session, #INF5803 - Deploy Hundreds of VMs Instantly via Forking (aka vSphere Instant Clone). During the session, Wrecking Crew Inc. was described as a modern online gaming company who was about to launch a new product. They found out that their game was going to be more popular than they had originally anticipated and needed to be able to quickly spin up a large number of instances of their game to handle the load.

Though this was a fictional company to help set the stage for the demo, the underlying use case is an actual challenge for many of our customers. One of the examples that Alan used was the traditional Black Friday sale which occurs on the Friday after Thanksgiving in US. As you can imagine, being able to quickly scale up your online application based on customer demand can really give retailers an real edge over their competition. In todays world, waiting a few additional seconds for a site to load can mean the difference between a sale and customer going else where for their business. Online retail is just one of the many industries and verticals that can easily benefit from the Instant Clone capability.

To help further demonstrate the use case, a recorded demo was shown utilizing the new PowerCLI Extensions which provides access to the Instant Clone feature from an Automation standpoint. I actually built the demo for Alan and Li's session and I thought I would share some more details along with the video in case you were not able to attend the session in person, which I was not able to do.

Below are the set of technologies that were used in the demo:

I wanted to also give a quick shoutout to my buddy Rawlinson Rivera for his help with Intel and getting us access to their 64 Node NVMe VSAN Cluster. Our friends over at Intel were kind enough to allow us to quickly record a demo before they had to pack and ship the hardware to San Francisco for VMworld.

To provide some additional context on what you will see in the video, a diagram is provided below. The environment initially consists of two Photon VMs which runs the Consul Cluster which provides service discovery and configuration capabilities and the Nginx Load Balancer which is highlighted in the purple and blue box. Each of those VMs run their respective application which runs inside of a Docker Container and is highlighted in the smaller boxes. On each ESXi host, we have Photon VM which contains our Node.js application that we will be Instant Cloning from. This Photon VM is also referred to as the Parent VM and where all the new forked children VMs will be spawn off of.

This Photon VM also runs an instance of Consul, Registrator and the Node.js application inside of a Docker Container. As new instances are brought online which is highlighted in the black box, they will automatically register themselves with the Consul Cluster. As the Docker Containers are started up, Registrator will be notified to automatically add the new instances to the Nginx Load Balancer, making our application servers available for use immediately. When our application server powers down, Registrator will automatically detect that the Docker Container for our Node.js application is no longer running and automatically unregister it from the Nginx Load Balancer. To simply scale up or down the application, it is simply Instant Cloning our Photon (Parent) VM and powering it on or off, there are no additional steps required.

Without further ado, here is the video of the demo. If you want additional commentary and you attended VMworld US but could not make it to Alan and Li's session, you can now watch it online here. This session will also be repeated in VMworld EMEA for those attending in a couple of weeks. I would highly recommend you check out the session as there is a lot of awesomeness in the session along with technical deep dive of the Instant Clone technology.

VMworld 2015 Demo of how Wrecking Crew Inc. leverages vSphere's Instant Clone feature from lamw on Vimeo.

For more details about Instant Clone, be sure to check out these resources below which includes a Instant Clone script repository for several GuestOSes, including Photon which was used in the demo:

How to automatically log all VM configuration changes using a vCenter Server Alarm?

08.18.2015 by William Lam // 9 Comments

If you followed my previous blog post on How to audit VM reconfigurations and see what exactly changed, you may have concluded that historical VM configuration events may potentially be unavailable by the time you need to perform an audit. The reason for this is that the VM Events tables may have rotated out depending on the retention policy of your vCenter Server Database. This is where we can take advantage of the powerful vCenter Server Alarm feature which would allows us to capture every single VM reconfiguration and store this information outside of the vCenter Server. This not only allows you to reduce the amount of data stored in the vCenter Server Database but it also allows you to efficiently archive this data into a data warehouse or Big Data platform that provides more advanced analytics and reporting capabilities.

Building on top of our previous script, I have created a slightly modified version called Get-VMConfigChangesFromAlarm.ps1. The main difference is instead of passing in a VM object to look for past configuration changes, the script can now pull in information from a triggered VmReconfigureEvent and log the specific changes associated with that VM reconfiguration. The way in which it does this is by querying the following two environmental variables which are set when the vCenter Server alarm is triggered and below is an example of their values:

Alarm Environmental Variable	Value
VMWARE_ALARM_TRIGGERINGSUMMARY	Event: VM reconfigured (9322)
VMWARE_ALARM_TARGET_ID	vm-125

In the first variable, what we care about is the eventId which is associated with the VM reconfiguration. What I have found is you have to take this ID and subtract 1 to get the event which actually contains the reconfiguration information that we want. The second variable provides us with the MoRef ID of the VM that was configured. Using these two pieces of information, we can then perform an event lookup to pull out the configuration changes that were made to that particular VM.

Here are the steps for creating the vCenter Server Alarm:

Step 1 - Ensure that the latest PowerCLI 6.0 Release 1 is installed on your vCenter Server. I will assume that this is how you wish to execute the PowerCLI script. If not, other options can include sending an SNMP trap to vRealize Orchestrator and have it perform the execution of the script.

Step 2 - Create a new Alarm and fill in the general settings as shown in the screenshots below.

Step 3 - Add the "VM reconfigured" trigger with the status set to "Unset", this will ensure you do not have a notification icon when the alarm is activated.

Step 4 - Select "Run a command" as the action and then paste the full path to where a "wrapper.bat" script will be located in the vCenter Server.

Step 5 - Create the "wrapper.bat" script on the vCenter Server system with the following (adjust the paths to fit your environment):

C:\Windows\System32\cmd.exe /c powershell.exe -noninteractive -noprofile -file C:\Users\primp\Desktop\Get-VMConfigChangesFromAlarm.ps1 >> C:\Users\primp\Desktop\alarm.log

The following snippet will execute the PowerCLI script and any console output will be re-directed to the alarm.log file (which can be helpful in troubleshooting errors in the script itself).

Step 6 - Download the Get-VMConfigChangesFromAlarm.ps1 script and place it on the vCenter Server system and ensure it aligns with the path of the wrapper.bat script. You will also need to edit the script to update the vCenter Server credentials as well as the log file in which the VM configurations will be stored. Currently it will just append to a file called alarm.txt.

If everything was configured successfully, you can now test the alarm by simply editing one of your VMs. If you click under Monitor->Events for the VM, you should see the alarm being triggered and that the script was executed.

If we take a look at our alarm.txt file, we should hopefully see the VM reconfiguration details logged like the following:

ChangeVersion : 2015-08-13T21:10:58.248345Z
DeviceChange : {VMware.Vim.VirtualDeviceConfigSpec}
Files : VMware.Vim.VirtualMachineFileInfo
MemoryMB : 8192
NumCPUs : 2
VMName : Test-VM
Start : 8/13/2015 9:11:17 PM
End : 8/13/2015 9:11:19 PM
State : success
User : VGHETTO.LOCAL\Administrator
Device : VirtualDisk
Operation : edit

Logging the output to a file is just one way of storing this data. I am sure some of you may want to modify the script to forward to a remote syslog collector like vRealize Log Insight for example or directly storing it into a Big Data platform. I will leave that as an exercise for my readers but hopefully this give you idea on how you can automatically archive all VM reconfigurations along with the what, when and who details for auditing and/or reporting purposes in the future.

Lastly, I wanted to give a big thanks to Jonathan Medd who helped me with a problem I was running into when trying to get a PowerCLI script to execute when using a vCenter Server Alarm. It turned out that I had the vCenter Server service configure to run as a local admin and switching it over to a Domain Account resolved my problem.

How to audit VM reconfigurations and see what exactly changed?

08.13.2015 by William Lam // 27 Comments

A question that I almost always see come up on a regular basis is around the topic of auditing or understanding what configuration changes were made to a given Virtual Machine. Today, the process of identifying this information is actually quite difficult at least without resorting to a custom built solution which allows you to compare the configuration changes over time. This is definitely an area that VMware is investing heavily in and although I can not go into specific details, believe me when I say they are taking this very seriously both from a data completeness standpoint as well as simplifying the user experience.

Having said that, what options exists for customers today?

Custom Solution - A system that could periodically snapshot your VM configurations into some type of data warehouse or CMDB platform. There are many challenges here but the biggest one is efficiently capturing the changes and ensuring you do not impact the overall performance of your vCenter Server, especially with larger inventories.
vRealize Air Compliance - This is a new VMware SaaS offering which provides both compliance and remediation for your VM Configurations. I had a chance to preview this awhile back and I have to say it is a pretty slick solution. You can easily step back in time to see exactly what has changed for a given set of VMs, very intuitive UI. You can also add additional configurations to monitor and be alerted on when you are out of compliance. Definitely something worth checking out, especially for customers who must go through regular audit reviews.
vCenter Configuration Manager - I have not personally used this tool before, but I have been told it would also be possible to detect configuration changes for your VMs.
Enabling "Trivia" Logging in vCenter Server - Though this is an option, it is not one that I recommend for variety of reasons. The "Trivia" logging mode is very verbose and will generate huge amounts of data which will causes your logs to quickly rotate out if you are not forwarding to a remote syslog server. There's also additional overhead cost for this type of logging and more importantly, it may not capture all of the required data. This is an approach that some customers have tried but is not really a practical solution.

This topic has always been interesting to me and with several recent inquiries from the field, it got me thinking about this area again. While working on a completely different project, I ended up on Luc Dekens awesome blog and came across his Events Part 3: Auditing VM Device Changes article. If you take a look at the article, you will see that Luc shows you how you can easily audit changes to a VM's devices (e.g. Virtual Disk, CD-ROM, etc). What Luc demonstrated in his script is just a specific type of configuration, but the point is that this type of information has always been available, just not easily consumable.

The secret is to key off of the VmReconfiguredEvent which includes a configSpec property that captures the exact set of configuration changes for a given VM. Below is an example of the configSpec dump of one of these events. We can clearly see that this VM had its vCPUs modified to 4 and its vMEM modified to 20GB.

With this information, we can now easily query the configuration changes for a given VM by looking through its past events. Leveraging the awesome work that Lud has already done with his script, I slightly enhanced it to cover more than just device changes but overall VM configuration changes. With that, here is a PowerCLI script/function that I created called Get-VMConfigChanges.ps1

Note: The amount of historical events that you will be able to search through will purely depend on your Center Server DB's retention period of Tasks/Events. For VMs which have been deleted, you will not be able to retrieve any events as they must be associated with an object in the database.

Below is an example of how to use this function which accepts a VM object and the number of hours (default to 8) to search through the VM's events:

$vcserver = "192.168.1.150"
$vcusername = "*protected email*"
$vcpassword = "VMware1!"

Connect-VIServer -Server $vcserver -User $vcusername -Password $vcpassword

$vm = Get-VM "Test-VM"

Get-VMConfigChanges -vm $vm -hours 12

Disconnect-VIServer -Server $vcserver -Confirm:$false

From the output below, we can clearly see the following configuration changes have been applied:

Change vCPU to 2
Change vMEM to 4GB
Change vMEM to 5GB and Edited Virtual Disk (you can of course get further details by dumping more information)

Although this solution is not as clean as the vRealize Air Compliance offering, it does allow anyone to quickly pull out the relevant configuration changes for a given VM along with the user and time the configurations was performed. Hopefully this goes to show how powerful the vSphere Platform APIs really are and it is definitely worth while in learning how they work.