VMware recently published a new knowledge base (KB) article 52085 that outlines instructions for enabling the Hypervisor-Assisted Guest Mitigation (CVE-2017-5715), also known as the Spectre vulnerability. This KB also provides steps to verify the updated microcode (included in the ESXi patch) has been applied along with Virtual Machine verification (those applicable) to ensure that they are seeing the new CPU features. While following the KB and patching one of my vSphere environments, I had noticed the verification steps was not only manual but it also to difficult to scale beyond a few VMs as it required customers to look for a specific set of strings from within the vmware.log file which is generated for each powered on VM, which could easily be several hundreds or thousands of VMs.
I figured there had to be a better way to help customers automate this at scale and remove the human factor. The other reason I was not fond of the current method is that it would require customers to either enable ESXi Shell/SSH access or to manually or through automation to download every single vmware.log file to inspect for specific log entries which can take quite a bit of time for any sizable environment. I had an idea on how this could be done without having to look at the vmware.log file while leveraging our vSphere APIs and did some investigation. Before proceeding further, please familiarize yourself with KB 52085. For complete background on both Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) as it relates to all VMware products, please carefully read through this top level KB 52245 which is being updated as new information is available, so you may want to subscribe to the KB for all the latest updates.
UPDATE 4 (01/23/18) – VMware has just updated KB 52345 with the current list of Intel CPUs affected by Intel Sightings. I have also updated my script to reflect all these changes. Make sure to download the latest version to ensure you have the latest changes.
UPDATE 3 (01/16/18) – I have just enhanced the script further to also collect the current microcode version running on a given ESXi host. Unfortunately, this information can only be collected when SSH is enabled and is something a user must explicitly allow. The benefit here is that Intel Sighting impact reporting is more robust as the code now checks for both impacted CPU signature as well as the microcode affected by Intel Sighting as outline in KB 52345. See below for more details.
UPDATE 2 (01/14/18) – As mentioned in the last update, I have been working on a Intel Sighting remediation script which can help customers automate the temporary workaround as recommended in KB 52345. Please see this blog post for complete details.
UPDATE 1 (01/13/18) – VMware just published a new KB 52345 outlining certain Intel Broadwell and Haswell CPUs being affected by Intel Sightings after applying latest microcode updates. Please refer to the KB for the complete details. I am currently working on a script to help with the remediation as the remediation method outlined in the KB would require customers to enable SSH access and manually update /etc/vmware/config. In the meantime, I wanted to provide a way for customers to easily identify whether their system are affected by Intel Sightings and thanks to community member Adam Robinson who just submitted a patch this morning to update my existing script to include these details. I have also added the CPU model to the output as additional information.
Note: This script only provides information about your existing vSphere environment, it does not make any changes or provides any remediation steps, please follow the KB for those instructions.
The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and it contains the following two functions:
The first function verifies both ESXi hosts and VMs and provides the output from the VM's point of view. The script will ensure VMs are at least vHW9+ (as older vHW are not applicable), powered on and that one of the three new CPU features are available as outlined in the KB. If the HypervisorAssistedGuestAffected column for a given VM row shows false, it means BOTH the microcode + ESXi patch has been applied and that the VM has gone through a complete power cycle to see the new CPU features. The second function only verifies that the ESXi microcode has been applied (this could have come a hardware vendor BIOS/Firmware update) but as mentioned earlier, it is also bundled within the ESXi patch. Similar to the first function, it also verifies that one of the three new CPU features are exposed to the ESXi host and you will also see an HypervisorAssistedGuestAffected column for each ESXi host. For columns which show true, please refer back to the KB above and follow the remediation procedure. In addition to information related to KB 52085, the second function has also been recently enhanced (see Update #1, 2 & 3 above) which include details about Intel Sighting. You can find more details below.
The Verify-ESXiMicrocodePatchAndVM function can run in one of three ways. If you do not pass in any parameters, then it will query all VMs for a given vSphere environment whether that is connecting to a vCenter Server or directly to a specific ESXi host. You have the ability to limit the scope by providing a specific vSphere Cluster by using the -ClusterName parameter or you can check a particular VM by using the -VMName parameter. Below is a screenshot exercising the three different methods:
The Verify-ESXiMicrocodePatch function can also be run in one of three ways. If you do not pass in any parameters, then it will query all ESXi hosts for a given vSphere environment when connected to a vCenter Server or a specific ESXi host if you are directly connected to a host. You have the ability to limit the scope by providing a specific vSphere Cluster by using the -ClusterName parameter or you can check a particular ESXi host by using the -VMhostName parameter. Below is a screenshot exercising the three different methods:
The value of IntelSighting can contain four potential values:
- True = ESXi host contains microcode update is affected by Intel Sighting, you will need to apply the workaround as outlined in KB 52345
- AffectedOncePatched = CPU is affected by Intel Sighting, but does not need the work around unless it has been patched or has a BIOS update that includes the fault microcode
- False = CPU not affected by Intel Sighting it is currently recommended to only apply one of the ESXi patches (until Intel provides a microcode update fix), please refer to KB 52345 for full details
- N/A = CPU is not Intel
The "default" method in how the script identifies whether an ESXi host is affected by IntelSighting is by comparing against the known CPU signatures as outlined in KB 52345 which can be done so using the vSphere API. Although this method is sufficient at this time since no updated microcodes have not been released (as far as I know), the results may be incorrect when new microcodes have been released and applied to remediate against Intel Sighting. A more robust solution would be to verify BOTH the affected CPU signature and the microcode version that are known to be impacted by Intel Sighting which is also available in the KB. The main issue from an Automation standpoint is that to be able to retrieve the microcode version that is currently deployed on an ESXi host, you would need to either enable ESXi Shell and/or SSH to system and use the vsish interface as this information is currently not available through a remote vSphere API.
Here is the vsish command you can manually run to retrieve the current microcode version:
vsish -e cat /hardware/cpu/cpuList/0 | grep "Current Revision:"
Since information can not be retrieved through a remote API, I have decided to enhance the function for those customers who wish to collect this information and have SSH enabled (can just be for the duration of running the script). To do so, I have introduced four optional parameters that must be provided if you wish to retrieve the microcode version:
- IncludeMicrocodeVerCheck – By default, this is set to $false, you will need to set to $true
- PlinkPath – You will need to provide the full path to the Plink executable which I use as an SSH client to perform the remote operation
- ESXiUsername – The username to login to ESXi host and perform the operation, this would normally be root
- ESXiPassword – The password for the ESXi username provided
Note: If you enable microcode version retrieval and SSH is disabled on the ESXi host, then you will see the microcode column have a value of SSHNeedsToBeEnabled. You can easily mass enable/disable SSH access for your ESXi hosts using PowerCLI. Here is an example blog post which shows you how.
Here is a screenshot of an ESXi host which is impacted by IntelSighting which the script has been able to determine by checking BOTH the CPU signature and the microcode version (which is now available as part of the output) as seen below.