In case you may not be aware, Intel recently notified VMware that certain Intel Broadwell and Haswell CPUs are affected by Intel Sighting after applying the latest microcode update to remediate against the Spectre vulnerability. VMware has published the following KB 52345 which provides more details on the affected Intel CPUs along with the recommended workaround in case you have already applied the latest ESXi patches containing the faulty microcode. I highly recommend you carefully read over the KB before, even if you have not applied the ESXi patches proceeding further.
With this updated news, I have also updated my existing Spectre verification script (found here) to include the additional Intel Sighting information which can help customers easily identify whether they have ESXi hosts that are impacted. In this article, I will provide a solution to help automate the deployment of the Intel Sighting remediation as outlined in the KB article, but unlike the manual steps outlined in the KB, SSH access to the underlying ESXi host will NOT required.
Step 1 - Download the VerifyESXiMicrocode.ps1 PowerCLI script as well as the new PowerCLI script IntelSightingWorkaround.ps1
Step 2 - Run the "Verify-ESXiMicrocodePatch" function against a specific vSphere Cluster or ESXi host to determine if you are impacted by the Intel Sighting issue. Below is a screenshot for a system which is affected by Intel Sighting and we can determine this by seeing a value of "True" under the IntelSighting column.
Step 3 (Optional) - This step is optional, but I wanted to demonstrate how you can tell whether the Intel Sighting workaround has been applied correctly. You can use the "Verify-ESXiMicrocodePatchAndVM" function which provides information from a Virtual Machine's perspective and whether the new CPU instructions are exposed to the VM. In the screenshot below, I have a Test VM called TestVM-03 that has been powered on and as you can see, the three new CPU instructions (IBR,IBPB and STIB) are present as I have not applied the Intel Sighting workaround.
Step 4 (Optional) - This step is also optional, but I wanted to demonstrate how you can quickly check the contents of /etc/vmware/config without requiring SSH access or direct ESXi Shell access. You can use the "Get-Esxconfig" function and specify an ESXi host to query. The screenshot below confirms that we have not deployed the Intel Sighting workaround.
Step 5 - Once we have identified the ESXi hosts that are impacted by Intel Sightin (See step 2), we need to create a text file that contains the Hostname/IP of ESXi hosts (one on each line) that we wish to remediate. This will then be used as an input to the remediation function. In the example here, I have a single host to remediate and have added its name to a file that I have called "affected_hosts".txt (it can be named anything).
Note: The remediation does NOT require the ESXi host to reboot for the changes to go into effect, but you may still want to consider following standard procedures of putting the host into Maintenance Mode if you wish to control when VMs will see the masked instruction which will require a VM power cycle (OS restart is not sufficient).
To apply the remediation, you will use the "Set-IntelSightingsWorkaround" function which has a single mandatory parameter called AffectedHostList. Below is a screenshot of running the function and the remediation is fairly quick as it is merely appending a single string to the /etc/vmware/config configuration file. After the remediation has completed, we can then use the "Get-Esxconfig" function to confirm that we have added the masked CPU instruction to the ESXi host as shown in screenshot below.
Step 6 - As the KB mentions, for the changes to go into affect, you will need to perform a complete power cycle of your VMs before they can see the new masked CPU instructions. If you recall earlier, I had a VM called TestVM-03 which saw the three new CPU instructions. If we power cycle the VM and re-run the "Verify-ESXiMicrocodePatchAndVM" function against our VM, we should now see that the three CPU instructions have been properly masked away as shown in the screenshot below.
When it is time to remove the workaround, the process is also very straight forward as well. You can use the "Remove-IntelSightingsWorkaround" function which also accepts a list of ESXi hosts to remediate. Once the remediation has completed and you wish to expose the new CPU instructions to the VMs, you will also need to power cycle the VMs for CPU instructions to be unmasked.
Yet another reason why you deserve being elected the No.1 blog at http://thevpad.com/
Thanks, please note that the "Set-IntelSightingsWorkaround" function only functions properly when connected to a single vCenter server at a time.
What's being returned by Verify-ESXiMicrocodePatchAndVM? I'm trying to do some filtering, but I'm having a hard time filtering on VM names or anything else..
This is what I'm trying to do:
Verify-ESXiMicrocodePatchAndVM -ClusterName | Where-Object { $_.VM -eq "ABC123" }
.. but this gives nothing as a result 🙁
I solved this, had to remove "| ft" from the script, and then I could play around with the objects as I intended 🙂
Thank you so much!!! this script saved me a lot of time!
Hi
Verify-ESXiMicrocodePatch
$tmp = [pscustomobject] @{
VMHost = $vmhostDisplayName;
CPU = $cpuModel;
IBRPresent = $IBRSPass;
IBPBPresent = $IBPBPass;
STIBPresent = $STIBPPass;
HypervisorAssistedGuestAffected = $vmhostAffected;
IntelSighting = $intelSighting;
}
As per above code there is no "Affected" value. I am getting below output, please check and confirm if "HypervisorAssistedGuestAffected" is equivalent to "Affected".
VMHost CPU IBRPresent IBPBPresent STIBPresent HypervisorAssistedGuestAffected IntelSighting
------ --- ---------- ----------- ----------- ------------------------------- -------------
esxi01 Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz False False False True False
Yes. Since the scope of the script has been expanded to also cover Intel Sighting, the "Affected" column might be confusing and I had recently made that change. Let me update the article to cover that note.
Thanks buddy. This worked great.
Thanks very much for this. I have the following results on what I believe to be a patched system and VM, can someone confirm this is the expected result please. I am still receiving speculation control failures from the guest owner.Thanks
hostname Intel(R) Xeon(R) CPU E5-26... , True, True, True, False, True,
vmname False, False, False, vmx-11, True
Hi William,
If it's only a handful of hosts, would it also be possible to undo the patch by simply booting to the alternate bootbank of ESXi?
Applying/Removing the workaround does NOT require an ESXi host to reboot. If you're going to boot the altbootbank, you're now forcing an unnecessary reboot ... doesn't seem to make a ton of sense if you ask me
You're right, because the microcode was already patched of course. I misunderstood what the patch effectively did, sorry for that. All clear now.
William,
I'd like to point out that restoring the altbootbank did actually bring the host back to its previous state where the microcode isn't loaded when the hypervisor boots, since the microcode isn't actually flashed to the CPU but loaded before ESXi boots.
The only disadvantage is that VM's on hosts that have the microcode patch applied, can't be vMotioned to a host that had it's previous bootbank restored, because it's missing several CPU instructions. You need to shut down your VM's and then vMotion them back to your restored host.
This might a bit more laborious, but if it's only a smaller environment this method would prevent you from having to remove this setting after a proper patch has been released.
I double checked this with your script and didn't see the sightings after restoring the bootbank.
Is there an easy way to script a powercycle on next reboot? Or use vrealize orchestrator.
They only way I have found so far is to use vrealize orchestrator and use the upgrade vmx version on next boot.
Can anyone tell me why the script is not spitting the information out on powershell? I am connected to the host and or vcenter it still doesnt want to pull data from the hosts? why is this happening to me? how come you guys are not having the issue??
The script did not work on me so i had to issue Import-Module .\VerifyESXiMicrocodePatch.ps1
I have zero knowledge on script.
I am trying to run set-interlsigtingsworkaround but it does not work.
Is there a reason simply uninstalling the vib isn’t good enough? I recently patched two hosts after learning the patch was being pulled. So I did a vib uninstall on the esxi host. Is that not sufficient?
Please follow the guidance as outlined in the KB
My specific question isn’t answered in the KB, and VMware support has been unable to provide a concise answer which is why I took a stab at asking you. Seemed like perhaps you were in touch work folks that are in the know. Thanks anyway!
Has anyone encountered any performance issues after masking the registers? This is my biggest concern as we use our systems to the edge with intensive cpu usage. The last thing I need is the systems running slower.. I don't see anyone commenting on this possiblity.
Will the changes made by this script need to be reversed after Intel releases their updates?
William,
I just got off the phone with VMWare support who said that these changes are no longer needed and the updates can be installed without issue, can you confirm? He sighted this article as proof but I'm not sure if I understand: https://www.vmware.com/us/security/advisories/VMSA-2018-0002.html
According to the support tech as long as your build number is above 7388607 then this problem is not applicable. The host I just updated shows build 7526125 and I can see both ESXi650-201801402-BG and ESXi650-201801401-BG are both installed in VUM. You seem to be the authority on this patch (according to Google) so I wanted to check with you before taking him at his word.
thanks,
Jonathan