security

Automating Intel Sighting remediation using PowerCLI (SSH not required)

01.14.2018 by William Lam // 22 Comments

In case you may not be aware, Intel recently notified VMware that certain Intel Broadwell and Haswell CPUs are affected by Intel Sighting after applying the latest microcode update to remediate against the Spectre vulnerability. VMware has published the following KB 52345 which provides more details on the affected Intel CPUs along with the recommended workaround in case you have already applied the latest ESXi patches containing the faulty microcode. I highly recommend you carefully read over the KB before, even if you have not applied the ESXi patches proceeding further.

With this updated news, I have also updated my existing Spectre verification script (found here) to include the additional Intel Sighting information which can help customers easily identify whether they have ESXi hosts that are impacted. In this article, I will provide a solution to help automate the deployment of the Intel Sighting remediation as outlined in the KB article, but unlike the manual steps outlined in the KB, SSH access to the underlying ESXi host will NOT required.

Step 1 - Download the VerifyESXiMicrocode.ps1 PowerCLI script as well as the new PowerCLI script IntelSightingWorkaround.ps1

Step 2 - Run the "Verify-ESXiMicrocodePatch" function against a specific vSphere Cluster or ESXi host to determine if you are impacted by the Intel Sighting issue. Below is a screenshot for a system which is affected by Intel Sighting and we can determine this by seeing a value of "True" under the IntelSighting column.

Step 3 (Optional) - This step is optional, but I wanted to demonstrate how you can tell whether the Intel Sighting workaround has been applied correctly. You can use the "Verify-ESXiMicrocodePatchAndVM" function which provides information from a Virtual Machine's perspective and whether the new CPU instructions are exposed to the VM. In the screenshot below, I have a Test VM called TestVM-03 that has been powered on and as you can see, the three new CPU instructions (IBR,IBPB and STIB) are present as I have not applied the Intel Sighting workaround.

Step 4 (Optional) - This step is also optional, but I wanted to demonstrate how you can quickly check the contents of /etc/vmware/config without requiring SSH access or direct ESXi Shell access. You can use the "Get-Esxconfig" function and specify an ESXi host to query. The screenshot below confirms that we have not deployed the Intel Sighting workaround.

Step 5 - Once we have identified the ESXi hosts that are impacted by Intel Sightin (See step 2), we need to create a text file that contains the Hostname/IP of ESXi hosts (one on each line) that we wish to remediate. This will then be used as an input to the remediation function. In the example here, I have a single host to remediate and have added its name to a file that I have called "affected_hosts".txt (it can be named anything).

Note: The remediation does NOT require the ESXi host to reboot for the changes to go into effect, but you may still want to consider following standard procedures of putting the host into Maintenance Mode if you wish to control when VMs will see the masked instruction which will require a VM power cycle (OS restart is not sufficient).

To apply the remediation, you will use the "Set-IntelSightingsWorkaround" function which has a single mandatory parameter called AffectedHostList. Below is a screenshot of running the function and the remediation is fairly quick as it is merely appending a single string to the /etc/vmware/config configuration file. After the remediation has completed, we can then use the "Get-Esxconfig" function to confirm that we have added the masked CPU instruction to the ESXi host as shown in screenshot below.

Step 6 - As the KB mentions, for the changes to go into affect, you will need to perform a complete power cycle of your VMs before they can see the new masked CPU instructions. If you recall earlier, I had a VM called TestVM-03 which saw the three new CPU instructions. If we power cycle the VM and re-run the "Verify-ESXiMicrocodePatchAndVM" function against our VM, we should now see that the three CPU instructions have been properly masked away as shown in the screenshot below.

When it is time to remove the workaround, the process is also very straight forward as well. You can use the "Remove-IntelSightingsWorkaround" function which also accepts a list of ESXi hosts to remediate. Once the remediation has completed and you wish to expose the new CPU instructions to the VMs, you will also need to power cycle the VMs for CPU instructions to be unmasked.

Quick Tip - Pre-filled credentials in the vSphere 6.0 Web Client

08.24.2015 by William Lam // 17 Comments

This past weekend I was finishing up a couple of demo recordings for my VMworld sessions in case the live demos fail for whatever reason, which has happened to me in the past. A few of the demos involve the vSphere Web Client UI and I thought instead of wasting time and potentially fat fingering credentials up on stage, I would try to do everything I can to remove any potential hiccups. In vSphere 6.0, the vCenter Single Sign-On page is now completely in HTML and this not only means you can customize the UI as I have shown here but you can also do some other neat tricks with it.

I decided to update the HTML page to automatically pre-fill both the SSO username and password, so that when I need to login to the vSphere Web Client, I just have to hit the tab key and then click on the login button.

Disclaimer: Outside of a home lab or demo purposes, there is really no good reason for this. I can already hear Mike Foley sighing right now 😉 This also means that anyone who knows the address of your vSphere Web Client can just login, so you may want to only pre-fill the username and still type out the password in case you are concerned with that.

To pre-fill the value for the SSO username and/or password, you will need to edit the following file:

Windows VC: C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\webapps\websso\WEB-INF\views\unpentry.jsp
VCSA: /usr/lib/vmware-sso/vmware-sts/webapps/websso/WEB-INF/views/unpentry.jsp

For pre-filling the username, you will need to add a "value" property along with its actual value in the following section:

<input id="username" class="margeTextInput" type="text" value="*protected email*"/>

For pre-filling the password, you will need to add a "value" property along with its actual value in the following section:

<input id="password" class="margeTextInput" type="password" value="VMware1!"/>

Once you have saved your changes, you can then reload the browser and you should see that the vSphere Web Client now has both the username and password automatically pre-filled when the webpage loads.

Configuring a "Whitelist" for VM advanced settings in vCloud Director

05.10.2014 by William Lam // 3 Comments

The ability to preserve a Virtual Machine's "virtual hardware personality" during an export has been around since the release of vSphere and vCloud Director 5.1 also known as a Lossless OVF Export. However, when it comes time to import that Virtual Machine back into your environment, will it be a "lossless" import? For vSphere, the answer is yes as long as you click on the "Accept extra configuration options" during the import of your OVF/OVA in the vSphere Web Client.

For vCloud Director however, the answer is sadly no. If you try to import an OVF/OVA that contains additional VM advanced settings, vCloud Director will automatically strip away these parameters before importing the Virtual Machine. As annoying as this is for end users, there is a very good reason why this happens.

In a public cloud you do not know who your users are and whether they have malicious intentions. Since the "Extra Configuration" of a Virtual Machine can not be validated, the safest way to guard against this type of a security risk is by sanitizing the content prior to upload.

Although I have known about this default behavior in vCloud Director for some time now, it was only in a recent meeting with some of our internal folks managing our OneCloud environment did I realize how annoying this behavior was for "private" clouds as well. I was looking to import an existing OVF that contains several VM Advanced settings, but realized this would not be supported due to the sanitization of the OVF.

The other issue is that vApps within OneCloud are also exported/re-imported into other vCloud Director environments. Even if our Cloud Administrator were to make a one time exception and manually add these these VM Advanced Settings, they would automatically be stripped away the next time they were re-imported into another environment. After our meeting, I thought there had to be a better way! (at least I hope)

Disclaimer: This may or may not be officially supported by VMware, please use at your own risk.

In doing a quick bit of research, it turns out this is actually a pretty elegant solution that allows you to create a "Whitelist" of trusted VM advanced settings that would be accepted as part of the import process in vCloud Director. To configure this "Whitelist", you will need to add a new category called ExtraConfigWhitelist into the vCloud Director Database for each and every VM advanced setting that you wish to allow.

To demonstrate this functionality, I have the latest vCloud Director appliance (I believe this should also work on 5.1 as well) running and I would like to add the following VM advanced settings to the whitelist:

scsi0:0.virtualSSD
scsi0:1.virtualSSD
scsi0:2.virtualSSD
disk.enableUUID

Before starting, you will need to SSH to our vCloud Director instance and stop the VCD service by running the following command:

/etc/init.d/vmware-vcloud stop

Next, you will need to switch over to the "oracle" user by running the following command:

su - oracle

sqlplus vcloud/VCloud

Once you have logged in, you will need to insert the following rows into the "config" table of VCD Database:

INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:0.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:1.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:2.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'disk.enableUUID', 'true', 0);

Note: The syntax used here is for an Oracle database. You may need to tweak it if you are using MS SQL Server.

Finally, for the changes to go live you just need to start up your VCD instance:

/etc/init.d/vmware-vcloud start

Once the VCD service has returned, you can now import any OVF/OVA that contains a list of approved VM advanced settings and VCD will ensure these settings are preserved as part of the import. I hope this solution is something our OneCloud team will consider as it solves my immediate problem but I also think this can greatly benefit the overall use base, especially those building out content.

Note: Even though this "whitelist" option is available, I would still recommend you carefully determine which VM advanced setting is required for your environment and minimize the size of the list as much as possible from an operational perspective, especially when it comes to new installation or upgrades.