WilliamLam.com

ESXi 4.1 - Major Security Issue

by // 11 Comments

On Thursday July 15th, a user raised a question on the VMTN forums regarding an ESXi 4.1 password issue. The problem was described as the following:

Hi all
It seems that authentication only requires the first 8 characters to be correct. My root password is 11 characters long, but so long as the first 8 characters are correct, I can put whatever I like after that and it still authenticates me. Tested this on three ESXi boxes, all running 260247 (release)

I performed a few quick tests to validate the user's claim and in fact this was the case with a new installation of ESXi 4.1. Though in my lab, I had upgraded from ESXi 4.0 Update 2 to ESXi 4.1 and I was not able re-produce the issue. I then tried to reset my password to the same initial password and to my surprise, I hit the exact problem as described by the user. I notified VMware of the issue and was told that they were looking into the problem. As of today, there is still not official word from VMware on whether this is a bug or an expected behavior.

VMware does have a KB article (KB1012033) documenting both ESX and ESXi password requirements, by default a minimum of 8 characters is required but can be changed by modifying the PAM module pam_passwdqc.so which checks for the quality of a password.

Further looking into the issue over the weekend, I have found the following:

  • pre ESXi 4.1 (e.g. ESXi 3.5, 4.0, 4.0u1, 4.0u2) does not have this password issue
  • password issue affects ALL methods of authentication to an ESXi host: vSphere Client, local and remote Tech Support Mode, /bin/login, SSH localhost and vSphere MOB. (did not get a chance to validate AD logins)
  • password issue will NOT affect hosts that were upgraded to ESXi 4.1, unless you change or update your password afterwards, in which case only the first 8 characters will be checked

During my testing, I provisioned a newly built ESXi 4.0 Update 2 (called esxi4-4) and ESXi 4.1 (called esxi4-3) and used a modified SSH login expect script to run my tests. You can download my modified version of the script here. Both hosts had their password created using the DCUI and was set to "VMware123"

Here is an example of SSHing to both host and running "vmware -l" command to show that hosts accepts the password that was created:

As you can see, I am able to login to both hosts using the defined password of "VMware123" and displaying the output of "vmware -l" command.

In the next example, I will purposely provide the wrong password of "VMware12" which should fail:

This should have failed for both hosts, but did not in the case of ESXi 4.1 and still allowed me to login. If you tried to login to the ESXi 4.1 host using "VMware1", access is denied. Basically, if you have a password that is greater than 8 characters, only the first 8 characters are validated and any combination there after is accepted! This in my mind is a big security hole, especially if this is an unexpected behavior that users are not made aware of.

I also performed one additional test by modifying the default password requirement from 8 characters to 9 by editing /etc/pam.d/system-auth:

I then used the passwd utility to create a new root password, I attempted to use the password "VMware12" which should fail as it does not meet the minimum password requirement, which it does:

Once I used password that was 9 characters or greater, the utility allowed me to set the new password:

From what I can understand, the password in fact is being set properly and the rules that govern the length are being regulated. The problem seems to be with the authentication module where it is validating the password and only checking for the first 8 characters. To me this sounds like a authentication bug and one that can severely impact any security policies put in-place for an organization.

I hope that VMware provides either a clarification on the issue and potentially a fix or patch for this problem. In the meantime, if you have security policies in your environment pertaining to password requirements and plan on upgrading to ESXi 4.1, ensure you do not change the password. If you do, change it before the upgrade.

UPDATE1:

It looks Didier Pironet found the solution and documented the steps in his follow-up blog post. I am still hoping VMware will address this security issue as soon as possible and provide a patch. The above work around does resolve the problem but the change is not persisted through a reboot without manual hacks to force a backup of PAM configuration file.

UPDATE2:

VMware just released KB article regarding the issue: KB1024500

UPDATE3:

VMware has just updated the KB article KB1024500 with slightly more details pertaining to the solution of the problem. They still have not confirmed the bug that is identified in the PAM module where DES encryption is used over MD5, but can be assumed via the solution provided. The problem actually exists in both vSphere ESX and ESXi 4.1 and the KB includes a fix to both including a way to persist the changes on ESXi. The security issue apparently is not important enough to get an immediate patch but may eventually get a permanent patch sometime in the future.

UPDATE4: VMware just first the first patch set (p01) for both ESX and ESXi 4.1 that resolves this security issue along with other bugs that have been identified since the release of vSphere 4.1. Here is the specific KB article regarding the fix - http://kb.vmware.com/kb/1027024 and here is the link to the details p01 patch set - http://kb.vmware.com/kb/1027027 As with anything, please test and validate in your test environment prior to rolling out to production.

How to get vSphere 4.1 to work with CapacityIQ 1.0.3 using unsupported method

by // 5 Comments

Part of my process before upgrading to new software in a production environment, whether it's a major or minor release, is to let it bake in a development lab. While finishing up the upgrade of my vSphere 4.1 lab, I had one additional component to install, CapacityIQ. To my surprise (but not really), it is incompatible with vSphere 4.1.

CapacityIQ is distributed as a VMware virtual appliance that runs on CentOS 5.2. As part of the installation you are asked to configure both a password for both the root and ciqadmin account. Since this is just a linux host, I decided to SSH in and poke around to see where this check might be taking place. It only took a few minutes using the linux command find to locate some Perl modules that seem to be in the very early stages of a client side stub to a CapacityIQ API.

There is a Perl module called CIQExtension.pm which actually does the validation of your vCenter Server build. Within the script, there are two hardcoded values 2.5.0 and 4.0.0 which are the version that are supported by CapIQ.

What I did was add an extra "OR" statement to allow for vCenter Server 4.1.0:

Once I saved the changes, I went back to the CapIQ admin page to try to re-register my vCenter 4.1 and voila, it fully registered and started CapIQ!

I also verified by checking the status tab:

At this point, I was optimistic that the change has worked and decided to login to vCenter server to verify by loading the CapIQ application:

There you have it! CapacityIQ 1.0.3 running on vSphere 4.1

Even though there are new performance statistics with the release of vSphere 4.1, these stats will most likely not be used or seen by CapIQ, which leads me to believe that this should work. However, I would not recommend doing this in a live production environment without consulting with VMware first. If you do work for VMware or part of the CapacityIQ team, I would love to know if this hack works or do we just need to wait for another release of CapIQ?
 
I was aware of the VMware View Issue but I was not able to find any documentation or KB article documenting this compatibility issue with the latest version of CapacityIQ 1.0.3. This tends to happen more often where there is a disconnect between the core VMware products (ESX, ESXi, vCenter) and the rest of the auxiliary products that VMware provides. Sometimes, I wonder if the various product teams actually talk to each other with regards to integration and compatibility.

UPDATE: 

While browsing around, I also found an undocumented Perl script called ciq-admin.pl which can also be used to administrator and configure your CapacityIQ. Using the utility, you can register your vCenter with CapIQ and there is a hidden flag which allows you to bypass the version check. This also allows you to get vSphere 4.1 working with CapIQ without having to manually edit any files.

To use this utility, you must login as the ciqadmin user. If you are logged in as root, you can just switch user.

The syntax will be the following:

/usr/lib/ciq/ciq-admin.pl register --vc-server [server] --force --user [username] --password [password] --noversioncheck

Here is an example of registering vSphere 4.1 vCenter using the CLI:

You can verify on the command line that operation is successful by running the status command, which is the same status tab in the CapIQ admin page :

New way of enabling and disabling services using vSphere 4.1

by // 2 Comments

While checking out the PlanetV12n feed, I noticed a new video from David Davis about the new vSphere 4.1 Tech Support Mode. In the short video, David goes over the new method of enabling "hidden" unsupported Busybox Console, also known as Tech Support Mode. In the past, you had to be on the console of your ESXi host, type ALT+F1, and then "unsupported" to gain access. Once in, to enable remote SSH access or Remote Tech Support Mode, you had to edit /etc/inetd.conf and restart inetd service. This was pretty tedious if you needed access for a short period of time. In the video, David goes over the new method showing how it can be done using the DCUI and the old method is no longer required.

What surprised me after watching the video was that he did not mention the other method of enabling and disabling Tech Support Mode both local and remote. One issue I had with past releases of ESXi is that you could restart some services such as ntp or vmware-vpxa via the vSphere API, but others were just not available. In vSphere 4.1, VMware introduces a few new services around their Likewise Active Directory integration but also includes controlling both local and remote Tech Support Mode as well as DCUI itself.

These services can be enabled and disabled using the vSphere Client, here is a screenshot:

To enable or disable TSM, just click on the service and then click on options:

You will then have the option to configure the startup policy including enabling or disabling the service:

If you needed to perform this operation against one or two host, it is not that big of a deal. Though if you needed to enable remote Tech Support Mode (SSH access) across few dozen hosts, then can still be tedious. Luckily I wrote a script (hostServiceMangement.pl) last year that allowed you to enable and disable supported services using the vSphere API. Without any modifications, it supports vSphere 4.1 and can take advantage of the new services that are available for control.

Here is an example of listing the services on an ESXi 4.1 host:

Here is an example enabling remote Tech Support Mode:

Here is an example of disabling remote TSM:

The script can be executed on a host that has vCLI 4.1 installed or on vMA 4.1 and can bulk update a list of ESX/ESXi host or individual host. For more details, please check out the documentation for hostServiceManagement.pl.