WilliamLam.com

Ghetto Groups

by // 1 Comment

Back in December, VMware upgraded their VMTN (VMware Technology Network) forum software Jive and introduced a completely new layout of the forums that would hopefully enhance the user experience. Though it brought many new features, it also brought on several new issues. The one bug that affected me was the incorrect conversion of the ghettoVCB document, because the conversion was unsuccessful it was decided to be left alone until the issue could be resolved. If you visited the document, it would display a "Forbidden" error message. Unfortunately due to the time it took to resolve, even Google cache started to get stale and stopped serving the cached contents.

Luckily, with the help of Alex Maier (VMTN Community Manager) and her team, she was able to get the ball rolling and got the fix tested and rolled out to production. The ghettoVCB document is once again alive and hopefully in no time it will be returned as the first search result on various search engines.

Going through the pain of receiving dozen of emails, private messages, tweets, etc. per week regarding the issue, I came to realize that VMTN document itself was not the right medium to host both content and user discussions. As it stands today, there are over 1,100+ comments which is pretty significant and managing and keeping up with the conversations is a pretty daunting task. I enjoy the feedback that community provides and the collaboration that takes place and I realize that this can be solved by using the new Groups feature.

To be honest, I did not spend much time looking at Groups when the VMTN software was upgraded, but now that the ghettoVCB document has been fixed, I realized this would fit this need perfectly. With the help of categories users can now post their feedback, discussions/issues and feature request and it can be easier consumed by both new users and myself. Starting today, I will have the following groups based on the top 5 most popular and active scripts:

I have also disabled any new comments on these VMTN documents and will ask that all new comments be re-directed to respective VMware Groups. I'm currently working with Alex to see if there is an easy way to convert the existing comments into a document and attached that as a download to help minimize the complexity of the document. In the worse case, the comments will be left alone as read-only as I think the discussion that currently exists are invaluable. All other VMTN documents that I maintain in the vGhetto Repository will continue to use comments and depending on how well the groups go, I may migrate those over as well.

I hope these new groups will be beneficial for everyone and I am looking forward to the collaboration. Thanks for your support and please help spread the word!

Updated vSphere Health Check & Security Hardening Report Scripts

by // 1 Comment

Check out the minor update to the popular vSphere Health Check Script, now at v4.1.5, there was a minor typo issue that has now been resolved.

The vSphere 4.0 Security Hardening Guide has gotten a few updates from VMware since it's initial draft released back in January this year. The latest version includes a few revisions and my vSphere Security Hardening Report script v0.2 has not gotten an update since it's first release four days after VMware's draft. I finally found some time over the holiday to update the script and incorporate the feedback I received throughout the year. There are some new features and bug fixes and you can take a look at the change log more details. I encourage you to check out the new 1.0 version here.

Here is an updated sample report.
vmwarevSphereSecurityHardeningReport-SAMPLE.html

If you run into any issues or would like to report any bugs, please leave a message in the appropriate VMTN document.

ESXi 4.1 - Major Security Issue

by // 11 Comments

On Thursday July 15th, a user raised a question on the VMTN forums regarding an ESXi 4.1 password issue. The problem was described as the following:

Hi all
It seems that authentication only requires the first 8 characters to be correct. My root password is 11 characters long, but so long as the first 8 characters are correct, I can put whatever I like after that and it still authenticates me. Tested this on three ESXi boxes, all running 260247 (release)

I performed a few quick tests to validate the user's claim and in fact this was the case with a new installation of ESXi 4.1. Though in my lab, I had upgraded from ESXi 4.0 Update 2 to ESXi 4.1 and I was not able re-produce the issue. I then tried to reset my password to the same initial password and to my surprise, I hit the exact problem as described by the user. I notified VMware of the issue and was told that they were looking into the problem. As of today, there is still not official word from VMware on whether this is a bug or an expected behavior.

VMware does have a KB article (KB1012033) documenting both ESX and ESXi password requirements, by default a minimum of 8 characters is required but can be changed by modifying the PAM module pam_passwdqc.so which checks for the quality of a password.

Further looking into the issue over the weekend, I have found the following:

  • pre ESXi 4.1 (e.g. ESXi 3.5, 4.0, 4.0u1, 4.0u2) does not have this password issue
  • password issue affects ALL methods of authentication to an ESXi host: vSphere Client, local and remote Tech Support Mode, /bin/login, SSH localhost and vSphere MOB. (did not get a chance to validate AD logins)
  • password issue will NOT affect hosts that were upgraded to ESXi 4.1, unless you change or update your password afterwards, in which case only the first 8 characters will be checked

During my testing, I provisioned a newly built ESXi 4.0 Update 2 (called esxi4-4) and ESXi 4.1 (called esxi4-3) and used a modified SSH login expect script to run my tests. You can download my modified version of the script here. Both hosts had their password created using the DCUI and was set to "VMware123"

Here is an example of SSHing to both host and running "vmware -l" command to show that hosts accepts the password that was created:

As you can see, I am able to login to both hosts using the defined password of "VMware123" and displaying the output of "vmware -l" command.

In the next example, I will purposely provide the wrong password of "VMware12" which should fail:

This should have failed for both hosts, but did not in the case of ESXi 4.1 and still allowed me to login. If you tried to login to the ESXi 4.1 host using "VMware1", access is denied. Basically, if you have a password that is greater than 8 characters, only the first 8 characters are validated and any combination there after is accepted! This in my mind is a big security hole, especially if this is an unexpected behavior that users are not made aware of.

I also performed one additional test by modifying the default password requirement from 8 characters to 9 by editing /etc/pam.d/system-auth:

I then used the passwd utility to create a new root password, I attempted to use the password "VMware12" which should fail as it does not meet the minimum password requirement, which it does:

Once I used password that was 9 characters or greater, the utility allowed me to set the new password:

From what I can understand, the password in fact is being set properly and the rules that govern the length are being regulated. The problem seems to be with the authentication module where it is validating the password and only checking for the first 8 characters. To me this sounds like a authentication bug and one that can severely impact any security policies put in-place for an organization.

I hope that VMware provides either a clarification on the issue and potentially a fix or patch for this problem. In the meantime, if you have security policies in your environment pertaining to password requirements and plan on upgrading to ESXi 4.1, ensure you do not change the password. If you do, change it before the upgrade.

UPDATE1:

It looks Didier Pironet found the solution and documented the steps in his follow-up blog post. I am still hoping VMware will address this security issue as soon as possible and provide a patch. The above work around does resolve the problem but the change is not persisted through a reboot without manual hacks to force a backup of PAM configuration file.

UPDATE2:

VMware just released KB article regarding the issue: KB1024500

UPDATE3:

VMware has just updated the KB article KB1024500 with slightly more details pertaining to the solution of the problem. They still have not confirmed the bug that is identified in the PAM module where DES encryption is used over MD5, but can be assumed via the solution provided. The problem actually exists in both vSphere ESX and ESXi 4.1 and the KB includes a fix to both including a way to persist the changes on ESXi. The security issue apparently is not important enough to get an immediate patch but may eventually get a permanent patch sometime in the future.

UPDATE4: VMware just first the first patch set (p01) for both ESX and ESXi 4.1 that resolves this security issue along with other bugs that have been identified since the release of vSphere 4.1. Here is the specific KB article regarding the fix - http://kb.vmware.com/kb/1027024 and here is the link to the details p01 patch set - http://kb.vmware.com/kb/1027027 As with anything, please test and validate in your test environment prior to rolling out to production.