I was going through my twitter-feed this morning and came across an interesting article by @herseyc Locked out of the vCenter Server Virtual Appliance. I recommend you give Hersey's article a read as it contains some very useful information about failed login attempts to the VCSA (vCenter Server Appliance).
There was a question posed at the end of the article on how to increase the number of login attempts before an account would be locked out. The answer is to simply modify one of the PAM modules on the VCSA, specifically /etc/pam.d/common-auth
The system default for login attempts before locking out an account is 3 and you can modify this by changing the following line, where X is the number of attempts:
auth requisite pam_tally.so deny=X
You also have the option of specifying an "unlock" time which will lock the account for the specified period of time after reaching the max failed login attempts. This can useful if you do not want to manually reset a user account due to a user fat fingering a password. For more details about these parameters, you can search on Google or refer to this article.
Note: The login attempts here is specific to the OS system login on the VCSA (5.0 & 5.1) and not vCenter Server. If you successfully login before hitting the maximum attempts, the tally will automatically reset back to 0.
In vSphere 5.1, and with the use of vCenter SSO, you now have an easy way of controlling password and lock out policy using the new vSphere Web Client. Here is a screenshot of where the configurations are located at:
Note: These policies only pertain to identity sources connected to vCenter SSO and not OS system logins.