How to identify the origin of a vSphere login?

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would  be re-spawned automatically. 

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.