vSphere – Page 36

How to identify the origin of a vSphere login?

12.24.2010 by William Lam // 16 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would be re-spawned automatically.

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent	Client
VMware VI Client/4.x.x	vSphere Client
Java/1.6.0_20	VI Java
VI Perl	vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952)	PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.

vMotion' With Style

10.20.2010 by William Lam // Leave a Comment

I got this idea after catching an interesting tweet by Cody Bunch last Friday:

I had just recently finished a post on How to Ack & Reset vCenter Alarm implementing hidden API method and thought this might work by using a vCenter alarm. After few hours in our skunkwork's lab, I found "a" method to exactly what Cody wished for. If you can not wait, jump straight to the video at the bottom 🙂

The following tools were used:

PsExec -Windows utility to perform remote operations
SayStatic - Windows text to speech utility
A random mp3 audio file found online

The environment consisted of two vESXi 4.1 hosts and Windows XP VM residing on shared storage that was vMotion functional:

You will need to upload some files to your vCenter server and to the desktop in which you want to implement this hack.

Local Desktop Server Setup

1) Download SayStatic to your local desktop ( in the example, it is located on the desktop)

2) Download a playable mp3 file (in the example, it is located on the desktop)

Note: If you decide to change the path to the files above, please make note of the path as it will be used later in the alarm script.

Here is a screenshot of my local desktop:

vCenter Server Setup

1) Download PsTools toolkit and extract the contents to your vCenter Server and make a note of the path (in the example, it is located on the desktop)

2) Create an alarm script and make a note of the path ( in the example, it is located in C:\alarm.cmd)

The alarm.cmd should contain the following:

You will need to edit the script and update at minimum the following variables:
REMOTE_SERVER = This is the hostname or IP address of your local desktop, make sure you keep the double slashes which is needed

REMOTE_USERNAME = This is the username to login to your local desktop and it should be the one you use to login else you will not see anything interesting. Make sure you include both the Domain and username, if your system is not part of a domain, use the local system name else the script will fail.

REMOTE_PASSWORD = The password to the account aove

Note: If you have changed the path of the files on either the local desktop or vCenter Server, you will need edit the remainder variables that specify where to look for the executable's.

Here is a screenshot of vCenter Server desktop:

vCenter Alarm:

Finally, we need to create an alarm in vCenter, you can create this at any level of the vCenter infrastructure.

1) Create an alarm and give it a name and ensure the type is of "Virtual Machines" and monitor type is "Monitor for specific events occurring on this object" (in the example, I call it VMOTIONIN_WITH_STYLE)

2) Now click on the Triggers tab and look for an event type called "VM emigrating", no this is not a typo. VMware apparently has both "VM migrating" and "VM emigrating", apparently the functional alarm is under the name emigrating ... don't ask me why they called it that ;). Make sure the status is set to "alert"

3) Next, click on the Actions tab and set the action to "Run a command" and specify the configuration to the path of the alarm.cmd. In the example, I stored the script in C:\alarm.cmd and then make sure it alarms when it hits "red" or error and then click okay for the alarm to be created.

Now you are ready to go. Instead of explaining and providing screenshots, I thought I show you what this would look like (explanation of what is going on is at the very bottom).

Without further ado, here is a recorded video of this in action:

vMotion' With Style from lamw on Vimeo.

What is actually going on:
When a vMotion is triggered, it will fire off the alarm.cmd script which basically uses SayStatic.exe to remotely execute on your local desktop to announce the virtual machine being vMotioned by capturing the VMware specific environmental variables and then it will also remotely playing the local audio file using Windows Media Player.One caveat that I was not able to solve, was clearing the alarm after the vMotion. You will notice the virtual machines that vMotion and fire off this alarm will stay alerted and will not play again until it has been resetted to green. I thought about creating another alarm to clear this initial alarm but it did not actually clear the alarm.

There you have it, vMotioin' with style ... though cool in concept, I doubt you will last very long with this kicking off on every single vMotion in your environment.

Big thanks to Cody Bunch for the idea! 🙂

Script: ghettoVCB Email Support (Experimental Support)

07.28.2010 by William Lam // 14 Comments

Check out the latest update to ghettoVCB which now support backup logs to be emailed upon competition. This feature is in experimental support and require the availability of nc (netcat) utility which is included in ESX(i) 4.0+. One can utilize netcat for variety of networking tasks, including communicating to an email server. The reason this feature is in experimental support is it may not be compatible with all email servers. The email functionality has been tested against a default installation of Postfix and is provided as-is.

For more details, please take a look at the ghettoVCB documentation.