api – Page 10

How to identify the origin of a vSphere login?

12.24.2010 by William Lam // 16 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would be re-spawned automatically.

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent	Client
VMware VI Client/4.x.x	vSphere Client
Java/1.6.0_20	VI Java
VI Perl	vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952)	PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.

VMware API Related Acronyms

08.04.2010 by William Lam // 3 Comments

Duncan Epping recently wrote an article about VMware related acronyms and shared with the community the history and origins of some of the acronyms (e.g. ESX, GSX). After reading Duncan's article, I realized the acronym list primarily focused on product names and features. They did not cover some of the API related acronyms that I was curious about. If you have worked with the VMware APIs, SDKs or vmkernel, vmkwarning, hostd, vpxa, vpxd, etc. logs you may have noticed some of these acronyms and wondered what they actually stood for.

Here is my compiled list of VMware API related acronyms that I have been able to dig up by looking at these various locations:

DMS: Data Management Service
EAM: ESX Agent Manager
FDM: Fault Domain Manager
HMO: Host Managed Object
LS: License Service
MOB: Managed Object Browser
MOR: Managed Object Reference
OMS: Operation Management Service
RBD: Rule Based Deployment (Auto Deploy)
SMS: Storage Monitoring Service
SPS: Storage Policy Service
STOREMON: Storage Utilization Monitoring
VCI: (vci-integrity deals with VMware Update Manager, but no info on acronym)
VMAM: VM Application Monitoring
VMODL: VMware Managed Object Design Language
VMOMI: VMware Managed Object Management Interface
VOD: VPX Operational Dashboard
VOB/VOBD: VMkernel Observation
VORB: VMOMI Object Request Broker Library/Application
VSM: vService Manager
VWS: VMware Web Services
WOE: Workflow Orchestration Engine

Disclaimer: These namespaces may change without notice as they are part of the internal VMware APIs. These are not publicly documented by VMware and should not be relied on other than for informative purposes.

There was also a recent VMTN question regarding the following two licensed features that is displayed for a given ESX(i) host and it was not clear what they were used for:

Here are the actual names and purposes:

dpvmotion: Direct Path vMotion (VM DirectPath I/O)
dvfilter: vNetwork Appliance API (dvfilter was an internal name)

The following are not VMware acronyms, but you may see these from time to time if you work with the vSphere API/SDKs:

API: Application Programming Interface
SDK: Software Developement Kit
WSDL: Web Services Definition Language
SOAP: Simple Object Access Protocol

I would also like to thank both Steve Jin and Reuben Stump from VMware in helping me track down the last few acronyms that I could not figure out.

If there are any additional acronyms that I missed or ones I should add to the list, please feel free to leave me a comment.

New way of enabling and disabling services using vSphere 4.1

07.14.2010 by William Lam // 2 Comments

While checking out the PlanetV12n feed, I noticed a new video from David Davis about the new vSphere 4.1 Tech Support Mode. In the short video, David goes over the new method of enabling "hidden" unsupported Busybox Console, also known as Tech Support Mode. In the past, you had to be on the console of your ESXi host, type ALT+F1, and then "unsupported" to gain access. Once in, to enable remote SSH access or Remote Tech Support Mode, you had to edit /etc/inetd.conf and restart inetd service. This was pretty tedious if you needed access for a short period of time. In the video, David goes over the new method showing how it can be done using the DCUI and the old method is no longer required.

What surprised me after watching the video was that he did not mention the other method of enabling and disabling Tech Support Mode both local and remote. One issue I had with past releases of ESXi is that you could restart some services such as ntp or vmware-vpxa via the vSphere API, but others were just not available. In vSphere 4.1, VMware introduces a few new services around their Likewise Active Directory integration but also includes controlling both local and remote Tech Support Mode as well as DCUI itself.

These services can be enabled and disabled using the vSphere Client, here is a screenshot:

To enable or disable TSM, just click on the service and then click on options:

You will then have the option to configure the startup policy including enabling or disabling the service:

If you needed to perform this operation against one or two host, it is not that big of a deal. Though if you needed to enable remote Tech Support Mode (SSH access) across few dozen hosts, then can still be tedious. Luckily I wrote a script (hostServiceMangement.pl) last year that allowed you to enable and disable supported services using the vSphere API. Without any modifications, it supports vSphere 4.1 and can take advantage of the new services that are available for control.

Here is an example of listing the services on an ESXi 4.1 host:

Here is an example enabling remote Tech Support Mode:

Here is an example of disabling remote TSM:

The script can be executed on a host that has vCLI 4.1 installed or on vMA 4.1 and can bulk update a list of ESX/ESXi host or individual host. For more details, please check out the documentation for hostServiceManagement.pl.