WilliamLam.com

How to identify the origin of a vSphere login?

12.24.2010 by William Lam // 16 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would be re-spawned automatically.

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent	Client
VMware VI Client/4.x.x	vSphere Client
Java/1.6.0_20	VI Java
VI Perl	vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952)	PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.

How to inject custom drivers into an ESXi 4.1 image using vibddi?

11.28.2010 by William Lam // 11 Comments

Over the holiday break, I spent some time cleaning up some of the development virtual machines in our ghettoDatacenter. I came across the VMware Auto Deploy appliance that I deployed awhile back ago. I did not think I had a use for it since we already have an automated deployment system using PXE and kickstart. Auto Deploy was launched relatively recently from the VMware Flings lab. It was originally slated for release as part of vSphere 4.1 but during the transition from the BETA to RC, it was dropped and never made it into the GA release of vSphere 4.1

I decided to give the documentation one last read before deleting and to my surprise, I stumbled across an interesting gem, vibddi. vibddi (pronounced vib d-d-i) stands for VIB (vSphere Instalaltion Bundle) Disk Dump Image, which is actually a Perl utility that was created to help customize ESXi images more easily.

If you ever had a need to customize an ESXi image and inject custom drivers or configurations, you know it can be long and complex process. There are many tutorials on the internet including a recent post by Eric Sloof on injecting drivers into an ESXi installer. vibddi is meant to expedite the process and make it much simpler to inject custom drivers into an ESXi image.

****Disclaimer Since this tool is not very well documented and it is most likely not officially supported by VMware, please use test and validate the images generated prior to using in an production environment Disclaimer****

To run vibddi, you need to use sudo. Here are the available options:

[vi-admin@autodeploy ~]$ sudo vibddi -h
Password:

vibddi: Query and update vibs on a VMvisor dd image or device

Usage:
vibddi -h --- Print this

vibddi -i -q --- Query vibs installed on the image

vibddi -i -c --- Check bootbank filesystems on the image

vibddi -i -v [ -g ] [ -n ] --- Update the image with a single vib

vibddi -i -m -b [ -p ] [ -g ] [ -n ] --- Update the image with an online bulletin

vibddi -i -o [ -g ] [ -n ] --- Update the image with an offline bundle

vibddi -i -e [ -a ] --- Export boot modules from the image

vibddi -i -t --- Add/Remove a VMkernel option

vibddi -i -x --- Transform image to ThinESX format

vibddi -i -l --- Install a license file (vmware.lic) on the image

vibddi -d -q --- Query vibs installed on the device

vibddi -d -c --- Check bootbank filesystems on the device

vibddi -d -v [ -n ] --- Update the device with a single vib

vibddi -d -m -b [ -p ] [ -n ] --- Update the device with an online bulletin

vibddi -d -o [ -n ] --- Update the device with an offline bundle

vibddi -d -e [ -a ] --- Export boot modules from the device

vibddi -d -t --- Add/Remove a VMkernel option

vibddi -d -x --- Transform image to ThinESX format

vibddi -f -k --- Add a customized kickstart file to the ThinESX/Recovery CD ISO

Where:
VMvisor-dd - The VMvisor dd image that is going to be customized

VMvisor-dev - The VMvisor device that is going to be updated

vib-path - The local file path to the vib

metadata-URL - The URL to the metadata.zip file (Ex. http://www.oem.com/depot/metadata.zip)

bulletin-ID - The bulletin ID to install

bundle-path - The local file path to the offline bundle

proxy (OPTIONAL) - Proxy used to download vib, for update operation only

-g (OPTIONAL) - Generate customized ThinESX/Recovery CD ISOs

-n (OPTIONAL) - Bypass signature check, for update operation only

export-path - Directory to export boot modules

alternate-conf (OPTIONAL) - Alternate export configuration file

kernel-opt - VMkernel option

license-path - vmware.lic file (Format: 00000-00000-00000-00000-00000)

iso-path - The local file path to the ThinESX/Recovery CD ISO

kickstart-path - The local file path to the kickstart file

Here are a few examples of using the vibddi tool:

Mount ESXi 4.1 ISO to extract the DD image:

[vi-admin@autodeploy scratch]$ sudo mount -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/iso/

Unzip the DD image and extract to current directory:

[vi-admin@autodeploy scratch]$ sudo bunzip2 -c /mnt/iso/imagedd.bz2 > imagedd

You now should have the DD image called imagedd located in your current working directory.You can name the file anything you want, but I'm using the suggested name as noted in the Auto Deploy documentation.

To list vibs installed on the image, you'll use the following command:

sudo vibddi -i [imagedd] -q

Here is an example of the vibs installed with default installation of ESXi 4.1:

To inject the image with an offline bundle, you'll use the following command:

sudo vibddi -i [imagedd] -o [offline_bundle] -n

Note: The -n flag should be used when performing updates as it bypasses the signature checks, else you will get an error.

Here is an example of injecting the Cisco Nexus 1000 Virtual Ethernet Module offline bundle as part of the default ESXi 4.1 installation:

We can confirm the Cisco VEM is part of the default image by running the query command again:

To inject the image with a single VIB, you'll use the following command:

sudo vibddi -i [imagedd] -v [vib] -n

Here is an example injecting the Cisco Nexus 1000 Virtual Ethernet Module VIB as part of the default ESXi 4.1 installation:

To inject VMkernel boot parameters, you'll use the following command:

sudo vibddi -i [imagedd] -t [vmkernel_option]

Note: Here is a list of a few VMkernel options documented by Dave Mishchenko. The -t argument only accepts one VMkernel option at a time. If you want to updated more than one option, you will need to run the command for each VMkernel option.

With a default installation of ESXi 4.1, there are no VMkernel options defined. To see whether or not these have been defined, you will need to login to Tech Support Mode and view boot.cfg:

~ # cat bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- aam.vgz
build=4.1.0-260247
updated=1
bootstate=0

Here is an example of injecting the following two VMkernel options: noACPI and nopowerManagement:

To inject a license file, you'll use the following command:

sudo vibddi -i [imagedd] -l [license_file]

Note: The license file must contain a single entry using the following format - 00000-00000-00000-00000-00000

Here is an example of injecting license file:

To inject a custom kickstart configuration, you'll use the following command:

sudo vibddi -f [esxi-iso-path] -k [kickstart_file]

Here is an example of injecting a custom kickstart file:

Note: This actually injects a custom ks.cfg into the ESXi .iso which can then be used to deploy an ESXi host including the custom configurations found in the kickstart file. A brand new .iso will be created in the current working directory which includes the timestamp of kickstart injection as part of its filename.

We now can loop mount the new .iso and verify the custom kickstart has been injected:

Note: I'm using the sample ks.cfg found on Kendrick Colemans's site.

You can also extract certain items from the DD image, you'll use the following command:

sudo vibddi -i [imagedd] -e [export-path]

Here is an example of extracting the entire DD image to a temporarily directory:

To check the bootbank filesystem, you'll use the following command:

sudo vibddi -i [imagedd] -c

Here is an example of verifying bootbank filesystem:

Once the imagedd has been updated with all the drivers, you will need to compress the image back to .bz2 using bzip2. From here, you will have two options: A) copy the modified imagedd.bz2 over to your PXE/TFTP server used for automated kickstart installation B) Create a new ESXi .iso, there are a bunch of tutorials online such as here and here.

If you need to troubleshoot or would like to view the process of vibddi, you can take a look at the logs stored in /var/log/vibddi.log. You can also see the injection process which includes both informational and debugging logs in /var/log/esxupdate.log.

As you can see, this tool is extremely useful for injecting and customizing ESXi images. Hopefully one day VMware will officially release this tool and make it available on both UNIX/Linux and Windows platform so that everyone can benefit. For now, if you want to use vibddi, you will need to download and use Auto Deploy appliance. Looks like I'll be keeping this appliance around 😉

What is the VMware VIX API and it's future?

11.21.2010 by William Lam // 4 Comments

Some recent twitter conversations about vSphere SDK improvements, specifically around VMware VIX API and few emails around the use cases was the motivation for this post.

I will provide a very high level of what the VMware VIX API is, since there is a awesome presentation by Matt LaMantia at TechExchange in VMworld 2010 SF that goes into more details about VIX (video below). The VIX API (Virtual Infrastructure eXtension) is an API that provides guest management operations inside of a virtual machine that maybe running on VMware vSphere, Fusion, Workstation or Player. These operations are executed on behalf of the vmware-tools service that must be running within the virtual machine and guest credentials are required prior to execution.

The current incarnation of the VIX API is a little odd because it provides guest management operations which is unique to the VIX API but it also provides virtual machine operations. These virtual machine operation overlaps with some of the functionality provided by the vSphere API which is used for managing your vSphere infrastructure. Part of this overlap was due to origins of VIX which started from Workstation on the desktop. I almost consider this VMware API sprawl 🙂

Currently if you want to automate your vSphere infrastructure and want hooks into your virtual machine and guestOSes you need to leverage both the vSphere API and VIX API. I have seen this cause some confusion in the communities. Users wanting to standardize on VIX API and realize that 80% of what they want to do is available in VIX but the portion requires the use of the vSphere APIs. Don't get me wrong, the functionality of VIX is very powerful. When I first learn in version 1.6.2 of VIX where it officially supported vSphere, a whole new set of possibilities just opened up for administrators and developers.

There are several ways of using the VIX API today. If you are using PowerCLI, there are actually a few cmdlets that directly integrate with VIX, which requires you to install the VIX client libraries local to your PowerCLI installation. One such cmdlet is the Invoke-VMScript and here are a few example use cases:

If you don't prefer the dark side 😉 and want to run something like the vSphere SDK for Perl, C or COM, the VIX client libraries also includes client side bindings to these languages. If you are a Java shop and want to leverage Steve Jin's VI Java there is also an open source project called VIX Java Toolkit that can be used. Here is an example use case with vSphere SDK for Perl:

http://www.virtuallyghetto.com/2010/07/automate-update-manager-operations.html

You can also just install the VIX client libraries which also includes a pre-compiled binary called vmrun that provides majority of the VIX operations all bundled into one utility. It also supports VMware vSphere, Workstation, Fusion and Player. Here is an example use case:

http://professionalvmware.com/2008/12/vmware-vix-changing-ips-of-a-guest-vm/

There is also a very interesting application called VGC (Virtual Guest Console) created by the VMware Lab guys also known as flings. This is a graphical interface similar to VMware Remote Console, but it is 1000x better and provides integration to all the VIX operations. The application allows you to view running processes in a guest, kill a particular process, deploy application to a guest, download/upload files and browse guest filesystem and much much more. I would highly recommend taking a look at this tool!

As you can see, the possibilities are endless! but you still have to use two separate APIs. I along with others in the community have asked for a consolidation of the VIX API and merging with the vSphere API, makes the most sense. I think our feedback has finally been heard and if you watch the presentation given by Matt, you will see that future of VIX API is exactly that, consolidatation with the vSphere API. Though we will not see it anytime soon until the next major release of vSphere dubbed vSphere.next (MN) and vSphere.next^2 (OP), it is a change I am looking forward to.

One other interesting thing to point out, during the first release of the VIX API that supported vSphere which was 1.6.2, there was a tiny bug that was identified regarding licensing. As you may or may not know, VMware requires that you have a "paid" license to be able to leverage the various APIs and CLIs for automation, configuration and management of your vSphere infrastructure. This is generally not an issue when dealing with vCenter or ESX, but with ESXi, you have option of a free license.

In the VIX 1.6.2 release, you actually have full VIX read and write operations against an ESXi host using the free license. This was of course fixed in subsequent releases and the trick only works with ESXi 4.0 or older. You will notice on the VIX landing page, there is no mention of the release notes for 1.6.2 release other than 1.7.1 superseding it. You however still can download 1.6.2 release under VIX 1.6 and you can still see the release notes if you search on VMware's site or on Google.

For more details about VIX and downloads, take a look at the following:

PPC15 Guest Operations Using VMware VIX APIs and Beyond:

Guest Operations using VMware VIX APIs and Beyond from heyitspablo on Vimeo.