William Lam

The folks over at VMware Labs continue to pump out awesome new Flings and just yesterday, they released another one called VistualEsxtop, which is an enhanced version of esxtop and resxtop. It is no surprise, the Fling was developed by VMware Engineers working in the Performance group such as Priya Sethuraman, Zhelong Pan, Haiping Yang and Joanna Guan, some of which who helped develop the original esxtop utility.

VisualEsxtop can connect to directly to an ESX(i) host or a vCenter Server and provides support going all the way back to ESXi 3.5 or vCenter Server 4.0. Here is a summary of all the features for VisualEsxtop:

Features

Live connection to ESX host or vCenter Server
Flexible way of batch output
Load batch output and replay them
Multiple windows to display different data at the same time
Line chart for selected performance counters
Flexible counter selection and filtering
Embedded tooltip for counter description
Color coding for important counters

While reading the instructions, I noticed VisualEsxtop is supported on both Windows and Linux and is loaded by a simple .bat or .sh script. So I wondered if it could run on Mac OS X? Well, it turns out the script uses a utility called readlink which does not operate the same on Mac OS X and will thrown an error. However, since VisualEsxtop is just a Java application, you can manually load the VisualEsxtop and the necessary library files.

To do so, you just need to change into the visaulEsxtop directory and run the following command:

java -cp lib/vtop-ui.jar com.vmware.vtop.VTopMain

Note: For ease of use, you can even create a shell script which executes the command so you do not have to type it out each time.

You should now see visualEsxtop launch after executing the above command:

One really cool feature of VisualEsxtop is the use of color coating for important counters and issues. In the screenshot below, you can see I have a VM that is dropping packets and it is automatically highlighted for me.

I would highly recommend you check out VisualEsxtop and add this to your toolbelt of tools for troubleshooting and diagnosing performance issues in a vSphere environment! If you have any feedback or questions for the engineers, please leave them in the comment section of the VisualEsxtop Fling.

Kenneth Hui recently published a number of interesting articles diving into the latest VMware vSphere integration with the OpenStack Grizzly release called OpenStack For VMware Admins: Nova Compute With vSphere Part1 and Part2. There has definitely been a lot of chatter around OpenStack lately and I agree with Kenneth, there is also a lot of confusion around the topic in general. Although I have not used OpenStack personally, one very important concept to understand is that OpenStack is really just a framework that allows you to build a Cloud solution that is comprised of the best of breed products that can then be plugged into the underlying compute, network, storage and management infrastructure.

One example of this is OpenStack's Nova compute component which supports a variety of Hypervisor solutions including KVM, XEN and now also VMware vSphere. Another example is OpenStack's Neutron (formally Quantum) networking component which also supports a variety of networking platforms including the leader in this space which is VMware's Nicira NVP (Networking Virtualization Platform).

Having said all that, since I have never worked with OpenStack before, I thought this would be a great opportunity to give OpenStack a test run with my vSphere home lab environment. With a quick Google search, I found an OpenStack Wiki guide for setting up VMware's Nova integration and I thought I should be able to just follow that. As it turns out, some of the commands no longer function due to some recent code changes in OpenStack and the instructions were also incomplete for a few steps. With the assistance of the OpenStack development team at VMware, I was able to get everything working and I wanted to share the details while the Wiki gets corrected.

Here is a diagram of what a vSphere and OpenStack solution could look like and we will be primarily focusing on the Nova component:

Pre-requesites:

vSphere ready environment with vCenter Server and at least one ESXi host (I recommend using the vCenter Server Appliance for quick setup)
vanilla installation of Ubuntu 12.04 LTS (you can find more details here)

Here is what my vSphere inventory looks like and the nice about this is you can use an existing vSphere environment. As you can see I have my Apple Mac Mini running ESXi, which is also hosting my vCenter Server along with my OpenStack virtual machine.

Installation:

Step 1 - Install git and we will be using that to clone out the latest DevStack which is basically a huge shell script that helps you quickly stand up an OpenStack instance for testing/development as it is not a trivial task to install OpenStack. Run the following commands on your Ubuntu OpenStack host:

sudo apt-get -y install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

Step 2 - Next we will need to setup a Tun/Tap interface which can do userspace networking and this helps ensure we do not mess with our primary interface (eth0) that is used to connect to the OpenStack VM. Run the following commands:

sudo ip tuntap add dev tapfoo mode tap
sudo ifconfig tapfoo 172.30.0.1 up

Note: You can select any IP Address that is not being used, I chose 172.30.0.1

To confirm the software interface was created correctly, you can run the ifconfig command and you should see a "tapfoo" interface with the IP Address that you had specified from above.

Step 3 - Now we need to create a file called localrc in the devstack directory with the following configurations listed below which will be used by DevStack to build and configure our OpenStack instance.

ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP=192.168.1.127
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD=vmware
VMWAREAPI_CLUSTER=Cluster
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
FLAT_INTERFACE=tapfoo
HOST_IP=192.168.1.143
SCREEN_LOGDIR=/home/primp/devstack-logs
SYSLOG=True
SYSLOG_HOST=192.168.1.104
SYSLOG_PORT=514

The configurations in BLACK are required, where as the ones in GREEN are optional and I will explain those in a bit.

VMWAREAPI_IP is the IP Address of your vCenter Server
VMWAREAPI_PASSWORD is the password of your vCenter Server
VMWAREAPI_CLUSTER is the name of the vSphere Cluster if you have one, else you can leave it blank
HOST_IP is the IP Address of your OpenStack Ubuntu host

Optional configurations:

SCREEN_LOGDIR will log all the OpenStack logs to a directory of your choice. By default, DevStack will log to standard out and only visible through the Screen sessions of each component which is not very user friendly nor easy for troubleshooting.

If you wish to forward OpenStack logs to a remote syslog host, you can also enable the following three configurations which should be pretty straight forward:

SYSLOG=True
SYSLOG_HOST is the IP Address of your remote syslog host (more details on this towards the bottom)
SYSLOG_PORT is the port of your remote syslog host, default will be 514

Note: If you want to learn about other DevStack localrc options, take a look a the documentation here

Step 4 - We are now ready to build and deploy our OpenStack instance. To start, just run the following command:

./stack.sh

This process will take a few minutes depending on how fast your system is and the connection to download all the necessary packages. If everything was successful, you should see a summary about logging into your OpenStack instance and the URL for the Horizon UI as shown in the screenshot below.

Step 5 - Go ahead and confirm you can access the Horizon UI by opening up a browser and pointing it to the IP Address of your OpenStack instance.

Step 6 - To start using OpenStack, we will need to first upload a virtual machine disk to OpenStack's Glance component which handles VM images. There is a sample Debian VMDK that is available on the OpenStack Wiki that we will be downloading to our OpenStack instance. To do so, we will set our credentials on the command-line for the next step and perform a wget to download the VMDK by running the following commands:

source openrc demo demo
wget https://www.dropbox.com/s/utvri5bw3zztty6/Debian-flat.vmdk

Step 7 - We will now use the following Glance CLI to upload our virtual disk and we can also list it once it is uploaded:

glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < Debian-flat.vmdk
glance image-list

Step 8 - To deploy a new instance of the image we have just uploaded, we will switch over to the Nova CLI and specify the Image ID from the previous step and run the following command which will deploy to our vSphere environment.

nova boot --image --flavor 1 my-first-openstack-vm
nova list

Step 9 - We can continue to run "nova list" to view the status, but it would be more interesting to see this from the Horizon UI. You can head over to the OpenStack UI and see the progress under the Instances tab.

Once the VM is ready, we should see an IP Address assignment and the status set to ready and the VM should show powered on.

To confirm that we have actually provisioned the VM onto our vSphere compute cluster, we can login to either the vSphere Web Client or vSphere C# Client and we should see our newly deployed VM running.

If you wish to deploy using the Horizon UI, you can go to Project -> Instances -> Launch Instance and go through the wizard selecting the image, specifying a name and configuration flavor and then click on Launch once you are ready to deploy.

Step 10 - Once you are finished, you can run the ./unstack.sh command which will reset and clean up your environment and delete any images that were uploaded. Again, DevStack is not meant for running production workloads, but can be used for quickly testing or developing against OpenStack. You can also view the consoles of each of the OpenStack components by using screen -x stack.

Using DevStack, you can quickly get a basic OpenStack instance up and running without too much hassle but this is not to say that OpenStack is easy or trivial to install. If this is your first time, I would highly recommend configuring your localrc to store the logs in a directory so you can either go through them if you run into any issues or more likely forward it over to an OpenStack expert to help you decipher. I personally had ran into a few issues and it was partially due to some errors in the Wiki, but troubleshooting can be like search for a needle in a haystack.

DevStack Syslog Configuration

If you recall earlier in the localrc configuration, there is a section that specifies remote syslog configurations for the OpenStack instance. Since I am a fan of the new vCenter Log Insight product that was just released as a beta from VMware, I thought it would be neat to forward the OpenStack logs to it. After a bit of trial and error, it turns out that DevStack configures rsyslog (which is the syslog daemon) running on the Ubuntu host to forward logs using RELP format which is not supported by vCenter Log Insight. If you want to get this working, you will need to disable RELP format by tweaking the rsyslog configuration in /etc/rsyslog.d/90-stack-s.conf

You will need to replace :omrelp:

*.* :omrelp:192.168.1.104:514

to just @@:

*.* @@192.168.1.104:514

Finally, you need to restart the rsyslog service for the changes to take effect by running the following command:

service rsyslog restart

If we login to our vCenter Log Insight UI, we should now see our OpenStack instance logging remotely. Once you unstack and run stack, the configurations will default back to the original.

Additional Resources:

How to Run VMware's New Fling VisualEsxtop on Mac OS X

Features

How to Quickly Get Started with VMware vSphere & OpenStack?

DevStack Syslog Configuration

New Windows 8.1 Preview on vSphere 5.1 Update 1