WilliamLam.com

vOpenData: An Open Virtualization Community Database

by // 11 Comments

Recently, I had the opportunity to help out with a very unique and cool project called vOpenData which was created by Ben Thomas (a former VMware GSS Technical Engineer). The idea for the project was sparked by a very simple tweet that came from Duncan Epping:

Ben wanted to help answer Duncan’s question but more importantly he wanted to help answer a bigger set of questions: what are some of the common virtual infrastructure deployment configurations, averages and consolidation ratios? These questions cross the minds of the everyday vSphere administrators, architects and consultants. It would be quite difficult and nearly impossible to answer these questions outside of their own environment.

Ben reached out to me with his idea and asked if I could help develop a script to collect basic configuration information from a vSphere environment to help test out his idea. I was immediately intrigued with his idea and saw the huge potential value that Ben’s unique solution could bring to the virtualization community. The coolest thing about this project is that we were able to put together a working prototype within a week’s time!

Note: Also be sure to check out Ben's article vOpenData - Crunching everyone's data fun for fun and knowledge and his perspective on how he was able to quickly develop a prototype leveraging a PaaS solution.

What is vOpenData?

vOpenData is an open community project that grew from the question "What is the average VMDK size for deployed virtual machines?” We wanted to create an open community database that is purely driven by users submitting their virtual infrastructure configurations. Leveraging the powerful virtualization community and applying simple analytics we are able to provide various trending statistics and data for virtualized environments. This is 100% community driven and the results will be available for everyone to view and hopefully you will contribute to the overall dataset!

What information do we collect?

We made an effort to not collect specific information such as hostnames or even display names that could be used to identify a particular organization. Instead, we are using UUIDs which are automatically generated by the virtualization platform to uniquely identify a particular object. This allows us to keep track of changes in the our database when a new data set is uploaded from an existing environment. In addition we are collecting various configuration data and you can find a complete list in the Data FAQs

More info on the data we collect is here: Data FAQs

What will this data be used for?

We are planning on using this data to create some interesting statistics and data modeling for the community to use in capacity planning and analysis. Most of this data will be made available through a dashboard or reports and eventually through an API to be mixed into other applications.

What about privacy concerns?

Though the data that is collected is already anonymized and non-identifying, please ensure that you are abiding by the privacy policies of your organization when uploading this data. If you are concerned about the data, it is recommended that you audit the zip contents before uploading which are just CSV files. We only ask that you do not modify the schema at all.

How do to get started?

Step 1 - Check out the sexy vOpenData Public Dashboard here to get a glimpse of some of the information you will find by submitting your configuration data.

Step 2 - Download either the PowerCLI or vSphere SDK for Perl script which you will run against a vCenter Server which will produces a compressed zip file containing several CSV files. Instructions are available on the download page. You may rename the default file name vopendata-stats.zip to something else, as long as you do not modify the contents of the file.

Step 3 - Open a browser and go to http://www.vopendata.org and sign up for new account.

Step 4 - Click on the “Infrastructures” tab at the upper left hand corner. An Infrastructure is a logical view that can help you organize the data you have collected. You can associate a single vCenter Server with an infrastructure or you can combine multiple vCenter Server data sets into a single infrastructure. The choice is really up to you on how you would like to visualize your data and whether you would like to map that to the physical location of your virtual infrastructure.

Step 5 - Once you have created your Infrastructures, you will then upload your data files to their respective Infrastructure. This may take some time as the data processing is executed in the background and will also depend on the number of users and uploads occurring at the moment. We ask that you please be patient and check back in a bit and you can refresh the page which will let you know when the processing is complete

Step 6 - After the data is uploaded to the system, there is a scheduled job that performs the analytics and calculations which occurs in periodic batches. These calculations can take up to 45minutes to an 1hour before the results are reflected in the public dashboard and is primarily governed by the single worker we have on the backend due to resource constraints. To view the results of the public dash board visit http://dash.vopendata.org

We hope you frequent the vOpenData site regularly as the community uploads more and more data and see how statistics are trending over time. We would also like thank the following people who were part of our early alpha program and assisted with both testing as well as code contributions: Frederic Martin, Raphaël SCHITZ, Timo Sugliani and of course my Automation colleague Alan Renouf! If you would like to learn more about the vOpenData project, we have also submitted a session for VMworld 2013 4976 - vOpenData - Crunching Everyone's Data For Fun And Knowledge, be sure to vote for it!

You can follow @vopendata on Twitter for new updates and notifications as well as both Ben Thomas at @wazoo and William Lam at @lamw

How can I help or contribute?

First and foremost, you can get involved by signing up for a free account and begin contributing your data to the open community database! We are also open to any suggestions and feedback as they would be very valuable to us, feel free to join the vOpenData VMTN Community Group to discuss further. We know that in this first release we are not going to be able to show everything, but have plan to show much more. Lastly, all the infrastructure that is used to provide the dashboard, the backend database and processing is all hosted and paid out of our own pockets. If you have found this to be a useful resource and would like to contribute either with a donation or sponsorship to help us continue developing this project, please contact us at vopendata[at]gmail[dot]com

How To Quickly Get Started With The New VMware Puppet Modules

by // 5 Comments

Yesterday, VMware Automation gurus Nick Weaver and Nan Liu just announced the release of four awesome new VMware Puppet modules that can help you manage and configure vCenter Server (including ESXi) and vCloud Networking & Security (vShield). You can read all about the details here and here and if you were lucky enough to have attended PEX (Partner Exchange) you might even have caught the demo given by Nick in his session.

I have used Puppet in the past, but it was pretty limited and specifically in How to Deploy ESXi 5 Using Razor & Puppet. I thought this might be a good time to revisit Puppet and try out the new VMware Puppet modules. I took a look at some of the examples provided by Nan on his blog but for new users to Puppet, it may not provide enough details to quickly get started (including myself). I thought I document the minimal steps I took to quickly get started (I also ran into a few bugs which Nan has fixed).

Step 1 - Install Ubuntu Precise (Ubuntu Server 12.04 LTS - See more at: http://www.virtuallyghetto.com/2012/05/how-to-deploy-esxi-5-using-razor-puppet.html

Step 1 - Install Ubuntu Precise (Ubuntu Server 12.04 LTS). You can use other distros, I just choose Ubuntu as I had the image lying around.

Step 2 - Download Puppet Labs package repository by running the following commands:

wget http://apt.puppetlabs.com/puppetlabs-release-$(lsb_release -c | cut -f 2).deb
dpkg -i puppetlabs-release-$(lsb_release -c | cut -f 2).deb
apt-get update

Step 3 - Install all the necessary packages such as Ruby, Ruby Gems, Puppet, etc. by running the following commands:

apt-get install -y libxslt-dev libxml2-dev ruby rubygems puppet
gem install nokogiri
gem install net-ssh

Step 4 - Install the VMware Puppet modules by running the following command:

puppet module install vmware/vcsa
puppet module install vmware/vcenter
puppet module install vmware/vshield

To start using the VMware Puppet modules, you will need to create what's known as a manifest file that contains the resources which maps to the actions you wish to perform (e.g. configure a newly deployed VCSA appliance or create a Cluster in vCenter Server and add an ESXi host to that cluster). You can find a bunch of example manifest files in each of the Puppet modules, here is the path to each:

/etc/puppet/modules/vcsa/tests/
/etc/puppet/modules/vcenter/tests/
/etc/puppet/modules/vshield/tests/

You will see in some of the examples, they import a file in each directory called data.pp which contains the actual definitions of your VCSA, vCNS and ESXi hosts but you can also just specify that in the main manifest file as well for simplicity. The latter option provides more flexibility as you can easily reference various configurations for different environments. For your convenience, I have created the following manifest files that you can use and you just need to modify them to fit your environment.

Here is what my lab environment looks like and their respective IP Addresses for your reference (these must already be deployed and vCenter & vCNS does not need to be configured but just accessible over network):

vCenter Server = 172.30.0.135
vCloud Networking and Security = 172.30.0.136
ESXi Host = 172.30.0.137

Step 5 - As mentioned by Nan, a custom Rbvmomi was used and we will need to ensure our Puppet management host (Ubuntu system we are on) includes it. To ensure all the necessary packages are downloaded for us, we will use the rbvmomi.pp manifest file for our host and use Puppet to apply the policy. Replace management_server in rbvmomi.pp with the hostname or IP Address of your Ubuntu host and then run the following command:

puppet apply rbvmomi.pp

Note:  You can safely ignore the red warnings, it must not have liked something in my environment.

Step 6 - We will start off by configuring the VCSA so we can then perform operations such as adding in Datacenters, Clusters, ESXi hosts, etc. We will use the configure-vcsa.pp manifest file by running the following command:

puppet apply configure-vcsa.pp

Step 7 - Next we will create a Datacenter, Cluster and add our ESXi host by using the setup-vcenter.pp manifest file by running the following command:

puppet apply setup-vcenter.pp

Step 8 - We are now onto configuring vCloud Networking and Security and we will also associate it with our vCenter Server by using the configure-vcns.pp manifest file and running the following command:

puppet apply configure-vcns.pp

Step 9 - After configuring vCloud Networking and Security, we can now deploy a vCloud Networking and Security Edge Gateway to provide various networking services to our vSphere environment using the deploy-edge.pp and by running the following command:

puppet apply deploy-edge.pp

In about 5-10 minutes, you will have a fully configured vSphere environment that contains your vCenter Server, vCloud Networking and Security Manager and Edge Gateway and ESXi hosts all ready to start providing compute and networking services for your virtual machines and applications! I want to stress the above is a very simplistic example of what you can do with the new VMware Puppet modules. There are definitely more advanced capabilities provided in the modules and I would recommend you take a look in the samples directory of each module for more details.

Overall, I was pretty impressed with the VMware integration that Nick, Nan and team built with Puppet. This was a great learning experience for myself, I learned quite a bit with just trying out these modules and I think I might have found a reason to dive more into Puppet! 🙂

Big thanks to Nan for helping me out with some of my Puppet questions!

How to Access vSphere Remote Console Using vSphere & VMRC API

by // 42 Comments

Similar to my vCloud Director article, you can also provide access to the remote console of a virtual machine in a vSphere environment to your end users. This can be accomplished by leveraging both the vSphere and VMRC (Virtual Machine Remote Console) APIs and can be useful if you are building a custom portal for users to access their virtual machines. To use the VMRC API, you can download the latest VMRC 5.1 SDK which provides the following:

The VMRC SDK allows you to use a Web-based application to connect to a vCenter- or vCloud Director-managed virtual machine and access that virtual machine’s console in a browser window. You can interact with the virtual machine console input and screen. You can also use the VMRC SDK to manage virtual and physical device connections on a vCenter-managed virtual machine. 

The VMRC SDK includes documentation to the API as well as a sample webpage implementing some basic functionality of the VMRC API. I recently received a question on how to get started with the sample as it was not completely intuitive and thought I take you through the required steps to get the sample working.

Step 1 - Download the VMRC 5.1 SDK and extract the contents to your local desktop.

Step 2 - Open the vmrc-embed-example.html using a web browser located in the docs folder.

Note: In my example, I uploaded both the javascript and html file to a web server and accessed the sample by connecting to the server instead of running it locally on my desktop. This was to show how users could access the custom portal using the VMRC SDK.

Step 3 - At the top of the page where it says "VMRC Modes", make "MKS" is selected in the drop down box and click on the "+" icon to add. Then go ahead and click on the "Start" button to start a VMRC instance and ensure you see a success message on the right hand side of the console box.

Step 4 - To authenticate to VMRC, you will need a session ticket which will be obtained through the use of the vSphere API using the acquireCloneTicket() method provided by the SessionManager managed object. In this example, we will be using the vSphere MOB to quickly retrieve our session ticket, but in a real implementation, you would programamtically retrieve the session ticket along with few othe pieces of information to connect to the VMRC. Open up a new tab in your web browser and connect to the following URL:

https://reflex.primp-industries.com/mob/?moid=SessionManager&method=acquireCloneTicket

Note: Make sure you substitute in your vCenter Server IP Address/Hostname

Once you have authenticated, go ahead and click on the "Invoke Method" which should generate a session ticket:

Step 5 - Copy the session ticket string and go back to our VMRC sample page. We will now need to fill out the following sections before we can access the remote console of a virtual machine:

  • Hostname (IP Address/Hostname of your vCenter Server)
  • Allow SSL Validation Errors (Check this if you are using self signed SSL certificates)
  • Ticket (Paste the session ticket from the previous step here)
  • VM ID (This is the MoRef ID of the VM you wish to connect to the remote console)

Once you have filled out the minimum required fields, go ahead and click on the "Connect" button and if everything was successful, you should now see the remote console of the virtual machine you selected.