WilliamLam.com

EMC Project OnRack now RackHD

by // 1 Comment

Back in May, EMC announced a new initiative at EMC World called Project OnRack which had an ambitious goal of providing a new software abstraction layer that would sit on top of existing "industry standards" for server out-of-band management. Standards such as IPMI, CIM, SMI-S and CIM-SMASH to just name a few were supposed to help IT administrators manage and operate the life-cycle of their physical servers. Instead, we ended up with even more complexity and inconsistency due to the different implementations of these "industry standards" across vendors and sometimes even within the same vendor. Trying to keep firmware, BIOS, hardware drivers, etc. up to date across different hardware platforms from the same vendor in a consistent and automated fashion was already painful enough. As If this was not already challenging enough, try doing this for a mix of hardware platforms across different vendors and you have just given your operational and datacenter team a never ending nightmare.

Frankly, I am pretty surprised that it has taken us this long to finally tackle this problem. This is something we have needed for quite some time now and I still remember the early days as an admin trying to script around the inconsistencies of IPMI to configure things like asset tags and serial numbers across different hardware platforms.

In my opinion, having a consistent and programmable interface to this low level of hardware is a critical component to a Software-Defined Datacenter and has often been overlooked. Kudos to EMC for taking on this initiative and more importantly driving this change through open-source and the community in mind.

Since the announcement back in May, things have been been pretty quiet about OnRack, until recently that is. I was listening to a recent episode of The Hot Aisle Podcast with guest Brad Maltz of EMC talking about Hyper-Converged Infrastructure. Among the different topics discussed, OnRack was brought up along with dis-aggregated hardware/infrastructure where individual compute resources can scale up independently of each other. There were a couple of nice tidbits mentioned on the podcast. First, it looks like OnRack which was the internal EMC project name has now been renamed to RackHD as the external project name. Second, it looks like the RackHD repo is already on Github with some initial content including some pretty detailed documentation on the architecture and components which can be found here.

The OnRack project looks to be made up of the following sub-projects per the documentation:

  • on-tftp - NodeJS application provided TFTP service integrated with the workflow engine
  • on-http - internal HTTP REST API interfaces integrated with the workflow engine
  • on-syslog - syslog endpoint integrated to feed data into workflow engine
  • on-taskgraph - NodeJS application providing the workflow engine
  • on-dhcp-proxy - NodeJS application providing DHCP proxy support integrated into the workflow engine
  • onserve - OnServe Engine
  • core library - Core libraries in use across NodeJS applications
  • task library - NodeJS task library for the workflow engine
  • tools - Useful dev tools for running locally
  • webui - Initial web interfaces to some of the APIs - multiple interfaces embedded into a single project
  • integration tests - Integration tests with code for deploying and running those tests, as well as the tests themselves
  • statsd - A local statsD implementation that makes it easy to deploy on a local machine for capturing application metrics

Brand mentioned that many of the Github repos are still marked private as they are still working through the process of releasing RackHD to the public. It looks like RackHD and all relevant repos are now all open source as of Monday Nov 2nd, for more details please visit the Github repo here. I am definitely excited to see how this project will evolve with the larger community and some of the new innovations which will be unlocked due to this barrier being removed. Hopefully we will see positive collaboration from other hardware vendors which will help us move forward and finally solve this problem once and for all! I can already see huge benefits for software only vendors like VMware who can integrate RackHD directly into provisioning tools like Auto Deploy or configuration management tools like Puppet, Chef and Ansible for example. It will also be interesting to see how other startups in this area like NodePrime and another stealth company, who is also working on solving a similar problem and whether they would leverage RackHD or not.

Did you know that VMware Host Profile is extensible by 3rd Parties?

by // 1 Comment

Managing ESXi host configurations can be challenging and the potential risk for configuration drift between the running environment and the set of configuration scripts or worse, manual configuration is quite high. On top of that, how do you ensure proper compliance of all your ESXi host configurations in your environment and easily prove that in an internal or security audit?

This is where VMware Host Profile can help which allows administrators to capture the running configurations of an ESXi host and automatically creating a template (Host Profile) that can then be applied across new or existing ESXi hosts. By leveraging Host Profile, administrators can ensure that all their ESXi host configurations are always consistent and configuration drifts can easily be prevented through automatic compliance checks.

Recently, while searching for something on VMware's HCL website, I accidentally stumbled onto what appears to be 3rd party Host Profiles? There were only two listed, one from Brocade for managing and configuring Brocade storage adapters and the other from Dell for managing and configuring Dell's EqualLogic MEM (Multipathing Extension Module). I was actually quite surprise to learn about these custom 3rd party Host Profiles. In doing a bit of digging and research it turns out that VMware Host Profile are in fact extensible by design, which was something new to me.

Note: For a technical overview of Host Profile, you can take a look at this whitepaper here. 

Host Profile Architecture

Host Profile was first introduced with the release of vSphere 4.1 and the brain of the system is known as the Host Profile Engine which was part of the vCenter Server. In vSphere 5.0, Host Profile was re-architected and the Host Profile Engine was moved into the ESXi host which allowed for Host Profile Plugins to be added to an ESXi Image and expose new Host Profiles through the Host Profile Engine.

A Host Profile is actually a hierarchical composition of multiple sub-profiles and policies. Each policy defines a set of parameters that a user can select from and apply to an ESXi host. For instance, the default VMware Host Profile is composed up of 12 individual sub-profiles: authentication, datetime, firewall, memory, network, option, security, service, storage, userAccount and userGroupAccount.

With this new re-architecutre, Host Profile can be extended by 3rd party partners/vendors to create custom Host Profile Plugins to expose vendor specific hardware or software configurations and made available through a common Host Profile API/UI for customers to consume.

Host Profile Extensibility Options

To build a Host Profile Plugin, you will need to use the Host Profile SDK which is only available as part of VMware TAP (Technology Alliance Partner) Program. A Host Profile Plugin basically wraps the actual configuration work and can be backed by one of three ways:

  1. CIM Provider using the CIM SDK
  2. ESXCLI plugins
  3. Userworld binaries

As you can see, creating a Host Profile Plugin is quite flexible and can be exposed through several mechanisms. The most shocking discovery that I found was the lack of 3rd party vendor Host Profiles that exists today, especially from the big server hardware vendors. Coming from a Systems Administrator background, I would loved to have been able to configure and manage my server's firmware, BIOS, out-of-band management (iLO/DRAC), etc. through either a custom ESXCLI plugin or Host Profile Plugin. This would really benefit customers from having to manage configurations using multiple tools and allowing them centralize their management including compliance capabilities all from a single interface.

Hopefully this was an educational post for everyone and if you are a customer and would like to see certain functionality exposed by our 3rd party partners, feel free to leave a message and perhaps one of them may consider adding a custom Host Profile Plugin 🙂

CIM monitoring caveat with ESXi

by // 9 Comments

I recently started to play around with the CIM API to monitor the hardware on an ESXi host. Instead of relying on hardware agents/scripts that would normaly run in the traditional Service Console of ESX, CIM is an API that allows you to monitor the health of your ESX or ESXi host. You can see the health status of your host by logging directly into the ESX(i) host using the vSphere Client and clicking on Configuration->Health Status tab.

I decided to start off with a small python script that would run on vMA and using the wbem python module to make a simple connection to query for the ESXi version. Here are the steps to get the following script working:

1. Download pywbem onto vMA

2. Extract the contents of pywbem

tar -zxvf pywbem-0.7.0.tar.gz

3. Install pywbem

sudo python setup.py install

4. The script expects the hostname/IP of your ESX(i) host as it's first argument and the username as the second argument and then you will be prompted for the password

./cim.py himalaya.primp-industries.com root

If everything went according plan, you should see the version of your ESX(i) host printed on the screen.

Next I wanted to create a dedicated service account so that I do not have to use the root account, I thought a read-only role would suffice.

To my surprise, when I ran the script again with this user, I received an unauthorized access error.

At first I thought the user account required an "Administrator" role to perform the operation but after further investigation, I found that the user account must be part of the "root" user group. Even for a read operation, it still needed to be in that that group.

After I made the change using the vSphere Client and re-ran the script, it executed as I expected.

After speaking to someone at VMware regarding this issue, it was confirmed by engineering that this is in fact a software bug and it should not require the user account to be part of the "root" user group to query from the CIM API.

I was still interested in using CIM, but I wanted to lock down the account as much as possible and came up with the following snippet of code which can be included in your ESX(i) kickstart configuration.

The script creates a regular user who does NOT have login access to the ESX(i) host. It then puts the user into the "root" user group and then creates a new role called CIM with a single privilege Host.Cim.CimInteraction and then associates this user with this role.This ensures that the account can only perform read-only operations against the CIM API and does not allow for host logins. Until the bug is resolved, this should be an acceptable work around.

So what type of monitoring can you do with CIM? Well pretty much anything and everything. There is a popular Nagios script that monitors the hardware health of an ESXi host using the CIM API called check_esxi_webem.py that one can implement to alert on your hardware components.

The script currently expects three arguments: hostname/IP of ESX(i) host, username and password on the command line (this can be changed with minor modifications). If you run it using those defaults, you will either get an OK or WARN/ERROR which will include additional information about the component that is alarming.

If you would like to get more details on the components being checked, you can pass in a fourth parameter called "verbose" and the script will provide more information on what is being checked.

If you are not big on python, there is also a Perl SDK for CIM/WSMAN as part of the vCLI installation and if you are using vMA, you can find some great examples under /usr/share/doc/vmware-vcli/samples/WSMan

The checksensorhealth.pl is definitely one to take a look at, here is an example output:

If you are interested to learn more about CIM, take a look at the these resources:
CIM SDK
7 Part series on ESXi Chronicles blogs about CIM and hardware monitoring