Cool Undocumented Features in vCloud Director 1.5

While working on the updated script in Automating vCloud Director 1.5 & Oracle DB Installation, I did some digging in my lab deployment and noticed a few interesting things about the new vCloud Director 1.5 installation.

The first thing I noticed after configuring a new Provider vDC and the vCloud Agent (stored in /opt/vmware/vcloud-director/agent) is pushed out to the ESXi 5 hosts, a new esxcli module is added for vCloud Director under /usr/lib/vmware/esxcli-vcloud

There are 6 namespaces that ranges from simple configuration query, network fence management, account manage and also something called "esxvm" which I'll go into a little bit later. I am not sure why this is not in the vCloud Director documentation, I was not able to find any reference to the new esxcli operations. You may also notice the use of legacy "vslauser" (Virtual Software Lifecycle Automation) throughout vCloud Director, even though it was re-written from the ground up, it looks like VMware decided to either keep the name or some of the code related to the service account.

Here is an example of running "esxcli vcloud about get" command:

Here is an example of running "esxcli vcloud fence getfenceinfo" command:

Lastly, here is an example of what "esxvm" namespace provides:

As you can see above, there are two operations: disable/enable support for 64-bit nested virtual machines. This is exactly the same configuration as I blogged about in How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 but using esxcli interface with vCloud Director 1.5. Let's take a look at what happens when we run the "enable64bitnested" operation.

No surprise, we see that it automatically appends the required vhv.allow = "TRUE" flag which enables the support of running nested 64-bit virtual machines within a physical ESXi 5 host.

You might be asking, why is this in vCloud Director? Well if you attended VMworld 2011 or previous VMworlds and took part in the hands on labs, you will know that VMware utilizes vPods or nested ESXi to deploy their labs. I suspect, this functionality was added into vCloud Director so that VMware can easily leverage nested ESXi for hands on labs or vSel deployments just like they did with Lab Manager previously.

While look into this, I recall a very interesting article by Jason Boche - Deploy ESX & ESXi With Hidden Lab Manager 4 Switch in which Jason identifies a hidden flag in the Lab Manager database that enables a special feature in deploying nested ESX(i) VMs including customization through the use of a special version of VMware Tools for ESX(i). I was curious to see if something similar existed in the new vCloud Director that provided similar functionality.

Looking at the SQL install scripts located in /opt/vmware/vcloud-director/db/{oracle/mssql}, I noticed an interesting config called "extension.esxvm.enabled" in NewInstall_Data.sql file

As you can see from the insert statement, by default this value is set to "false" and we can also confirm this after vCloud Director has been installed and configured by querying the database. Let's go ahead and update this value to "true" and let's see what happens. 

Once you have verified the value has been successfully updated, I decide to use the same trick that Jason had identified with the special "Uber Admin Screen" to load the changes. To my surprise, the trick still worked but the page was not super Uber .... To enable the screen, you will need to click on the "About" page and then click CTRL+U (ctr + shift + u), which will toggle the "Uber Admin Screen".

The available options are quite limited as you can see but there are some new hidden options such as a new debug and console toggle. When you enable these options, you will see them at the bottom right of your screen including a counter of the amount of memory being used for your vCloud Director deployment.

After toggling the hidden database feature, I was not able to see any additional pages relating to nested ESXi hosts, even after restarting vCloud Director. Through some testing, I found that the "extension.esxvm.enabled" actually controlled whether or not nested 64bit VM was enabled when the vCloud Agent was pushed out to ESXi 5 hosts. Instead of manually adding vhv.allow = "TRUE" or using esxcli vcloud esxvm enable64bitnested, vCloud Director will automatically configure the ESXi hosts for you. I still suspect there is probably a hidden interface in managing vESXi hosts and leveraging a specialized version of VMware Tools to automate the deployment of nested ESXi, but I have not found out yet.

UPDATE: Take a look at this blog post for the full details on building your own vSEL - The Missing Piece In Creating Your Own Ghetto vSEL Cloud