Nested Virtualization

OVF template for creating Nested ESXi 3 or 32 node VSAN Cluster

04.15.2014 by William Lam // 14 Comments

Last week I had to build a couple of Nested VSAN environments for testing and of course I used my VSAN Nested ESXi OVF template to help expedite the deployment. After deploying the OVF for the third time to get my three Nested ESXi nodes, it hit me. Why am I doing this each time when I know I will need a minimum of three nodes for a proper VSAN environment? Not sure why I did not think of this earlier, but why not create a vApp that contains three Nested ESXi VM templates?

By leveraging the Dynamic Disk feature in OVF, I was able to create two tiny vApps (40KB & 410KB respectively) based off of my original Nested VSAN ESXi OVF template:

The only difference with these OVF templates is that you can now easily an quickly deploy a single OVF that will contain the minimal number of VSAN nodes up to the maximum supported which is 32.

Disclaimer: Nested Virtualization is not not officially supported by VMware, please use at your own risk

Prerequisite:

vSphere Web Client
- To deploy either the single VSAN Nested ESXi OVF template or these new ones, you need to make sure you deploy using the vSphere Web Client. The reason for this is that the lossless OVF import/export feature is only available when using the vSphere Web Client, else you the import will not capture all the settings the OVF template was configured with.
vSphere Cluster w/DRS enabled
- vApp creation is only possible when DRS is enabled

Step 1 - Deploy the OVF template using the vSphere Web Client and make sure you select "Accept extra configuration options" which contains extra parameters needed to run ESXi and VSAN in a nested environment.

Step 2 - Go through the OVF deployment wizard as you normally would. When you get to "Customize Template" you will notice each Nested ESXi VM is in its own Category as seen in the screenshot below. Here you can leave the defaults for a minimal VSAN deployment which contains 2GB disk for ESXi installation, 4GB disk for an "emulated" SSD and 8GB disk for MD or you can specify the size for each disk.

In just a couple of seconds, you will now have a vApp that contains either a 3-node Nested ESXi VM or you can go big and deploy a 32-node Nested ESXi environment.

Note: Please note there maybe other configurations changes such as this one and/or increase in VM resources to run larger VSAN Clusters.

I know these OVF templates will come in handy for myself when needing to quickly deploy a VSAN running in a Nested ESXi environment and hopefully it will also benefit others in the community as well!

Running Nested Xen Hypervisor with VMware Tools on ESXi

04.04.2014 by William Lam // 1 Comment

I recently saw an interesting tweet from former VMware colleague Steve Muir who use to run our VMware Academic Program (VMAP). From the looks of things, Steve has been exploring the virtual rabbit hole by performing some cool multi-level vInception by running Xen in a Nested ESXi Virtual Machine running on top of VMware Fusion 🙂 I wonder if he has tried to run a VM in the Xen instance yet?

Anyway, the thing that caught my eye was not the fact that you could run a Xen Hypervisor within ESXi, this has actually been done before in the past. What was interesting to me was the fact that he could not get VMware Tools to show up when Xen is installed and running. I was actually curious to see why this was the case and while waiting for some of my physical hosts to finish rebooting, I figure I give this a go myself. I was leveraging the Wiki page that Steve had initially provided but that gave me some issues as Xen would kernel panic upon a reboot.

In talking to Steve about this, he provided me with another link to a QuickStart guide for Xen which simplifies the setup and I was able to finally get it booted up (definitely not as easy as an ESXi install) . Afterwards, I just installed VMware Tools as I normally would and you can even use this one-liner script here that will automate the installation:

Funny enough, just by using this latest guide for the Xen installation, I was able to get VMware Tools to show up in the vSphere Web/C# Client without any additional tweaks. Here is a screenshot to prove this works:

If you are interested in setting this up, here are the high level steps:

Install latest CentOS which is 6.5 using either the full ISO installer or netinstall which is what I prefer to use
Install Xen per the Wiki
Install VMware Toosl (this step can be swapped with Step 2 if you, it works either way)

Thanks Steve for sharing your notes on getting Xen running on ESXi and I guess you ended up getting an answer to your question 😉

Quick Tip - Increasing capacity on a Nested VSAN Datastore

03.21.2014 by William Lam // 2 Comments

The other day I needed to increase the capacity on one of my Nested VSAN Datastores as one of our users required a larger VSAN datastore than it was initially configured for. I was expecting to be able to just increase the size of the underlying VMDKs like I would for a traditional Nested ESXi environment and rescan in ESXi to pick up the new capacity without any downtime. It turns out, this is was not exactly the case for a Nested VSAN environment.

Disclaimer: Nested Virtualization is not officially supported by VMware

When you first setup VSAN, regardless of how the disks were claimed, VSAN will consume the entire device (SSD or MD). The capacity that VSAN initially detects will then be used to create the necessary partition as part of the VSAN Disk Group creation. VSAN assumes that the capacity for the underlying devices would never change as in the "real" world, disks do not auto-magically get larger 🙂 and this is a valid assumption. In a Nested ESXi environment however, it can auto-magically get larger but VSAN was not built for this use case. What ends up happening is that the underlying devices can be "hot-extended" but the existing VSAN Disk Group can not detect this new capacity.

Having said that, there are two ways you can increase your VSAN datastore:

Option 1 - If you wish to preserve your VSAN Datastore, you can hot-add additional VMDK(s) to your existing VSAN Disk Group or if it is full, you can create a new disk group and add additional VMDK(s). This will modify your setup slightly if you wanted a particular set of disk groups but will allow you to preserve your data.

Option 2 - The latter option requires the deletion and re-creation of the VSAN Datastore which is not ideal if you already have data on it. You will need to increase the capacity of the underlying VMDKs and then re-create your VSAN Datastore, but this way you can keep the existing number of disks and disk groups you initially created your Nested ESXi environment with.

In my scenario, I could not destroy the VSAN Datastore as I had someone using it and so I opted for option #1. Here is what my configuration looked like before which was a single VSAN Disk Group with 1xSSD and 1xMD:

I then added an additional 10GB VMDK to each of my Nested ESXi hosts and issue a rescan so the ESXi host would pickup the new device:

In just a few seconds, I can see my new storage device. I can now head over to the VSAN management page which is located at the vSphere Cluster and once I refresh, I can see that VSAN has automatically added the new "MD" into the existing disk group and my storage has automatically expanded!