Nested Virtualization

Subscribe to vGhetto Nested ESXi Template Content Library in vSphere 6.0

04.08.2015 by William Lam // 23 Comments

During the early development of vSphere 6.0, one of the features that I got an early sneak preview of was the Content Library which originated from vCloud Director's Content Catalog capability and has now been pushed down into the core vSphere platform as part of VCD's "Convergence" plan.

Although there are some initial limitations with this first release of Content Library such as not being able to mount an ISO directly from the Content Library as example, which I do agree it should have just worked and not requiring a manual datastore browse for this operation to work. I know the Engineering team is aware of this as it was something our team had also provided feedback among other things, so hopefully this will be fixed very shortly.

Having said that, I do see a huge potential with the Content Library and all the interesting use cases it can enable not only for vSphere but also for other products such as vRealize Automation as well as vCloud Air. One area that caught my attention when I first heard about Content Library is the the fact that the publishing and subscription capability works over simple HTTP(s). I immediately had a light-bulb moment and thought would it not be cool if you could have a custom Content Library that would be hosted on some external cloud storage such as Amazon S3 as en example and be able to publish that so others could subscribe to it in their vSphere environment?

Fortunately, because Content Library works over standard HTTP(s) and with the help of one of the Content Library Engineers I was able to create my very own vGhetto Nested ESXi Template Content Library for both vSphere 6.0 and vCloud Directory based environments which is currently being hosted on Amazon S3.

vSphere: https://s3-us-west-1.amazonaws.com/vghetto-content-library/lib.json
vCD: https://s3-us-west-1.amazonaws.com/vghetto-content-library-vcd/lib.json

The library contains all of my Nested ESXi / VSAN OVF Templates that I have created over the years and by publishing them in in my public Content Library, anyone can now easily subscribe and pull down the latest OVF templates to deploy directly in their vSphere/vCD environment. You no longer have to manually download the OVFs and as I add new content, the Content Library will automatically synchronize the changes to your local environment.

The vGhetto Nested ESXi Template Content Library currently contains the following 7 OVF's which total to 1.43MB:

Nested-ESXi-3-Node-VSAN-6.0-All-Flash-Template
Nested-ESXi-3-Node-VSAN-6.0-Template
Nested-ESXi-3-Node-VSAN-Template
Nested-ESXi-32-Node-VSAN-Template (not available when subscribing from VCD)
Nested-ESXi-6-Node-VSAN-6.0-FD-Template
Nested-ESXi-64-Node-VSAN-6.0-Template (not available when subscribing from VCD)
Nested-ESXi-VM-Template

Here are the instructions for subscribing to my vGhetto Content Library using the vSphere 6.0 Web Client:

Step 1 - In the main Home page, click on the Content Library icon

Step 2 - Create a new Content Library which will be used to subscribe to my vGhetto Content Library.

Step 3 - Select "Subscribed content library" and enter the following URL: https://s3-us-west-1.amazonaws.com/vghetto-content-library/lib.json and make sure to select "Download all library content immediately". There is currently a limitation in the way the metadata is read from OVFs that prevents the on-demand setting to be used when subscribing to 3rd party Content Libraries.

Step 4 - Select the storage which will be backing your Content Library. This can be either a local filesystem path on your vCenter Server or by using a vSphere Datastore.

Step 5 - If the new Content Library had been successfully created, the content will start to synchronize to your local system and once the OVF/VM icon appears, then you know the Nested ESXi / VSAN OVFs templates are ready to be deployed in your environment as seen in the screenshot below.

Note: You will notice that even though we are pulling down all seven OVF templates, they are only consume a measly 1.43MB as these are empty VMs shells and I am also using the OVF Dynamic Disks feature which allows you to specify the size of the VMDK upon deployment.

UPDATE (4/24) - It turns out that 3rd Party Content Libraries can also be subscribed from within vCloud Director but because it does not support Dynamic Disks, the synchronized will fail. I have created a separate library specifically optimized for vCD which you can find the URL at the top and you can then subscribe to my vGhetto Content Library as seen in the screenshot below. You will notice that the 32 and 64 Node VSAN Template is not available and the reason for this is that apparently there is a limitation in the number of elements it can parse.

This is a pretty powerful feature in my opinion and I can already see custom content libraries not only from VMware but also from our partner eco-systems providing their latest solutions (Virtual Appliances) as well as other file content as Content Library can store pretty much anything. In a future blog post, I will go through the details on how you can create your own custom Content Library, so stay tune. In the mean time, if you are using vSphere 6.0 and and would like to be able to quickly pull down the latest Nested ESXi / VSAN OVF templates, be sure to subscribe to my vGhetto Content Library!

Quick Tip - Upgrading VMware Tools for Nested ESXi 6.0

04.03.2015 by William Lam // 2 Comments

I have received several questions about this in the last couple of weeks regarding the process of upgrading VMware Tools for running Nested ESXi 5.x and 6.0 when the physical ESXi host has been upgraded to ESXi 6.0. Instead of individual replies, I thought I would share this quick tip. First off, VMware Tools for Nested ESXi provides a very specific set of capabilities for Nested ESXi guests as shown below:

Provides guest OS information of the nested ESXi Hypervisor (eg. IP address, configured hostname, etc.).
Allows the nested ESXi VM to be cleanly shut down or restarted when performing power operations with the vSphere Web/C# Client or vSphere APIs.
Executes scripts that help automate ESXi guest OS operations when the guest’s power state changes.
Supports the Guest Operations API (formally known as the VIX API).

Unlike traditional VMware Tools which may provide updated capabilities with each new release, VMware Tools for Nested ESXi exposes only a subset of those capabilities which has not changed between ESXi 5.x and 6.0. This is an important fact to be aware of because you may see "Unsupported older version" for the VMware Tools status in the vSphere Web/C# Client and this is perfectly fine and expected.

Here is a screenshot of a Nested ESXi 5.5 VM with VMware Tools installed running on top of an upgraded physical ESXi 6.0 host:

In this scenario, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, you should not be alarm as the expected functionality listed above will continue to work without any problems and you can just ignore the UI warning. The only way to get rid of this warning is to upgrade the Nested ESXi VM to ESXi 6.0 which I go over in more details below. I know upgrading may not be an option if you still wish to run ESXi 5.x, but as far as I know, there will not be an update to VMware Tools VIB for ESXi 6.0 as it is now pre-installed with ESXi 6.0.

Here is a screenshot of the same VM which has now been upgraded to ESXi 6.0 running on top of an upgraded physical ESXi 6.0 host:

In this case, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, because VMware Tools now comes pre-installed with ESXi 6.0. We can easily remedy this by removing the VMware Tools VIB we installed for ESXi 5.x by running the following ESXCLI command and then rebooting:

esxcli software vib remove -n esx-tools-for-esxi

Once the ESXi host has rebooted, the VMware Tools that is pre-installed with ESXi 6.0 will automatically start up if it detects it is running as a VM. If you now look at your vSphere Web/C# Client, you will see that VMware Tools status shows current and is also the default behavior if you are running Nested ESXi 6.0 VM on top of physical ESXi 6.0 host.

With VMware Tools being pre-installed with ESXi 6.0 and only loaded when it detects it is being run as a VM, you no longer need to worry about manually installing additional VIBs get the benefits of having VMware Tools installed for your Nested ESXi VMs.

How to configure SMP-FT using Nested ESXi in vSphere 6?

03.06.2015 by William Lam // 1 Comment

Symmetric Multi-Processing Fault Tolerance (SMP-FT) has been a long-awaited feature by many VMware customers. With the release of vSphere 6.0, the SMP-FT capability is now finally available and if you want to try out this new feature and see how it works from a "functional" perspective, you can easily do so by running it in a Nested ESXi environment. SMP-FT no longer uses the "record/replay" capability like its younger brother Uniprocessing Fault Tolerance (UP-FT). Instead, SMP-FT now uses a new Fast Checkpointing technique which not only improves the overall performance of its predecessor but also greatly simplifies and reduces additional configurations when running in a Nested ESXi environment.

Disclaimer: Running SMP-FT in a Nested ESXi environment does not replace or substitute actual testing of physical hardware. For any type of performance testing, please test SMP-FT using real hardware.

Requirements:

pESXi host running either ESXi 5.5 or 6.0
vCenter Server 6.0
2 x Nested ESXi VMs running ESXi 6.0 (vHW9+)
Shared storage for the Nested ESXi VMs

Instructions:

Step 1 - Created a Nested ESXi VM using guestOS type "ESXi 5.5/6.0 or later". You will need at least 2 vCPU or greater, 4GB of memory or greater for the installation of ESXi and most importantly, a VMXNET3 network adapter. The reason a VMXNET3 adapter is required is that SMP-FT has a requirement for 10Gbit network connection and the VMXNET3 driver can simulate a 10Gbit connection for a Nested ESXi VM. For further instructions on creating a Nested ESXi VM, please take a look at this article. If you are unable to add VMXNET3 adapter, you may need to first change the guestOS type to "Other 64-bit", add the adapter and then change the guestOS type back.

Step 2 - Install ESXi 6.0 on the Nested ESXi VM and ensure you also have a vCenter Server 6.0 deployed if you have not done so already and add your Nested ESXi instances to a new vSphere Cluster which has vSphere HA enabled.

Step 3 - You will need to enable both vMotion and Fault Tolerance traffic type for the VMkernel interface that you wish to run FT traffic across.

Step 4 - At this point, you can create a real or dummy VM and power it on. Once you have the powered on VM, you can now enable either UP-FT or SMP-FT by right clicking and selecting "Enable Fault Tolerance".

As you can see from the screenshot above, I have successfully enabled FT on a VM with 4vCPU running inside of a Nested ESXi VM, how cool is that!? Hopefully this will help you get more familiar with the new SMP-FT feature when you are ready to give it a real spin on real hardware 🙂

Note: Intel Sandy Bridge is recommended when using SMP-FT (real physical hardware) but if you have older CPUs, you enable "Legacy FT" mode by adding the following VM Advanced Setting "vm.uselegacyft" to the VM you are enabling FT on.