WilliamLam.com

Ravello: An interesting solution using Nested Virtualization

by // 6 Comments

As many of you know, I am a huge fan of VMware Nested Virtualization and I am always interested to learn how customers and partners are using this technology to help enable them to solve interesting problems. I recently met up with a startup company called Ravello who has a product that leverages Nested Virtualization in a very unique way.

Note: Ravello is not the only company using Nested Virtualization in interesting ways. Bromium, another startup in the security space, is also doing interesting things with Nested Virtualization.

Ravello is a SaaS solution that allows you to take an existing VMware or KVM virtual machine and without any modifications to that VM, run it on a variety of public cloud infrastructures including Amazon EC2, HP Cloud, Rackspace and even private clouds that are running on vCloud Director (support coming soon). Ravello is basically "normalizing" the VM by virtualizing it in their Cloud Application Hypervisor so that it can run on any cloud infrastructure.  From the diagram below, the unmodified VM is actually running inside of another VM which runs a flavor of Linux. This Linux VM loads up their HVX Hypervisor and is running on one of the public cloud infrastructures.

Similar to a regular hypervisor, HVX provides an abstraction, but instead of the underlying physical hardware it abstracts away the underlying cloud infrastructure. The HVX hypervisor provides the following three core capabilities:

  • Presents a set of virtual hardware that is compatible with VMware ESXi, KVM and XEN virtual machines
  • Virtual networking layer that is a secure L2 overlay on top of the cloud infrastructure L3 networking using a protocol similar to GRE but running over UDP
  • Cloud storage abstraction that provides storage to the VM through Ravello Image Store that can be back-ended by Amazon S3, CloudFiles or even block/NFS volumes

My first thought after hearing how Ravello works, is that this is pretty neat! Of course the next logical question that I am sure most of you are asking is how is the performance? We know that running one level of Nested Virtualization will incur some performance penalty and this will continue with additional levels of Nested Virtualization. Ravello is also not leveraging Hardware-Assisted Virtualization but Binary Translation (a technique developed by VMware) as that can not be guaranteed to be available on all cloud infrastructures. In addition to Binary Translation, they are also using various techniques such as caching and chaining translated code, fast shadow MMU, direct execution of user space code and few others to efficiently run in a nested environment.

I was told that performance was still pretty good and sometimes even out performing regular cloud infrastructures. There was no mention of specific applications or performance numbers, so I guess this is something customers will need to validate in their own environment. I am also interested to see what the overhead is by doing two-levels of Nested Virtualization and what impact that has to the guestOS and more importantly, the applications. To be fair, Ravello's current target audience is Dev/Test workloads, so performance may not be the most critical factor. They also provide two modes of deployment based on cost optimized or performance and if the latter is selected, overcommitment of resources or consolidation will not be used.

Overall, I thought Ravello's solution was pretty interesting and could benefit some customers looking to run their workloads in other public cloud infrastructures. I think performance is just one of the things customers will need to consider but also how do they go about managing and operating this new VM container and how tightly integrated is Ravello with the VMware platform or other hypervisors for that matter. Though the VM and the underlying applications does not need to change, what operational challenges does this introduce to administrators? 

Ravello also recently presented their HVX Cloud Application Hypervisor at a recent USENIX conference and you can find more details in their presentation called HVX Virtualizing Cloud along with their research paper which can be found here.

One thing that I did want to point out after watching the presentation is that one of the presenter mentioned that their HVX nested hypervisor runs more efficiently than any other hypervisor out there and that others would require things like Intel's VMSC Shadowing feature to be comparable. I can not speak for other hypervisors, but when running VMware Hypervisors on top of our ESXi Hypervisor, our hypervisor has already been optimized for VMREAD/VMEXITS and Intel's VMSC Shadowing feature would only benefit slightly. You can read more about those techniques in this blog article.

Ravello will be at VMworld US booth #425 and I will probably drop by for a demo to see their solution in action.

Will Intel’s VMCS Shadowing Feature Benefit VMware’s Nested Virtualization?

by // 1 Comment

For many years now, VMware customers have been using Nested Virtualization, which is the ability to run a hypervisor such as vSphere ESXi within a virtual machine. Even though Nested Virtualization is not officially supported by VMware, customers have come to rely upon this technology for their lab environments and sometimes even production environments. VMware also heavily relies on this technology for their own internal development as well as their Hands On Lab for VMworld, which is now offered as an online SaaS (Software-as-a-Service) solution called Hands On lab Online.

Performance of Nested Virtualization has come a long way since its first introduction and it continues to get better with advancements made in hardware from both Intel and AMD. A couple of months back, I came across an article discussing a new feature from the upcoming Intel Haswell processor’s called VMCS Shadowing which aims to improve the performance of Nested Virtualization. This got me thinking about whether VMCS Shadowing could benefit VMware’s Nested Virtualization.

VMCS (Virtual Machine Control Structure) Shadowing works by reducing the frequency in which the guest VMM (virtual machine) requires assistance from the parent VMM. Its goal is to eliminate the VM-exits due to VMREAD and VMWRITE instructions executed by the guest hypervisor but this comes at a slight expense.

I reached out to one of the core engineers who helped to develop VMware’s Nested Virtualization technology, Jim Mattson, and asked whether or not we would benefit from the VMCS Shadowing feature. Well, it turns out that VMCS Shadowing can help, but we have also done some research in this area and developed some technology that would allow us to eliminate about 75% due to VMREAD and VMWRITE when running guest VMware Hypervisors using some interesting software techniques. The details of these software techniques are actually published in a research paper called Software Techniques for Avoiding Hardware Virtualization Exits on VMware’s Academic Program which is part of VMware Labs. Jim is one of the authors of the research paper and I would highly recommend you check it out if you are interested in more details.

To summarize, because of the techniques described in the paper, VMCS Shadowing will provide only a small benefit when running a VMware Hypervisor as virtual machine. However, it will greatly benefit other non-VMware Hypervisors running as a virtual machine, this is particular true for Hypervisors that perform egregious number of VMREAD and VMWRITE operations and that do not cluster well, such as VirtualBox for example.

The coolest part about the research and software techniques developed by Jim and team, is that the technology has already been incorporated into the existing VMware vSphere ESXi, Workstation and Fusion products. I often times forget that all the awesome-sauce technology that is being developed by VMware starts out in research academia and you can learn about other research topics by visiting the VMware’s Academic Program which includes publications, research papers and the popular VMware Technical Journals.