As many of you know, I am a huge fan of VMware Nested Virtualization and I am always interested to learn how customers and partners are using this technology to help enable them to solve interesting problems. I recently met up with a startup company called Ravello who has a product that leverages Nested Virtualization in a very unique way.
Note: Ravello is not the only company using Nested Virtualization in interesting ways. Bromium, another startup in the security space, is also doing interesting things with Nested Virtualization.
Ravello is a SaaS solution that allows you to take an existing VMware or KVM virtual machine and without any modifications to that VM, run it on a variety of public cloud infrastructures including Amazon EC2, HP Cloud, Rackspace and even private clouds that are running on vCloud Director (support coming soon). Ravello is basically "normalizing" the VM by virtualizing it in their Cloud Application Hypervisor so that it can run on any cloud infrastructure. From the diagram below, the unmodified VM is actually running inside of another VM which runs a flavor of Linux. This Linux VM loads up their HVX Hypervisor and is running on one of the public cloud infrastructures.
Similar to a regular hypervisor, HVX provides an abstraction, but instead of the underlying physical hardware it abstracts away the underlying cloud infrastructure. The HVX hypervisor provides the following three core capabilities:
- Presents a set of virtual hardware that is compatible with VMware ESXi, KVM and XEN virtual machines
- Virtual networking layer that is a secure L2 overlay on top of the cloud infrastructure L3 networking using a protocol similar to GRE but running over UDP
- Cloud storage abstraction that provides storage to the VM through Ravello Image Store that can be back-ended by Amazon S3, CloudFiles or even block/NFS volumes
My first thought after hearing how Ravello works, is that this is pretty neat! Of course the next logical question that I am sure most of you are asking is how is the performance? We know that running one level of Nested Virtualization will incur some performance penalty and this will continue with additional levels of Nested Virtualization. Ravello is also not leveraging Hardware-Assisted Virtualization but Binary Translation (a technique developed by VMware) as that can not be guaranteed to be available on all cloud infrastructures. In addition to Binary Translation, they are also using various techniques such as caching and chaining translated code, fast shadow MMU, direct execution of user space code and few others to efficiently run in a nested environment.
I was told that performance was still pretty good and sometimes even out performing regular cloud infrastructures. There was no mention of specific applications or performance numbers, so I guess this is something customers will need to validate in their own environment. I am also interested to see what the overhead is by doing two-levels of Nested Virtualization and what impact that has to the guestOS and more importantly, the applications. To be fair, Ravello's current target audience is Dev/Test workloads, so performance may not be the most critical factor. They also provide two modes of deployment based on cost optimized or performance and if the latter is selected, overcommitment of resources or consolidation will not be used.
Overall, I thought Ravello's solution was pretty interesting and could benefit some customers looking to run their workloads in other public cloud infrastructures. I think performance is just one of the things customers will need to consider but also how do they go about managing and operating this new VM container and how tightly integrated is Ravello with the VMware platform or other hypervisors for that matter. Though the VM and the underlying applications does not need to change, what operational challenges does this introduce to administrators?
Ravello also recently presented their HVX Cloud Application Hypervisor at a recent USENIX conference and you can find more details in their presentation called HVX Virtualizing Cloud along with their research paper which can be found here.
One thing that I did want to point out after watching the presentation is that one of the presenter mentioned that their HVX nested hypervisor runs more efficiently than any other hypervisor out there and that others would require things like Intel's VMSC Shadowing feature to be comparable. I can not speak for other hypervisors, but when running VMware Hypervisors on top of our ESXi Hypervisor, our hypervisor has already been optimized for VMREAD/VMEXITS and Intel's VMSC Shadowing feature would only benefit slightly. You can read more about those techniques in this blog article.
Ravello will be at VMworld US booth #425 and I will probably drop by for a demo to see their solution in action.
Looking forward to meeting you at VMworld.
BTW, nested virtualization is one element of the solution. The application centric networking/ IO overlay is just as critical. It enables enterprises to encapsulate entire multi-tier, multi-VM applications and deploy them on any cloud - public or private - without making any changes (no changes to the VM, devices, drivers, networking etc.).
In a sense this is the same as what VMware first did for a single VM (encapsulating and abstracting the server), but taken one level up, and applied to the whole multi-tier application environment. Another benefit of encapsulating the whole "application" is the ability to provision copies of it with one click/API call. Now the entire application becomes the management unit as opposed to individual VMs and infrastructure. As an example of its benefits - consider how organizations will be able to do continuous integration on replicas of production and take agile development to the next level. The moment a developer checks in code, the system can automatically spin up a replica of the production environment in any cloud, run the tests and shut the whole thing down.
--
Navin R. Thadani
SVP, Products
Ravello Systems, Inc.
Twitter: @navinthadani
William, good point on questions about nested virtualization performance. Check out this post with performance benchmarks: http://www.ravellosystems.com/blog/nested-virtualization-shootout-ravello-vmware-qemu/
A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work.
============================================
cloudmaven
On overheads:
http://www.theregister.co.uk/2013/02/05/ravello_systems_cloud_hypervisor/
Thadani says that on CPU-bound applications, the overhead imposed by HVX can be as little as 5 per cent and as much as 30 per cent compared to native KVM or Xen hypervisors, and that on I/O-bound workloads, the overhead can be quite a bit higher. So this is no free lunch, at least until HVX features are added to x86 processors.
Very interesting article with great useful information. I am also sharing this article with my friends you have written excellent content. Thanks for sharing this kind of informative blog.