For many years now, VMware customers have been using Nested Virtualization, which is the ability to run a hypervisor such as vSphere ESXi within a virtual machine. Even though Nested Virtualization is not officially supported by VMware, customers have come to rely upon this technology for their lab environments and sometimes even production environments. VMware also heavily relies on this technology for their own internal development as well as their Hands On Lab for VMworld, which is now offered as an online SaaS (Software-as-a-Service) solution called Hands On lab Online.
Performance of Nested Virtualization has come a long way since its first introduction and it continues to get better with advancements made in hardware from both Intel and AMD. A couple of months back, I came across an article discussing a new feature from the upcoming Intel Haswell processor’s called VMCS Shadowing which aims to improve the performance of Nested Virtualization. This got me thinking about whether VMCS Shadowing could benefit VMware’s Nested Virtualization.
VMCS (Virtual Machine Control Structure) Shadowing works by reducing the frequency in which the guest VMM (virtual machine) requires assistance from the parent VMM. Its goal is to eliminate the VM-exits due to VMREAD and VMWRITE instructions executed by the guest hypervisor but this comes at a slight expense.
I reached out to one of the core engineers who helped to develop VMware’s Nested Virtualization technology, Jim Mattson, and asked whether or not we would benefit from the VMCS Shadowing feature. Well, it turns out that VMCS Shadowing can help, but we have also done some research in this area and developed some technology that would allow us to eliminate about 75% due to VMREAD and VMWRITE when running guest VMware Hypervisors using some interesting software techniques. The details of these software techniques are actually published in a research paper called Software Techniques for Avoiding Hardware Virtualization Exits on VMware’s Academic Program which is part of VMware Labs. Jim is one of the authors of the research paper and I would highly recommend you check it out if you are interested in more details.
To summarize, because of the techniques described in the paper, VMCS Shadowing will provide only a small benefit when running a VMware Hypervisor as virtual machine. However, it will greatly benefit other non-VMware Hypervisors running as a virtual machine, this is particular true for Hypervisors that perform egregious number of VMREAD and VMWRITE operations and that do not cluster well, such as VirtualBox for example.
The coolest part about the research and software techniques developed by Jim and team, is that the technology has already been incorporated into the existing VMware vSphere ESXi, Workstation and Fusion products. I often times forget that all the awesome-sauce technology that is being developed by VMware starts out in research academia and you can learn about other research topics by visiting the VMware’s Academic Program which includes publications, research papers and the popular VMware Technical Journals.
