WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Virtual NVMe and Nested ESXi 6.5?

10.26.2016 by William Lam // 4 Comments

After publishing my Nested ESXi enhancements for vSphere 6.5 article, I had received a number of questions on whether the new Virtual NVMe (vNVMe) capability introduced in the upcoming vSphere 6.5 release would also work with a Nested ESXi VM? The answer is yes, similiar to PVSCSI and VMXNET3, we also have an NVMe driver for ESXi running in VM.

Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware, please use at your own risk.

To consume the new vNVMe for a Nested ESXi VM, you will need to use the latest ESXi 6.5 and later compatibility (vHW 13). Once that has been done, you can then add the a new NVMe Controller to your Nested ESXi VM and then assign that to one of the virtual disks as shown in the screenshot below.

nested-esxi-65-nvme-1
Next, you would install ESXi 6.5 as you normally would and the NVMe controller will automatically be detected and driver will be loaded. In the example below, you can see I only have a single disk which ESXi itself is installed on and it is backed by the NVMe Controller.

nested-esxi-65-nvme-0
One of the biggest benefit of using an NVMe interface over the traditional SCSI is that it can significantly reduce the amount of overhead compared to the SCSI protocol which in turn consumes less CPU cycles as well as reducing the overall amount of IO latency for your VM workloads. Obviously, when using it inside of a Nested ESXi VM, YMMV but hopefully you should also see an improvement there as well. For those who plan to give this a try in their environment, it would be good to hear what type of use cases you might have in mind for this and if you have any feedback (good/bad), feel free to leave a comment.

Categories // ESXi, Home Lab, Not Supported, vSphere 6.5 Tags // nested, Nested ESXi, nested virtualization, NVMe, vSphere 6.5

How to run a Docker Container on the vCenter Server Appliance (VCSA) 6.5?

10.24.2016 by William Lam // 8 Comments

One of the most notable changes in the vCenter Server Appliance (VCSA) in vSphere 6.5 is a switch of the underlying OS from SLES to VMware's very own Photon OS. With this change, VMware will now own the entire software stack within the VCSA (OS + Application). This will allow VMware to quickly respond and deliver OS and security updates to customers at a much quicker rate than it was possible before.

During my testing of the VCSA, I had a need to spin up a Docker Container. Given that the VCSA is now Photon OS based, this should be a pretty trivial thing to enable as it is with a standalone installation of Photon OS. After a bit of trial/error, I found what was needed to get this working on the VCSA. Before jumping into the solution, I should say that this is really for lab and educational purposes. In general, I would NOT recommend installing additional software on the VCSA, not only is this NOT supported by VMware but you may also potentially be impacting your vCenter Server by taking resources away from the main application. It is possible to constrain the amount of resources (CPU/Memory) allocated to the Docker Container, please refer to this resource for more information.

For smaller customers, the argument is that I can just run everything on a single system but in reality there are many benefits to having a separate management VM which can be Photon OS or any other OS that your organization supports. You can install additional management tools/scripts and you would not be artificially limited by the VCSA's environment which is really locked down to what is absolutely needed to run the vCenter Server application and its services.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Given that PowerCLI Core (Linux and Mac OS X) was just recently released, which also includes a Docker Container, I figure this would be a nice example to start with as I know a few of you have asked about this possibility 🙂

Step 1 - Install Docker by running the following command (you will need access to the internet either direct or proxy access from the VCSA)

tdnf -y install docker

Step 2 - Load the following kernel module which will allow us to start the Docker client by running the following command:

insmod /usr/lib/modules/$(uname -r)/kernel/net/bridge/bridge.ko

Note: The above command does not persist across reboots. If you would like to persist this configuration, please refer to the instructions at the very bottom.

Step 3 - Enable and start the Docker Client by running the following command:

systemctl enable docker
systemctl start docker

Step 4 - Pull down the PowerCLI Core Docker Image from Docker Hub by running the following command:

docker pull vmware/powerclicore

docker-container-on-vcsa-6-5-3
Step 5 - Start the PowerCLI Core Docker Container by running the following command:

docker run --rm -it --entrypoint='/usr/bin/powershell' vmware/powerclicore

docker-container-on-vcsa-6-5-4
As you can see from the screenshot above, you now have PowerShell and the PowerCLI module loaded running as a Docker Container on the VCSA 🙂 You can apply this to any Docker Container that you have created or pulling it directly from Docker Hub. If you prefer to build the PowerCLI Core Docker Container from the Dockerfile, you simply just need to download and extract the PowerCLI Core zip file onto the VCSA and then run the following command:

docker build -t vmware/powercli .

docker-container-on-vcsa-6-5-0

How to persist bridge module load across reboots:

Step 1 - Edit /etc/modprobe.d/modprobe.conf and remove the "install bridge /bin/false" entry.

Step 2 - Create a new file called /etc/modules-load.d/bridge.conf which contains the word "bridge" (no quotes). When the system boots up, it will iterate through all the module configuration file and load the respective modules. The bridge module is what is needed to start the Docker Daemon.

Categories // Automation, Docker, Not Supported, PowerCLI, VCSA, vSphere 6.5 Tags // Docker, Photon, vcenter server appliance, VCSA, vcva, vSphere 6.5

Nested ESXi Enhancements in vSphere 6.5

10.19.2016 by William Lam // 15 Comments

As many of you have probably heard by now, vSphere 6.5 was just announced at VMworld Barcelona this week and it is packed with a ton of new features and capabilities which you can read more about here. One area that is near and dear to me and has not been covered are some of the Nested ESXi enhancements that have been made in vSphere 6.5.

To be clear, VMware has NOT changed its support stance in vSphere 6.5. Both Nested ESXi as well as general Nested Virtualization is still NOT officially supported. Okay, so thats out of the way now, lets see what is new?

  • Paravirtual SCSI (PVSCSI) support
  • GuestOS Customization
  • Pre-vSphere 6.5 enablement on vSphere 6.0 Update 2
  • Virtual NVMe support

Lets take a closer look at each of these enhancements.

Paravirtual SCSI (PVSCSI) support

When vSphere 6.0 Update 2 was released, I had hinted that PVSCSI support might be a possibility in the near future. I am happy to announce that this is now possible with running Nested ESXi and having the guestOS as ESXi 6.5. In vSphere 6.5, a new GuestOS type has been introduced called vmkernel65 which is optimized for running ESXi 6.5 in a VM as shown in the screenshot below.

nested-esxi-enhancements-in-vsphere-6-5-4
As you can see from the VM Virtual Hardware configuration screen below, both the PVSCSI and VMXNET3 adapter are now the recommended default when creating a Nested ESXi VM for running ESXi 6.5.

nested-esxi-enhancements-in-vsphere-6-5-1
Similiar to the VMXNET3 driver, the PVSCSI driver is automatically bundled within the version of VMware Tools for Nested ESXi. That is to say, the drivers are included in the default ESXi image itself and is ONLY activated when ESXi detects that it is running inside of a VM. From a user standpoint, this means there are no additional configurations or installations that is required. You simply select ESXi 6.5 as the GuestOS and install ESXi as you normally would and this will automatically be enabled for you.

The only requirement to leverage this new capability is that BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. The underlying physical ESXi host can either be ESXi 6.0 or ESXi 6.5. In addition to new virtual hardware defaults, I have also found that the new ESXi 6.5 GuestOS type now uses EFI firmware over the legacy BIOS compared to previous ESXi 6.x/5.x/4.x GuestOS types.

For customers who wish to push their storage IO a bit more for Nested ESXi guests, this is a great addition, especially with lower overhead when using a PVSCSI adapter.

GuestOS Customization

One of the very last capability that has been missing from Nested ESXi is the ability to perform a simple GuestOS customization when cloning or deploying a VM from template running Nested ESXi. Today, you can deploy my Nested ESXi Virtual Appliance which basically provides you with the ability to customize your deployment but would it not be great if this was native in the platform? Well, I am pleased to say this is now possible!

When you go and clone or deploy a VM from template that is a Nested ESXi VM, you will now have the option to select the Customize guest OS option. As you can see from the screenshot below, you can now create a new Customization Spec which is based on the Linux customization spec. The customization only covers networking configuration (IP Address, Netmask, Gateway and Hostname) and only applies it to the first VMkernel interface, all others will be ignored. The thinking here is that once you have your Nested ESXi VM on the network, you can then fully manage and configure it using the vSphere API rather than re-creating the same functionality just for cloning.

nested-esxi-enhancements-in-vsphere-6-5-2
To use this new Nested ESXi GuestOS customization, there are two things you will need to do:

  • Perform two configuration changes within the Nested ESXi VM which will prepare them for cloning. You can find the configuration changes described in my blog post here
  • Ensure BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. This means that your underlying physical vSphere infrastructure can be running either vSphere 6.0 Update 2 or vSphere 6.5

You can monitor the progress of the guest customization by going to the VM's Monitor->Tasks & Events using the vSphere Web Client or vSphere API if you are doing this programmatically. Below is a screenshot of a successful Nested ESXi guest customization. If there are any errors, you can take a look at /var/log/vmware-imc/toolsDeployPkg.log within the cloned Nested ESXi VM to determine what went wrong.

nested-esxi-enhancements-in-vsphere-6-5-3
I know this will be a very welcome capability for customers who extensively use the guest customization feature or if you just want to quickly clone an existing Nested ESXi VM that you have already configured.

Pre-vSphere 6.5 enablement in vSphere 6.0 Update 2

By now, you probably have figured out what this last enhancement is all about 🙂 It is exactly as it sounds, we are enabling customers to try out ESXi 6.5 by running it as a Nested ESXi VM on your existing vSphere 6.0 environment and specifically the Update 2 release (this includes both vCenter Server as well as ESXi). Although this has always been possible with past releases of vSphere running newer versions, we are now pre-enabling ESXi 6.5 specific Nested ESXi capabilities in the latest release of vSphere 6.0 Update 2. This means when vSphere 6.5 is generally available, you will be able to test drive some of the new Nested ESXi 6.5 capabilities that I had mentioned on your existing vSphere infrastructure. This is pretty darn cool if you ask me!?

Virtual NVMe support

I had a few folks ask on whether the upcoming Virtual NVMe capability in vSphere 6.5 would be possible with Nested ESXi and the answer is yes. Please have a look at this post here for more details.

For those of you who use Nested ESXi, hopefully you will enjoy these new updates! As always, if you have any feedback or requests, feel free to leave them on my blog and I will be sure to share it with the Engineering team and who knows, it might show up in the future just like some of these updates which have been requested from customers in the past 😀

Categories // ESXi, Nested Virtualization, vSphere 6.5 Tags // guest customization, nested, Nested ESXi, nested virtualization, pvscsi, vmxnet3, vSphere 6.0 Update 2, vSphere 6.5

  • « Previous Page
  • 1
  • …
  • 24
  • 25
  • 26
  • 27
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...