WilliamLam.com

How to properly clone a Nested ESXi VM?

by // 53 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349 

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

  • The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
  • The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

w00t! VMware Tools for Nested ESXi!

by // 42 Comments

I have been working with Nested ESXi since it original inception and this technology has greatly benefited me and the entire VMware community, especially when it comes to learning about VMware software and being able to easily prototype something before installing it on actual hardware. However, one thing that I felt that has been missing for awhile now is the ability to run an instance of VMware Tools within a Nested ESXi VM. I have personally been asking for this feature for a couple of years and I know many in the VMware community have expressed interests as well.

I am super excited to announce that VMware has just released a new Fling that provides you with a VIB that you can install VMware Tools inside a Nested ESXi host. I originally showed a demo of this at VMworld Barcelona in my vBrownBag Tech Talk and as I mentioned we would be releasing this as a VMware Fling very soon. So here it is!

UPDATE (08/20/15) - An updated version of VMware Tools for Nested ESXi was just published, make sure to download latest version and you can find more details here.

Requirements:

  • Nested ESXi running 5.0, 5.1 or 5.5 

Installation:

To install the VIB, you simply just need to download it and upload the VIB it to your Nested ESXi datastore and then run the following commands:

esxcli system maintenanceMode set -e true
esxcli software vib install -v /vmfs/volumes/[VMFS-VOLUME-NAME]/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f
esxcli system shutdown reboot -r "Installed VMware Tools"

You can also install the VIB directly from VMware.com if you have direct or proxy internet connectivity from your ESXi host by running the following commands:

esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi_tools_for_guests/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f

Once the VIB has been successfully installed, you will need to reboot the host for the changes to take effect. To verify, you can now login to either your vSphere Web/C# Client and you should now see the status for VMware Tools for your Nested ESXi host showing green and the IP Address of the Nested ESXi host should be displayed.

So why would you want to do this? Well, there’s a couple of reasons actually. The first one is pretty basic, which is when I need to reboot or shutdown a Nested ESXi VM, instead of having to jump into the VM console or SSH into ESXi host, I could just right click in the vSphere Web/C# Client and just say shutdown or reboot. I also tend to do all sorts of craziness in my lab (I’m sure this is an understatement for folks that know me) and may often break networking connectivity to my Nested ESXi VM. In vSphere 5.0, we introduced the Guest Operations API (formally known as VIX API) which is now part of the vSphere API. This API is actually quite handy as it allows you to perform guest operations within the VM without needing network connectivity as it relies on the fact that VMware Tools is running (pretty cool stuff!).

Here is a screenshot demonstrating the executing of vmkfstools through the Guest Operations API to one of my Nested ESXi VM:

A couple of things to note:

  • If you install VMware Tools on Nested ESXi VM, you will NOT be able to just right click in the UI and say install/upgrade
  • If you wish to integrate this into you ESXi image, you can take a look at a community tool  called ESXi-Customizer created by Andreas Peetz which I have used in the past and works great. Image Builder does not support raw VIBs, only zip files which may need to contain additional metadata information. If you want to create an offline bundle instead to then use Image Builder to create your custom ISO, Andreas has a new tool you can take a look at here.

Finally, if you have any feedback (likes/dis-likes), thanks, comments please head over to the VMware's Fling page for VMware Tools for Nested ESXi and leave a comment. I am sure the Jim Mattson the engineer who built this Fling would greatly appreciate any feedback you may have.

How to run Nested ESXi on top of a VSAN datastore?

by // 35 Comments

Today I found an interesting article on my Twitter timeline regarding some issues when trying to install and run Nested ESXi on top of a VSAN datastore. Shortly after the ESXi installation begins, the following error message is observed:

This program has encountered an error:

Error (see log for more info):
Could not format a vmfs volume.
Command ‘/usr/sbin/vmkfstools -C vmfs5 -b 1m -S datastore1
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:3′ exited with status 1048320

This of course piqued my interest given the topic and I would have expected this to just work and thought this might have been some miss-configuration. I decided to try this out in my lab and to my surprise, I encountered the exact same problem.

Here is a quick screenshot of error message:

I pinged a couple of folks from the VSAN development team to see if this was a known issue and if so, why was it occurring? After a couple of email exchanges, it turns the problem is with a SCSI-2 reservation being generated as part of creating a default VMFS datastore. Even though VMFS-5 no longer uses SCSI-2 reservations, the underlying LVM (Logical Volume Manager) driver for VMFS still requires it. Since VSAN does not make use of SCSI-2 reservations, it did not make sense to support it and hence the issue. Having said that, since Nested Virtualization is heavily used at VMware, the VSAN development team has come up with a nifty solution as they too hit this problem early on during the development of VSAN. Big thanks to Christian Dickmann (Tech Lead on the VSAN Engineering team) for providing this little tidbit.

Disclaimer: Nested Virtualization is not officially supported by VMware nor are the configuration changes described below, please use at your own risk.

To get around this problem, the VSAN team added in an advanced ESXi setting that would "fake" SCSI Reservations and this needs to be configured for the physical ESXi hosts providing up the physical VSAN datastore.

Run the following ESXCLI command (either locally on ESXi Shell or remotely)

esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1

A system reboot is not required and this change can be done live on the ESXi host prior to starting the ESXi installation. Once this is done, you will now be able to proceed with installing Nested ESXi on top of a VSAN datastore. Here is a screenshot of my Nested ESXi VM running on top of Nested ESXi VSAN datastore 🙂

Hopefully this workaround will be useful for anyone running VSAN and would like to fully make use of this storage by running Nested ESXi for development or testing.