WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / ESXi / How to properly clone a Nested ESXi VM?

How to properly clone a Nested ESXi VM?

12.06.2013 by William Lam // 52 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349 

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

  • The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
  • The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

More from my site

  • VM serial logging to the rescue for capturing Nested ESXi PSOD
  • vSphere 6.0 Update 2 hints at Nested ESXi support for Paravirtual SCSI (PVSCSI) in the future
  • Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance
  • VMware Tools for Nested ESXi updated to v1.2
  • How to run Qemu & KVM on ESXi?

Categories // ESXi, Nested Virtualization Tags // clone, esxi, nested, nested virtualization, uuid

Comments

  1. Jim Millard says

    12/07/2013 at 4:45 pm

    These tips can also apply to cloning boot LUNs/volumes in physical hosts that use SAN instead of local disks for boot.

    Reply
  2. rnelson0wordpress1 says

    12/11/2013 at 4:12 am

    Good stuff, playing around with this and dynamic disks to create a golden OVF for nested ESXi in my home lab. Just in time for Christmas!

    Since it's really tiny and can't C&P from an image, here's the command to check the FollowHardwareMac setting:

    esxcli system settings advanced list -o /Net/FollowHardwareMac

    Reply
  3. Manfred says

    05/06/2014 at 12:33 am

    Hello William,
    great Post.

    I tried the "esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]" command but do get the message "No unresolved VMFS snapshots with volume label 'esxi2-local' found".

    Did I miss something?

    Thanks
    Manfred

    Reply
    • Akira Z says

      07/16/2014 at 11:14 am

      The part of resignaturing the vmfs volume is apparently incorrect since the esxi does not see the vmfs as copied or snapshoted thus resignaturing cannot be applied. The vmfs is indeed snapshoted at vmware workstation level instead of the esxi level.

      The only way I know that is working is as below, this method requires recreating the local vmfs datastore, hence all data on it WILL LOST.

      1. # vmkfstools --queryfs -h /vmfs/volumes/
      take note of the "Partitions spanned", it's kind like mpx.vmhbal:C0:T0:L0:3 (refered to as hereafter).

      2. # esxcli storage filesystem unmount -l
      make sure the local datastore is unmounted.

      3. # vmkfstools --createfs vmfs5 --blocksize 1m --setfsname /dev/disks/
      this will recreate the vmfs on the partition and give it a new UUID.

      is the local datastore label you want to have the UUID changed, mine was "datastore1". is the new datastore label, you can specify the same as .

      Reply
      • Giuliano says

        11/29/2014 at 10:28 pm

        Did you try these commands you're referring to? They don't seem so right...

        Reply
        • Jack says

          02/22/2015 at 11:13 am

          This should work:

          # enter maintenanceMode mode
          esxcli system maintenanceMode set --enable true --timeout=1

          # determine partition device (look under "Partitions spanned (on "lvm")" for mpx.)
          vmkfstools -P /vmfs/volumes/

          # dismount datastore by label
          esxcli storage filesystem unmount -l

          # recreate FS
          /usr/sbin/vmkfstools -C vmfs5 -b 1m -S /vmfs/devices/disks/mpx.

          # reboot system
          esxcli system shutdown reboot --reason="Updated System UUID"

          # exit maintenance mode
          esxcli system maintenanceMode set --enable false --timeout=1

          Reply
        • Jack says

          02/22/2015 at 11:17 am

          Now I see the problem.. this brain-damaged commenting system is purging everything in chevrons.. let's try one last time with parenthesis..

          # enter maintenanceMode mode
          esxcli system maintenanceMode set --enable true --timeout=1

          # determine partition device (look under "Partitions spanned (on "lvm")" for mpx.(something))
          vmkfstools -P /vmfs/volumes/(datastore-label)

          # dismount datastore by label
          esxcli storage filesystem unmount -l (datastore-label)

          # recreate FS
          /usr/sbin/vmkfstools -C vmfs5 -b 1m -S (newlabel) /vmfs/devices/disks/mpx.(something)

          # reboot system
          esxcli system shutdown reboot --reason="Updated System UUID"

          # exit maintenance mode
          esxcli system maintenanceMode set --enable false --timeout=1

          Reply
          • Arun says

            04/11/2015 at 6:59 am

            Above steps worked. Thanks for the solution

          • Mack says

            02/05/2016 at 7:37 am

            Dos this method erase data from datastore1. I cloned ESXi server2 from ESXi server1 by taking one boot disk out from server1 and then inserted on server2 and booted and works fine. But when I added ESX server1 and server2 in vcenter it shows as datastore1(22) mounted for both server1 & server2. I do not want to loose the data. Does this method erase data if I follow this procedure on ESXi server2?

  4. Mike P says

    08/15/2014 at 2:25 pm

    I created a virtual ESXi template using the listed FollowHardwareMac setting and deleted the UUID as described. This works great for vmk0. It seems to regenerate properly every time I deploy a new copy of the template. My template also has several other vmk interfaces as well. What I am finding though is that other any other vmks (except for vmk0) that are part of the cloned VM still retain the same MAC as the original VM after being cloned from the template. Do you have any ideas on this?

    Reply
  5. Gaurav Pruthi says

    09/04/2014 at 8:58 am

    Worked for me even after cloning. Thanks a ton !!

    Reply
  6. Joshua O'Brien says

    10/29/2014 at 4:36 pm

    I was able to successfully clone nested ESX by doing a standard clone then performing a factory reset inside ESXi.

    Reply
    • Ben Spencer says

      05/23/2019 at 5:43 pm

      The factory reset worked for me too, thanks!

      I created a cluster with VCSA 6.0 and ESXi 6.0, and could not get vSAN to run. After the factory reset, it worked.

      Reply
  7. deva says

    10/13/2015 at 10:37 am

    Hi,

    Your article is very useful to me and solved 80% of my problem, But I am not able to add clone vESXi to same VC.

    I ran your command "esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]". I am getting error. Can you please help me to fix this issue. How to change UUID of datastore for base image. So that Cloned vESXi will not get same UUID for datastore

    Reply
  8. JenBell says

    07/16/2016 at 9:54 am

    1. Entered follow hardware command via SSH to ESXi nested host.
    2. Used vi to remove UUID.
    3. Shutdown host.
    4. Using VC - get VMDK error.
    I am either missing a step or the solution does not work for 5.5. Can someone help?

    Reply
    • William Lam says

      07/16/2016 at 2:52 pm

      This works on both 5.x & 6.x. What exact error message are you seeing, as VMDK error is not specific enough

      Reply
      • JenBell says

        07/17/2016 at 7:31 am

        Hello William. Thank you for responding. I've blanked the config but it appeared to be an issue related to copying the VMDK. What I tried was manually copying the files to another folder, booted the ESXi 5.5 nested. ticked the option I copied it when it showed up. From the other posts under your main post, the it seems the local datastore (1) is linked via UUID or something. I tried doing the resig but SSH CLI was telling me nothing needed to be resig'ed. It would be really nice if you literally show us mere mortals a spoon-fed step by step of everything you did. Being able to clone an ESXi nested image would make my life much easier with VC.

        Reply
        • William Lam says

          07/17/2016 at 7:34 am

          I'm not sure why you're manually copying the VMDK? If you have vCenter Server, you just need to run through the preparation as shown in the blog post and then right click and select "Clone"

          Reply
          • Madhu says

            06/05/2017 at 7:31 am

            Hi William,

            Can you help in in the below situation please.

            When i follow the above steps am getting the below error while trying to generate new SSID.

            I am running this command: /usr/sbin/vmkfstools -C vmfs5 -b 1m -S name1 /vmfs/devices/disks/mpx.vmhba1:
            C0:T0:L0

            ends up saying
            Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
            Creating vmfs5 file system on "mpx.vmhba1:C0:T0:L0" with blockSize 1048576 and volume label "name1".
            /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0: Permission denied. (Have you set the partition type to 0xfb?)
            Error: Permission denied

            I have logged in as root.. but it syas permission denied.

            Regards,
            Madhu

  9. Niamh says

    01/18/2017 at 4:11 am

    We are cloning nested ESXi in vCloud Director. What doesn't seem to change on each linked clone is the bios.uuid which is causing is issues. We've done all the above and while system uuid is changed, the bios.uuid is not changing. Any suggestions? thanks!

    Reply
    • William Lam says

      01/18/2017 at 6:01 am

      Niamh,

      Have a look at https://kb.vmware.com/kb/2002506 for your answer

      Reply
  10. Niamh says

    01/30/2017 at 3:24 am

    thanks William, yep working on it now !!

    Reply
  11. Ajay Nadakuditi says

    02/08/2017 at 1:11 pm

    Hi will, Is it possible to strip the VMFS UUID before cloning?

    Reply
  12. Imran Khan says

    11/14/2017 at 11:24 pm

    Very Informative article. Thanks alot. It worked well. Other than following command (entering mantenance mode in CLI dint worked...) But that i easily managed by connecting directly and doing in GUI.

    # enter maintenanceMode mode
    esxcli system maintenanceMode set –enable true –timeout=1

    Reply
    • leemarzke says

      08/26/2018 at 4:18 pm

      I'm following your excellant article, and cloning an ESX 6.5 host.

      I've run "esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1" in the template, then cloned with customization wizard. The Management vmk0 kernel got a new Mac and assigned IP, however the vMotion vmk1, and NFS vmk2 did not get a new Mac, or the new assigned IP from the Wizard so I set them manually.

      I've confirmed in the vCenter dvs Port listing that my three cloned hosts have different Mgmt Mac's but duplicate Mac's on the vMotion and NFS vmkernel ports. This explains why I'm having so many intermittent network issues.

      Is the Customization wizard and FollowHardwareMac known to work for non Mgmt kernel ports on ESX 6.5?

      Lee

      Reply
  13. Mack says

    09/05/2018 at 6:48 am

    Hi,

    I have used clone method when I initially built ESXi 5.1. Then I was able to upgrade it to 5.5 with no issue. Now that I am upgrading to ESXi 6.5, its failing adn I am getting a purple screen and I see following error when I look at the debug output. "The System has found a problem on your machine and cannot continue. System with same UUID have been detected. Make sure you do not have two ESXi installation". Also I see message "cpu35:65895)FSS: 5817: Detected Duplicate image on device 'naa.600605b00aa3a4901f08161c4c258e19:5". I have two data stores. One for OS and one for APPL. this 19:5 is for APPL DS but I could not find such device 19:5 on my server.
    Any idea? I came across many blogs but so far none has helped including the one from vmware to identify why its finding this as duplicate image.

    Reply
  14. Ranjith says

    02/26/2020 at 3:02 am

    Thanks for details. I have created a nested esxi for the DHCP enabled VLAN,but for the new cloned nexted esxi I am not getting proper IP and getting some private IP. any idea?

    Reply
  15. AS says

    10/29/2020 at 10:10 am

    Hello there!
    Great text!
    I wonder, is it necessary to install vmware tools to nested ESXi 7.0 and later?
    I ask because you mention it at the end of the artile and also because there is a light verson of tools already installed from scratch (tools-light - 11.1.1.16303738-16850804).
    Thanks for the article and reply!

    Cheers!
    AS

    Reply
  16. Salvatore Saporito says

    03/29/2021 at 7:52 am

    Hi William,
    I create two esxi 7.0u2 nested in ESXi 6.7 U3.
    I try to run ping between the mgmt network (vmk0) of the nested esxi...
    If I try to add mgmt network redundancy, I'm unable to communicate with vmk0 until I remove one nic in teaming.

    I run this command:
    esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

    In my nested esxi, the vmk0 mac address is the same of "physical" vmnic.
    (could this be the problem?)...I did not find the mac address of vmk0 in the esx.conf file
    My nested esxi has 6 vmnic vmxnet3.

    Can you tell me what can i do ?

    Thanks a lot..

    Reply
  17. MikeP says

    07/17/2021 at 8:59 pm

    William, with the UUID related changes in 7.0 U2 is it safe to deploy multiple nested ESXi hosts from your OVA template into the same environment or will each of them have the same UUID and cause corruption issues?

    I recently had several iSCSI VMFS datastores shared by physical and nested hosts seemingly get corrupted. They were reporting in the vmkernel log that they were snapshots and wouldn't auto-mount during boot for both physical and nested ESXi hosts. Only way to mount them was via CLI and they would disappear after every reboot. I had to delete them and recreate them to resolve the problem but I never really figured out the root cause. This sounds like it could have been the root cause since I had 3 nested 7.0U2 VMs all deployed from your template that were mounting those datastores.

    Reply
    • William Lam says

      07/19/2021 at 2:07 pm

      Mike,

      I'm not aware of any issues with the latest 7.0u2a OVA, I know quite a few folks have it deployed including myself and haven't ran into any problems. If the ESXi host has duplicated UUID and has local datastore (which latest 70u2a OVA defaults to) and you attempt to add both hosts to vCenter Server, you'll normally see it complain and fail to add. You may want to double check that your physical host boot LUN wasn't somehow cloned. In theory, you should see the problem with just two Nested ESXi VMs, so just make sure these were all deployed fresh from OVA. If you're still having issues, just let me know

      Reply
  18. sangdeuk says

    11/22/2021 at 6:00 pm

    I am using ESXi 7.0.3 now.

    after clone nested vm without snapshot.

    Though I try to change datastore UUID, I can't chang local datastore.
    What should I do ?

    [[email protected]:~] vmkfstools -P /vmfs/volumes/5fa8a747-f9af54c7-a815-005056b1e7c8
    VMFS-L-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
    File system label (if any): OSDATA-5fa8a747-f9af54c7-a815-005056b1e7c8
    Mode: public
    Capacity 34091302912 (32512 file blocks * 1048576), 30833377280 (29405 blocks) avail, max supported file size 70368744177664
    Disk Block Size: 512/512/0
    UUID: 5fa8a747-f9af54c7-a815-005056b1e7c8
    Partitions spanned (on "lvm"):
    mpx.vmhba0:C0:T0:L0:7
    Is Native Snapshot Capable: NO
    [[email protected]:~] vmkfstools -P /vmfs/volumes/5fa8a747-f9af54c7-a815-005056b1e7c8
    VMFS-L-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
    File system label (if any): OSDATA-5fa8a747-f9af54c7-a815-005056b1e7c8
    Mode: public
    Capacity 34091302912 (32512 file blocks * 1048576), 30833377280 (29405 blocks) avail, max supported file size 70368744177664
    Disk Block Size: 512/512/0
    UUID: 5fa8a747-f9af54c7-a815-005056b1e7c8
    Partitions spanned (on "lvm"):
    mpx.vmhba0:C0:T0:L0:7
    Is Native Snapshot Capable: NO

    Reply
  19. Jason Kirk says

    01/24/2022 at 9:02 am

    All you have ever had to do to clone an ESXi VM or a Boot from SAN LUN is this:
    Login to the DCUI
    Select "Reset System Configuration"
    Power down instead of reboot
    If it's a boot from SAN LUN, unmap it.
    Clones will not have duplicate MACs or UUIDs.

    Reply

Trackbacks

  1. VMware VSAN APIs | virtuallyGhetto says:
    03/03/2014 at 2:53 pm

    […] ESXi host provides a vsanSystem manager at configManager->vsanSystem which provides the following methods for disk […]

    Reply
  2. Provisioning Nested vSphere Hosts with PowerCLI | Sean Crookston says:
    04/07/2014 at 2:47 pm

    […] you have a nested vSphere template available which has been prepared to be used for cloning. See here for further details on doing […]

    Reply
  3. duplicated datastore UUIDs when cloning nested esxi on the same local datastore | log here before I forget says:
    07/03/2014 at 3:47 am

    […] those points in wlam's blog, to add those cloned vESXi into the same vCenter, you will likely get errors complaining the […]

    Reply
  4. HowTo: Install Nested ESXi 5.5 | Cloud Solutions Architect says:
    07/26/2014 at 10:01 am

    […] http://www.virtuallyghetto.com/2013/12/how-to-properly-clone-nested-esxi-vm.html […]

    Reply
  5. Cloning Nested ESXi in Workstation - theITHollow says:
    10/06/2014 at 2:41 pm

    […] NOTE:  Steps one and two are completely ripped off from William's post.  If you want more information please see it directly here:  http://www.virtuallyghetto.com/2013/12/how-to-properly-clone-nested-esxi-vm.html […]

    Reply
  6. VMware Workstaion Home Lab Setup Part 5 – ESXi Template - Virtxpert says:
    11/06/2014 at 1:19 pm

    […] following steps are courtesy of William Lam and virutallyghetto.com, check out his site and the blog post for full details on resetting the virtual machine.  Since we do not have any VMFS datastores, there are only two steps we need to do before cloning, […]

    Reply
  7. Building a Nested ESXi Home Lab | SIPALOOZA says:
    01/21/2015 at 4:35 pm

    […] properly prepare the nested host before you clone it! It's William Lam again who wrote an excellent article about how to do this.  Here is a short […]

    Reply
  8. vShiza says:
    02/01/2015 at 11:35 pm

    […] properly prepare the nested host before you clone it! It's William Lam again who wrote an excellent article about how to do this.  Here is a short […]

    Reply
  9. Running Nested ESXi / VSAN Home Lab on Ravello | virtuallyGhetto says:
    03/06/2016 at 8:50 am

    […] not want to do. The steps outlined here will be following the process which I have documented in my How to properly clone a Nested ESXi VM? […]

    Reply
  10. How To Install And Clone Nested ESXi VM? | Settlersoman - A Settler In The SDDC World. says:
    03/13/2016 at 4:59 pm

    […] Before cloning our ESXi VM, we need to do two steps: […]

    Reply
  11. Creación de una imagen de VMware ESXi para laboratorio - VMware Blog says:
    06/20/2016 at 4:33 pm

    […] Antes de poder clonar la máquina tenemos que realizar otras configuraciones para siguiendo los pasos, una vez más, de William Lam en virtuallyGhetto […]

    Reply
  12. Nested ESXi Enhancements in vSphere 6.5 | virtuallyGhetto says:
    10/26/2016 at 5:46 am

    […] Perform two configuration changes within the Nested ESXi VM which will prepare them for cloning. You can find the configuration changes described in my blog post here […]

    Reply
  13. How to create a Nested VMware vSAN 6.6 environment. - Provirtualzone - Virtual Infraestructures says:
    06/05/2017 at 3:27 am

    […] Thanks to William Lam for is help and article HERE regarding cloning Nested […]

    Reply
  14. How to set up a nested vSphere 6 environment - Part 2 says:
    07/24/2017 at 4:32 am

    […] this part as it is beyond the scope of this post but if you wish to learn more, here's a good article on the […]

    Reply
  15. How to set up a nested vSphere 6 environment - Part 3 says:
    07/24/2017 at 5:51 am

    […] part as it is beyond the scope of this post but if you wish to learn more, here's a good article on the […]

    Reply
  16. Create an ESXi 6.7 VM Template | iThinkVirtual™ says:
    01/21/2019 at 1:21 pm

    […] How to properly clone nested ESXi VM […]

    Reply
  17. Create a VMware vSAN 6.7 U1 Nested ESXi Lab - Virtualization Howto says:
    04/27/2019 at 7:24 pm

    […] Lam has a great write up here on how to properly clone an ESXi host for nested lab purposes. Quoting from William's blog […]

    Reply
  18. Create a nested ESXi template in vSphere – thecjtlab says:
    11/17/2020 at 5:22 am

    […] How to properly clone a Nested ESXi VM? Enable SSH […]

    Reply
  19. Setup Nested ESXi Hosts – Virtualization and Cloud Computing says:
    12/23/2022 at 9:44 am

    […] Check blog from William Lam […]

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Changing the default HTTP(s) Reverse Proxy Ports on ESXi 8.0 03/22/2023
  • Quick Tip - How to download ESXi ISO image for all releases including patch updates? 03/15/2023
  • SSD with multiple NVMe namespaces for VMware Homelab 03/14/2023
  • Is my vSphere Cluster managed by vSphere Lifecycle Manager (vLCM) as a Desired Image or Baseline? 03/10/2023
  • Interesting VMware Homelab Kits for 2023 03/08/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023