I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.
UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349
First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.
Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.
- The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
- The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.
To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.
First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:
esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:
sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf
To ensure the file is persisted, run /sbin/auto-backup.sh
Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.
If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:
esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]
These tips can also apply to cloning boot LUNs/volumes in physical hosts that use SAN instead of local disks for boot.
Good stuff, playing around with this and dynamic disks to create a golden OVF for nested ESXi in my home lab. Just in time for Christmas!
Since it's really tiny and can't C&P from an image, here's the command to check the FollowHardwareMac setting:
esxcli system settings advanced list -o /Net/FollowHardwareMac
Hello William,
great Post.
I tried the "esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]" command but do get the message "No unresolved VMFS snapshots with volume label 'esxi2-local' found".
Did I miss something?
Thanks
Manfred
The part of resignaturing the vmfs volume is apparently incorrect since the esxi does not see the vmfs as copied or snapshoted thus resignaturing cannot be applied. The vmfs is indeed snapshoted at vmware workstation level instead of the esxi level.
The only way I know that is working is as below, this method requires recreating the local vmfs datastore, hence all data on it WILL LOST.
1. # vmkfstools --queryfs -h /vmfs/volumes/
take note of the "Partitions spanned", it's kind like mpx.vmhbal:C0:T0:L0:3 (refered to as hereafter).
2. # esxcli storage filesystem unmount -l
make sure the local datastore is unmounted.
3. # vmkfstools --createfs vmfs5 --blocksize 1m --setfsname /dev/disks/
this will recreate the vmfs on the partition and give it a new UUID.
is the local datastore label you want to have the UUID changed, mine was "datastore1". is the new datastore label, you can specify the same as .
Did you try these commands you're referring to? They don't seem so right...
This should work:
# enter maintenanceMode mode
esxcli system maintenanceMode set --enable true --timeout=1
# determine partition device (look under "Partitions spanned (on "lvm")" for mpx.)
vmkfstools -P /vmfs/volumes/
# dismount datastore by label
esxcli storage filesystem unmount -l
# recreate FS
/usr/sbin/vmkfstools -C vmfs5 -b 1m -S /vmfs/devices/disks/mpx.
# reboot system
esxcli system shutdown reboot --reason="Updated System UUID"
# exit maintenance mode
esxcli system maintenanceMode set --enable false --timeout=1
Now I see the problem.. this brain-damaged commenting system is purging everything in chevrons.. let's try one last time with parenthesis..
# enter maintenanceMode mode
esxcli system maintenanceMode set --enable true --timeout=1
# determine partition device (look under "Partitions spanned (on "lvm")" for mpx.(something))
vmkfstools -P /vmfs/volumes/(datastore-label)
# dismount datastore by label
esxcli storage filesystem unmount -l (datastore-label)
# recreate FS
/usr/sbin/vmkfstools -C vmfs5 -b 1m -S (newlabel) /vmfs/devices/disks/mpx.(something)
# reboot system
esxcli system shutdown reboot --reason="Updated System UUID"
# exit maintenance mode
esxcli system maintenanceMode set --enable false --timeout=1
Above steps worked. Thanks for the solution
Dos this method erase data from datastore1. I cloned ESXi server2 from ESXi server1 by taking one boot disk out from server1 and then inserted on server2 and booted and works fine. But when I added ESX server1 and server2 in vcenter it shows as datastore1(22) mounted for both server1 & server2. I do not want to loose the data. Does this method erase data if I follow this procedure on ESXi server2?
I created a virtual ESXi template using the listed FollowHardwareMac setting and deleted the UUID as described. This works great for vmk0. It seems to regenerate properly every time I deploy a new copy of the template. My template also has several other vmk interfaces as well. What I am finding though is that other any other vmks (except for vmk0) that are part of the cloned VM still retain the same MAC as the original VM after being cloned from the template. Do you have any ideas on this?
Worked for me even after cloning. Thanks a ton !!
I was able to successfully clone nested ESX by doing a standard clone then performing a factory reset inside ESXi.
The factory reset worked for me too, thanks!
I created a cluster with VCSA 6.0 and ESXi 6.0, and could not get vSAN to run. After the factory reset, it worked.
Hi,
Your article is very useful to me and solved 80% of my problem, But I am not able to add clone vESXi to same VC.
I ran your command "esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]". I am getting error. Can you please help me to fix this issue. How to change UUID of datastore for base image. So that Cloned vESXi will not get same UUID for datastore
1. Entered follow hardware command via SSH to ESXi nested host.
2. Used vi to remove UUID.
3. Shutdown host.
4. Using VC - get VMDK error.
I am either missing a step or the solution does not work for 5.5. Can someone help?
This works on both 5.x & 6.x. What exact error message are you seeing, as VMDK error is not specific enough
Hello William. Thank you for responding. I've blanked the config but it appeared to be an issue related to copying the VMDK. What I tried was manually copying the files to another folder, booted the ESXi 5.5 nested. ticked the option I copied it when it showed up. From the other posts under your main post, the it seems the local datastore (1) is linked via UUID or something. I tried doing the resig but SSH CLI was telling me nothing needed to be resig'ed. It would be really nice if you literally show us mere mortals a spoon-fed step by step of everything you did. Being able to clone an ESXi nested image would make my life much easier with VC.
I'm not sure why you're manually copying the VMDK? If you have vCenter Server, you just need to run through the preparation as shown in the blog post and then right click and select "Clone"
Hi William,
Can you help in in the below situation please.
When i follow the above steps am getting the below error while trying to generate new SSID.
I am running this command: /usr/sbin/vmkfstools -C vmfs5 -b 1m -S name1 /vmfs/devices/disks/mpx.vmhba1:
C0:T0:L0
ends up saying
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs5 file system on "mpx.vmhba1:C0:T0:L0" with blockSize 1048576 and volume label "name1".
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0: Permission denied. (Have you set the partition type to 0xfb?)
Error: Permission denied
I have logged in as root.. but it syas permission denied.
Regards,
Madhu
We are cloning nested ESXi in vCloud Director. What doesn't seem to change on each linked clone is the bios.uuid which is causing is issues. We've done all the above and while system uuid is changed, the bios.uuid is not changing. Any suggestions? thanks!
Niamh,
Have a look at https://kb.vmware.com/kb/2002506 for your answer
thanks William, yep working on it now !!
Hi will, Is it possible to strip the VMFS UUID before cloning?
Very Informative article. Thanks alot. It worked well. Other than following command (entering mantenance mode in CLI dint worked...) But that i easily managed by connecting directly and doing in GUI.
# enter maintenanceMode mode
esxcli system maintenanceMode set –enable true –timeout=1
I'm following your excellant article, and cloning an ESX 6.5 host.
I've run "esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1" in the template, then cloned with customization wizard. The Management vmk0 kernel got a new Mac and assigned IP, however the vMotion vmk1, and NFS vmk2 did not get a new Mac, or the new assigned IP from the Wizard so I set them manually.
I've confirmed in the vCenter dvs Port listing that my three cloned hosts have different Mgmt Mac's but duplicate Mac's on the vMotion and NFS vmkernel ports. This explains why I'm having so many intermittent network issues.
Is the Customization wizard and FollowHardwareMac known to work for non Mgmt kernel ports on ESX 6.5?
Lee
Hi,
I have used clone method when I initially built ESXi 5.1. Then I was able to upgrade it to 5.5 with no issue. Now that I am upgrading to ESXi 6.5, its failing adn I am getting a purple screen and I see following error when I look at the debug output. "The System has found a problem on your machine and cannot continue. System with same UUID have been detected. Make sure you do not have two ESXi installation". Also I see message "cpu35:65895)FSS: 5817: Detected Duplicate image on device 'naa.600605b00aa3a4901f08161c4c258e19:5". I have two data stores. One for OS and one for APPL. this 19:5 is for APPL DS but I could not find such device 19:5 on my server.
Any idea? I came across many blogs but so far none has helped including the one from vmware to identify why its finding this as duplicate image.
Thanks for details. I have created a nested esxi for the DHCP enabled VLAN,but for the new cloned nexted esxi I am not getting proper IP and getting some private IP. any idea?
Hello there!
Great text!
I wonder, is it necessary to install vmware tools to nested ESXi 7.0 and later?
I ask because you mention it at the end of the artile and also because there is a light verson of tools already installed from scratch (tools-light - 11.1.1.16303738-16850804).
Thanks for the article and reply!
Cheers!
AS
Hi William,
I create two esxi 7.0u2 nested in ESXi 6.7 U3.
I try to run ping between the mgmt network (vmk0) of the nested esxi...
If I try to add mgmt network redundancy, I'm unable to communicate with vmk0 until I remove one nic in teaming.
I run this command:
esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
In my nested esxi, the vmk0 mac address is the same of "physical" vmnic.
(could this be the problem?)...I did not find the mac address of vmk0 in the esx.conf file
My nested esxi has 6 vmnic vmxnet3.
Can you tell me what can i do ?
Thanks a lot..
William, with the UUID related changes in 7.0 U2 is it safe to deploy multiple nested ESXi hosts from your OVA template into the same environment or will each of them have the same UUID and cause corruption issues?
I recently had several iSCSI VMFS datastores shared by physical and nested hosts seemingly get corrupted. They were reporting in the vmkernel log that they were snapshots and wouldn't auto-mount during boot for both physical and nested ESXi hosts. Only way to mount them was via CLI and they would disappear after every reboot. I had to delete them and recreate them to resolve the problem but I never really figured out the root cause. This sounds like it could have been the root cause since I had 3 nested 7.0U2 VMs all deployed from your template that were mounting those datastores.
Mike,
I'm not aware of any issues with the latest 7.0u2a OVA, I know quite a few folks have it deployed including myself and haven't ran into any problems. If the ESXi host has duplicated UUID and has local datastore (which latest 70u2a OVA defaults to) and you attempt to add both hosts to vCenter Server, you'll normally see it complain and fail to add. You may want to double check that your physical host boot LUN wasn't somehow cloned. In theory, you should see the problem with just two Nested ESXi VMs, so just make sure these were all deployed fresh from OVA. If you're still having issues, just let me know
I am using ESXi 7.0.3 now.
after clone nested vm without snapshot.
Though I try to change datastore UUID, I can't chang local datastore.
What should I do ?
[root@nsxesx01:~] vmkfstools -P /vmfs/volumes/5fa8a747-f9af54c7-a815-005056b1e7c8
VMFS-L-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): OSDATA-5fa8a747-f9af54c7-a815-005056b1e7c8
Mode: public
Capacity 34091302912 (32512 file blocks * 1048576), 30833377280 (29405 blocks) avail, max supported file size 70368744177664
Disk Block Size: 512/512/0
UUID: 5fa8a747-f9af54c7-a815-005056b1e7c8
Partitions spanned (on "lvm"):
mpx.vmhba0:C0:T0:L0:7
Is Native Snapshot Capable: NO
[root@nsxesx01:~] vmkfstools -P /vmfs/volumes/5fa8a747-f9af54c7-a815-005056b1e7c8
VMFS-L-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): OSDATA-5fa8a747-f9af54c7-a815-005056b1e7c8
Mode: public
Capacity 34091302912 (32512 file blocks * 1048576), 30833377280 (29405 blocks) avail, max supported file size 70368744177664
Disk Block Size: 512/512/0
UUID: 5fa8a747-f9af54c7-a815-005056b1e7c8
Partitions spanned (on "lvm"):
mpx.vmhba0:C0:T0:L0:7
Is Native Snapshot Capable: NO
All you have ever had to do to clone an ESXi VM or a Boot from SAN LUN is this:
Login to the DCUI
Select "Reset System Configuration"
Power down instead of reboot
If it's a boot from SAN LUN, unmap it.
Clones will not have duplicate MACs or UUIDs.