The VMware Fling's team recently released an update to the existing PowerCLI Extensions which now exposes the new VMFork aka Instant Clone capability that was introduced in vSphere 6.0. The Fling contains a set of PowerCLI Extension Modules which in turn provides new PowerCLI cmdlets for accessing the Instant Clone feature. The idea behind the Fling is to help VMware understand how customers would like to consume the Instant Clone feature not only from a CLI point of view but also from an API and UI standpoint. Prior to this, Instant Clone was only available through the use of either Horizon View or the Big Data Extensions product. I think this is a great opportunity for customers and partners to help shape how Instant Clone should be consumed more generally.
One of the use cases I had in my mind when I had first heard about the Instant Clone feature was to be able to quickly instantiate new Nested ESXi VMs. When I got the opportunity to help test out early prototypes of the Instant Clone cmdlet to help provide feedback and usability improvements, I knew I had to give Nested ESXi a try!
Requirements:
- Fresh installation of Nested ESXi 6.0 in VM (unconfigured)
- PowerCLI 6.0 Release 1
- Instant Clone PowerCLI Extensions Fling
- Nested ESXi 6.0 Instant Clone Scripts
High level process:
- A "preparation" script will be manually uploaded & executed within the Nested ESXi VM (Parent VM) to prep the system for Instant Cloning
- As the Parent VM is quiesce, both the pre/post customization script will be uploaded to the Parent VM automatically. The "pre-customization" is also then executed within the Parent VM which properly setups the library path to the VMware Tools binary (applicable to ESXi 6.0 only) and is then placed in a ready state for creating Instant Clones
- As new Instant Clone (Child VMs) are spun up, the "post-customization" script is automatically executed to add additional configurations and most importantly ensure newly created Instant Cloned Nested ESXi VMs have unique network identities
Note: For Instant Cloning regular OSes, only step 2 and 3 are really needed. Due to a known issue with VMware Tools for Nested ESXi, I have found that it is easier to prepare the Nested ESXi VM prior to quiescing and creating Instant Clones from the Parent VM.
Instructions:
Step 1 - Download and install both PowerCLI 6.0 Release 1 & Instant Clone PowerCLI Extensions Fling.
Step 2 - Perform a fresh Nested ESXi 6.0 installation in a VM, do not configure additional settings outside of enabling ESXi Shell and SSH.
Step 3 - Download the Nested ESXi 6.0 Instant Clone Scripts which contains the following four files:
- prep-esxi60.sh - Prepares the Nested ESXi VM and ensures that new Child VMs will not retain the Parent VM's MAC Address which is baked in several places
- pre-esxi60.sh - Pre-customization script which is used to properly setup the library paths to use the VMware Tools daemon to retrieve guest properties from PowerCLI script
- post-esxi60.sh - Post-customization script which is used to apply networking configuration and hostnames for example
- vmfork-esxi60.ps1 - An example PowerCLI script which issues the Instant Clone cmdlets
Note: For out of the box use, the only script that needs to be modified is the PowerCLI "vmfork-esxi60.ps1" script, the rest of the scripts should work or require very little to no modifications assuming you have followed the instruction thus far.
Step 4 - Upload the prep-esxi60.sh to Nested ESXi 6.0 VM (Parent VM) and then execute it using either the ESXi Shell over SSH or through a VMRC session. If you use SSH, you will notice that the script hangs, that is because the VMkernel interface is deleted as part of the script.
Step 5 - Next, we need to make a few edits to the vmfork-esxi60.ps1 script to update the name of your ESXi VM, along with its credentials and the full path to both the pre and post customization scripts. Below is an example of the variables that you will need to edit:
$parentvm = 'vESXi6' $parentvm_username = 'root' $parentvm_password = 'vmware123' $precust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\pre-esxi60.sh' $postcust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\post-esxi60.sh'
The section shown below will also need to be edited which contains the customization properties which are then passed down to the guestOS for configuration as part of the Instant Clone process.
$configSettings = @{ 'hostname' = "$vmname.primp-industries"; 'ipaddress' = "192.168.1.$_"; 'netmask' = '255.255.255.0'; 'gateway' = '192.168.1.1'; }
Step 6 - Lastly, it is time to run the script by issuing the following command:
.\vmfork-esxi60.ps1
If everything was successful, you should see a couple of new powered on Instant Cloned Nested ESXi VMs that have been fully customized and ready for use!
Note: There have been a couple of times where newly Instant Clone VMs have not been properly customized and when looking in the Instant Clone logs under /var/tmp/quiesce.log you may find "Unable to fork" error message. I usually have to re-quiesce the Parent VM which I do so by reverting back to a snapshot that captures the state after Step 4. Once I re-run the PowerCLI script, I am able to successfully deploy N-Number of Instant Clone Nested ESXi VMs. For additional best practices and tips/tricks, be sure to check out this blog post here.
Big thanks to Jim Mattson for some of his earlier research and work on this topic which made implementing these scripts much easier.
Thanh says
Hi William,
Thanks you for sharing this awesome news. I was wondering if this method can be applied to instantly clone a vCenter, NSX manager or vShield manager with several esxi hosts (essentially I want to be able to clone the whole setup to multiple dev and qa group so that they don't have to spent time setup the environment).
Example: on vlan 100 I have a vCenter, vShield manager register with vCenter, and several esxi hosts added to vCenter, and I want to clones all these vm to vlan 200, 300, etc. (in the child clone setup I don't need to modify the IP address of any vm because all the vm will be on a separate vlan 200, 300, etc.).
Thanh
William Lam says
This should work as long as they're isolated on the network since you'll have duplicate MAC Addresses, etc. In fact, the PowerCLI Development team is doing exactly this in their environment for QE testing and using the Instant Clone feature to quickly spin up test environments
Thanh says
Hi William,
Do you know if there any limitation in what guest vm can be enable as instant clone vm, because I have no problem setup an instant clone of a vCenter server and virtual esxi host, but I am having trouble enable instant clone on the nsx manager guest vm.
PowerCLI C:\> $parentVM_nsx = Get-VM lab-nsx7
PowerCLI C:\> $parentVM_nsx
Name PowerState Num CPUs MemoryGB
---- ---------- -------- --------
lab-nsx7 PoweredOn 4 12.000
PowerCLI C:\> $parentForkVM_nsx = Enable-InstantCloneVM -VM $parentVM_nsx -guestUser "admin" -GuestPassword "Passw0rd123!" -Confirm:$false
Enable-InstantCloneVM : 8/18/2015 1:43:12 AM Enable-InstantCloneVM Guest Type and Guest OS Family
have no values for VM 'lab-nsx7'
At line:1 char:21
+ $parentForkVM_nsx = Enable-InstantCloneVM -VM $parentVM_nsx -guestUser "admin" - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidResult: (:) [Enable-InstantCloneVM], VimException
+ FullyQualifiedErrorId : Fling_InstantClone_GuestOperationsHelper_GetGuestType_UnknwnGuest,VMware.VimAu
tomation.Extensions.InstantClone.Commands.EnableInstantCloneVM
William Lam says
The issue that I suspect with the NSX Manager is that when you login, you're not placed in a regular shell which is required as vmtoolsd is called to actually enable the "Quiesce" operation. As you probably know when you login to NSX Manager, you're dropped into a limited shell which doesn't have access and I suspect this is the issue.
Thanh says
Hi Willian,
I have manage to gain root access to NSX manager by following a similar guide
http://blog.romant.net/technology/by-example-enabling-ssh-in-vshield-edge/
However, I still not able to enable instant clone on the nsx vm.
William Lam says
Do you get anything when you run "which vmtoolsd"? It might also help to investigate the Instant Clone logs located under /var/tmp/quiesce.log Recommend checking out this blog post http://www.virtuallyghetto.com/2015/08/instant-clone-powercli-cmdlets-best-practices-troubleshooting.html which gives you some tips&tricks / best practices
Thanh says
Here is the output on NSX
[root@lab-nsx7 /var/tmp]# which vmtoolsd
/usr/local/bin/vmtoolsd
[root@lab-nsx7 /var/tmp]# /usr/local/bin/vmtoolsd -v
VMware Tools daemon, version 9.4.6.33107 (build-1770165)
[root@lab-nsx7 /var/tmp]#
When I try to enable the instant clone on the NSX manager vm, I didn't see any log generate under /var/tmp/quiesce.log
Here is the output on VCSA 6 (build version is much more recent and path is also different)
lab-vc7:~ # which vmtoolsd
/usr/sbin/vmtoolsd
lab-vc7:~ #
lab-vc7:~ # /usr/sbin/vmtoolsd -v
VMware Tools daemon, version 9.10.0.43888 (build-2476743)
lab-vc7:~ #
William Lam says
Yea, I'm not sure then. It's possible that the version of VMware Tools may not work with Instant Clone.
alanrenouf says
Maybe I should get the team to blog what they are doing.
William Lam says
I think that would be an awesome idea to hear from the PowerCLI team on how they leveraged the Instant Clone feature to help with development/testing
Andrea says
Hi William,
Just out of curiosity, how does any Horizon product leverage the instant clone feature?
I didn't see anything new that would indicate new instant cloning capabilities in Horizon suite of products.
Dave OSullivan says
any ideas William?
https://communities.vmware.com/thread/539975
Rob Babb (@atlantauser) says
Always great information William!
I ran into an issue while trying to go crazy with vmfork. I'm getting errors about maximum vcpu limits and not sure how to bypass.
I'm running an Ubuntu Parent and vmfork children on top of nested ESXI 6.0 on top of ESXi 5.5 physical. When I was powering on VM's I was trying for 20 children and got 18 to work. 19 wouldn't power on and gives this message:
Maximum virtual machines limit reached: 19 (32768 worlds).
An error was received from the ESX host while powering on VM UbuntuClone190.
Failed to start the virtual machine.
Failed to power on VM.
Could not power on virtual machine: Number of running VCPUs limit exceeded.
So I did some more reading and found another blog that mentioned changing the maxvCPUsPerCore & maxvms values (https://deangrant.wordpress.com/2015/01/11/configuring-esxi-host-maximum-number-of-vcpus-on-a-single-core-and-virtual-machines-per-host/) After changing those and rebooting the nested ESXi 6.0 host it's stuck on booting at the sfcbd-watchdog service.
So I might have to rebuild my nested ESXi VM's, but aside from that though. Do you have any idea how to get more VM's per host and bypass any of the safeguards that may be limiting it?
Matt V says
Hi William. I'm using your 6.5u1 nested ESXi image to attempt to get this to work. Every time I start the "enable-instantVM" process, it goes for a little bit them comes back with the error (I did you your prep script before running this):
Enable-InstantCloneVM : 9/21/2017 5:19:20 PM Enable-InstantCloneVM Enable instant clone failed for vm xxxxxx. The in-guest quisce operation '/bin/ash -c ". /var/tmp/pre-esxi60.sh; export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/vmware/vmtools/lib; export PATH=$PATH:/usr/lib/vmware/vmtools/bin; vmtoolsd --cmd \"vmfork-begin -1 -1\"; . /var/tmp/post-esxi60.sh;" > /var/tmp/quiesce.log 2>&1' didn'tquiesce the VM.
For details see the created in-guest log file '/var/tmp/quiesce.log'
So then when I go to check the logs I get this error in the web client:
A server error occurred.
You have reached the maximum number of connected consoles: 40. Please contact your administrator.
and I can't get into the console at all. Any ideas? Thanks ahead of time.
Matt V says
I figured it out. Both the pre and post scripts had hidden windows carriage returns when I initially copy/pasted the text. I had to convert the files to unix based end of lines before running the powershell script. I'm honestly surprised no one else has had that issue.
Also, this works with ESXi 6.5u1 and vCenter 6.5 if anyone is wondering