WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How Wrecking Crew Inc. leveraged vSphere's Instant Clone to instantly provision hundreds of VMs

09.18.2015 by William Lam // 9 Comments

Okay, Wrecking Crew Inc. is not a real company but a fictitious one that was used during Alan Renouf and Li Zheng's VMworld session, #INF5803 - Deploy Hundreds of VMs Instantly via Forking (aka vSphere Instant Clone). During the session, Wrecking Crew Inc. was described as a modern online gaming company who was about to launch a new product. They found out that their game was going to be more popular than they had originally anticipated and needed to be able to quickly spin up a large number of instances of their game to handle the load.

Though this was a fictional company to help set the stage for the demo, the underlying use case is an actual challenge for many of our customers. One of the examples that Alan used was the traditional Black Friday sale which occurs on the Friday after Thanksgiving in US. As you can imagine, being able to quickly scale up your online application based on customer demand can really give retailers an real edge over their competition. In todays world, waiting a few additional seconds for a site to load can mean the difference between a sale and customer going else where for their business. Online retail is just one of the many industries and verticals that can easily benefit from the Instant Clone capability.

To help further demonstrate the use case, a recorded demo was shown utilizing the new PowerCLI Extensions which provides access to the Instant Clone feature from an Automation standpoint. I actually built the demo for Alan and Li's session and I thought I would share some more details along with the video in case you were not able to attend the session in person, which I was not able to do.

Below are the set of technologies that were used in the demo:

  • VMware vSphere + Virtual SAN
  • Intel Hardware
  • VMware Photon
  • HashiCorp Consul
  • Nginx
  • Registrator
  • Docker
  • Node.js

wrecking-crew-inc-using-instant-clone-to-provisioning-hundreds-to-thousands-of-vms-0
I wanted to also give a quick shoutout to my buddy Rawlinson Rivera for his help with Intel and getting us access to their 64 Node NVMe VSAN Cluster. Our friends over at Intel were kind enough to allow us to quickly record a demo before they had to pack and ship the hardware to San Francisco for VMworld.

To provide some additional context on what you will see in the video, a diagram is provided below. The environment initially consists of two Photon VMs which runs the Consul Cluster which provides service discovery and configuration capabilities and the Nginx Load Balancer which is highlighted in the purple and blue box. Each of those VMs run their respective application which runs inside of a Docker Container and is highlighted in the smaller boxes. On each ESXi host, we have Photon VM which contains our Node.js application that we will be Instant Cloning from. This Photon VM is also referred to as the Parent VM and where all the new forked children VMs will be spawn off of.

This Photon VM also runs an instance of Consul, Registrator and the Node.js application inside of a Docker Container. As new instances are brought online which is highlighted in the black box, they will automatically register themselves with the Consul Cluster. As the Docker Containers are started up, Registrator will be notified to automatically add the new instances to the Nginx Load Balancer, making our application servers available for use immediately. When our application server powers down, Registrator will automatically detect that the Docker Container for our Node.js application is no longer running and automatically unregister it from the Nginx Load Balancer. To simply scale up or down the application, it is simply Instant Cloning our Photon (Parent) VM and powering it on or off, there are no additional steps required.

wrecking-crew-inc-using-instant-clone-to-provisioning-hundreds-to-thousands-of-vms-1

Without further ado, here is the video of the demo. If you want additional commentary and you attended VMworld US but could not make it to Alan and Li's session, you can now watch it online here. This session will also be repeated in VMworld EMEA for those attending in a couple of weeks. I would highly recommend you check out the session as there is a lot of awesomeness in the session along with technical deep dive of the Instant Clone technology.

VMworld 2015 Demo of how Wrecking Crew Inc. leverages vSphere's Instant Clone feature from lamw on Vimeo.

For more details about Instant Clone, be sure to check out these resources below which includes a Instant Clone script repository for several GuestOSes, including Photon which was used in the demo:

  • Project Fargo aka VMFork – What is it?
  • Project Fargo aka VMFork and TPS?
  • Instant Clone PowerCLI cmdlets Best Practices & Troubleshooting
  • Instant Clone community customization script repository
  • How to VMFork aka Instant Clone Nested ESXi?
  • VMware Instant Clone is now at your fingertips with the updated PowerCLI Extensions fling!
  • Using VMware Instant Clone via PowerCLI Extensions Fling

Categories // Automation, Cloud Native, Docker, VSAN, vSphere 6.0 Tags // consul, Docker, instant clone, nginx, node.js, Photon, PowerCLI, registrator, Virtual SAN, vmfork, VSAN, vSphere 6.0

Instant Clone PowerCLI cmdlets Best Practices & Troubleshooting

08.06.2015 by William Lam // Leave a Comment

I was fortunate to have been given early access to the VMFork (Instant Clone) PowerCLI cmdlets to help provide early feedback and usability improvements before it was released to customers. Having spent some time with the Fling, I have learned a thing or two about how Instant Cloning works and some of the caveats or gotchas while creating the customization scripts that are used as part of the Instant Clone workflows. I wanted to put together a quick reference on some of my findings as well as well as other recommendations from Engineering who have worked closely with the Instant Clone feature.

The idea is to have this as a living document which I will update as new tips and tricks are identified.

Best Practices

  • Ensure VMware Tools is installed inside the guestOS and also good time to ensure you are running the latest
  • Both Pre/Post Customization scripts are uploaded to /var/tmp by the Enable-InstantCloneVM cmdlet
  • Do not delete Child VMs directly on ESXi, manage it through vCenter Server. There is currently a known issue in which deleting Child VMs will also delete the Parent VM's disk
  • Additional custom variables can be passed to the post-customization script by adding to the -ConfigParams array of variables.
    • An example could be passing in two custom properties called "foo" and "bar" which would look like:

    @{foo = "val1";bar ="val2"}

    • To retrieve the variable "foo" and "bar" from within the post-customization script, you would issue the following commands:

    vmtoolsd --cmd "info-get guestinfo.fork.foo"
    vmtoolsd --cmd "info-get guestinfo.fork.bar"

  • A Forked Child VM will also have a duplicate MAC Address which needs to be updated as it is not automatically picked up.
      • You can either manually set it by retrieving the guestinfo.fork.ethernet0.address with the post-customization script.
      • An easier way would be to reload it based on the guestOS type. On a Linux system, you can use the modprobe command like the following (Submited by George Hicken):

    modprobe -r vmxnet3;modprobe vmxnet3

  • A Forked Child VM may also have identical kernel entropy pools which means semi-predictable RNG, possibly including TCP sequence numbers (Submited by George Hicken)
  • A Forked Child VM's system clock may also be out of date (until you call hwclock --hctosys or similar) which can cause problems with ordering of file timestamps (Submited by George Hicken)
  • Shared host keys if you are using a PKI system or identical asset identifiers in the case of Windows and any sort of AD infrastructure would also need to be either removed prior or updated after a Child VM is created (Submited by George Hicken)
  • Instant Cloning Nested ESXi has been a bit tricky due to a known issue with the VMware Tools for Nested ESXi. I have found that manually preparing the guest prior to Instant Cloning has yield better results. For more information on how to Instant Clone Nested ESXi, check out the blog post here
  • Powering off the Parent VM means that the VM is no longer quiesced and this also means that new Child VMs can not be instantiated until all existing Forked Child VMs have been powered off and the Parent VM has been re-quiesced
  • If you plan on downloading or installing additional software packages on the Parent VM, it is recommended that you perform that operation directly in the VM and not within the pre-customization script. I have noticed that if pre-customization takes too long, the quiesce operation eventually fails even though the operations within the pre-customization script executed successfully.
  • To ensure Forked Child VMs do not contain duplicate disk ID's from Parent VM such as setting up a VSAN environment using Instant Clone Nested ESXi, add the disks after Forked Child VMs have been created.
  • For additional OS Customization Scripts, be sure to check out the Instant Clone community customization script repository and consider contributing back scripts that you have developed.
  • When you hard reset or power off on a child VM it will respawn from the parent, soft resets will not respawn (Submitted by Alan Renouf)

Troubleshooting

  • Instant Clone guestOS logs are stored in /var/tmp/quiesce.logvmfork-logs
  • Consider enabling tracing within your customization scripts. An example of this for a shell script is using

    set -x

  • Add additional echo or print statements like Start/Stop of certain sections like Pre/Post which can aide in reviewing the Instant Clone logs as seen in the screenshot above
  • For Instant Cloning Nested ESXi guestOSes, I recommend taking a snapshot after you have prepared the guest and removed any system specific information. This allows you to quickly revert back to a known state for ease of debugging. I found this to be very useful to be able to start back a known clean state while developing the customization scripts for Instant Cloning Nested ESXi
  • A known issue that is mentioned in the documentation of the Instant Clone cmdlets is after enabling a ParentVM for Instant Cloning, is that it is no longer available for migration to another ESXi host. The reason for this is that after powering off the VM, the "parentEnabled" boolean flag is still set to "true" which prevents the migration. Currently, there is not a work around but hopefully this will be resolved in a future update of the cmdlets. You can see this by running the following PowerCLI snippet:

    (Get-VM "MyParentVM").ExtensionData.Config.ForkConfigInfo

 

Categories // Automation, PowerCLI, vSphere 6.0 Tags // fling, instant clone, vmfork, vmtoolsd, vSphere 6.0

Instant Clone community customization script repository

08.04.2015 by William Lam // Leave a Comment

I am sure many of you are probably anxious to get your hands on the new VMFork aka Instant Clone PowerCLI Extensions Fling that was recently released! While using the Instant Clone cmdlets to help provide feedback and improvements, I have found that I spent the majority of my time on developing and fine-tuning the pre and post-customization scripts. Instead of having others hit similar issues that I ran into, I wanted to see how I could easily share some of the leanings but also incorporate collaboration?

After thinking about this for a bit, I realized this was a great opportunity to create a community Github repository of Instant Clone customization scripts that anyone can either use and/or contribute back to. I have already added a few OS customization scripts to the repo to start with like ESXi 6.0, Ubuntu 14.x and VMware Photon. To access the repository, simply visit https://github.com/lamw/vmfork-community-customization-scripts

The Instant Clone community customization script repository is broken up by series of OS directories which contain the relevent set of pre/post-customization scripts for that OS and any additional scripts that might be required. It may also contain further instructions on how to use the script as well as an example "driver" script which calls into the Instant Clone cmdlets demonstrating how to use the scripts.

Here is an example for ESXi 6.0 OS:

esxi60
├── post-esxi60.sh
├── pre-esxi60.sh
├── prep-esxi60.sh
└── vmfork-esxi60.ps1

For those that wish to contribute back, just fork the repository and send me a pull request. I am really looking forward to seeing what the community comes up with!

Categories // Automation, vSphere 6.0 Tags // fling, instant clone, Photon, PowerCLI, ubuntu, vmfork, vSphere 6.0

  • 1
  • 2
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...