WilliamLam.com

How to VMFork aka Instant Clone Nested ESXi?

by // 15 Comments

vmfork-aka-instant-clone
The VMware Fling's team recently released an update to the existing PowerCLI Extensions which now exposes the new VMFork aka Instant Clone capability that was introduced in vSphere 6.0. The Fling contains a set of PowerCLI Extension Modules which in turn provides new PowerCLI cmdlets for accessing the Instant Clone feature. The idea behind the Fling is to help VMware understand how customers would like to consume the Instant Clone feature not only from a CLI point of view but also from an API and UI standpoint. Prior to this, Instant Clone was only available through the use of either Horizon View or the Big Data Extensions product. I think this is a great opportunity for customers and partners to help shape how Instant Clone should be consumed more generally.

One of the use cases I had in my mind when I had first heard about the Instant Clone feature was to be able to quickly instantiate new Nested ESXi VMs. When I got the opportunity to help test out early prototypes of the Instant Clone cmdlet to help provide feedback and usability improvements, I knew I had to give Nested ESXi a try!

Requirements:

High level process:

  1. A "preparation" script will be manually uploaded & executed within the Nested ESXi VM (Parent VM) to prep the system for Instant Cloning
  2. As the Parent VM is quiesce, both the pre/post customization script will be uploaded to the Parent VM automatically. The "pre-customization" is also then executed within the Parent VM which properly setups the library path to the VMware Tools binary (applicable to ESXi 6.0 only) and is then placed in a ready state for creating Instant Clones
  3. As new Instant Clone (Child VMs) are spun up, the "post-customization" script is automatically executed to add additional configurations and most importantly ensure newly created Instant Cloned Nested ESXi VMs have unique network identities

Note: For Instant Cloning regular OSes, only step 2 and 3 are really needed. Due to a known issue with VMware Tools for Nested ESXi, I have found that it is easier to prepare the Nested ESXi VM prior to quiescing and creating Instant Clones from the Parent VM.

Instructions:

Step 1 - Download and install both PowerCLI 6.0 Release 1 & Instant Clone PowerCLI Extensions Fling.

Step 2 - Perform a fresh Nested ESXi 6.0 installation in a VM, do not configure additional settings outside of enabling ESXi Shell and SSH.

Step 3 - Download the Nested ESXi 6.0 Instant Clone Scripts which contains the following four files:

  • prep-esxi60.sh - Prepares the Nested ESXi VM and ensures that new Child VMs will not retain the Parent VM's MAC Address which is baked in several places
  • pre-esxi60.sh - Pre-customization script which is used to properly setup the library paths to use the VMware Tools daemon to retrieve guest properties from PowerCLI script
  • post-esxi60.sh - Post-customization script which is used to apply networking configuration and hostnames for example
  • vmfork-esxi60.ps1 - An example PowerCLI script which issues the Instant Clone cmdlets

Note: For out of the box use, the only script that needs to be modified is the PowerCLI "vmfork-esxi60.ps1" script, the rest of the scripts should work or require very little to no modifications assuming you have followed the instruction thus far.

Step 4 - Upload the prep-esxi60.sh to Nested ESXi 6.0 VM (Parent VM) and then execute it using either the ESXi Shell over SSH or through a VMRC session. If you use SSH, you will notice that the script hangs, that is because the VMkernel interface is deleted as part of the script.

Step 5 - Next, we need to make a few edits to the vmfork-esxi60.ps1 script to update the name of your ESXi VM, along with its credentials and the full path to both the pre and post customization scripts. Below is an example of the variables that you will need to edit:

$parentvm = 'vESXi6'
$parentvm_username = 'root'
$parentvm_password = 'vmware123'
$precust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\pre-esxi60.sh'
$postcust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\post-esxi60.sh'

The section shown below will also need to be edited which contains the customization properties which are then passed down to the guestOS for configuration as part of the Instant Clone process.

 $configSettings = @{
 'hostname' = "$vmname.primp-industries";
 'ipaddress' = "192.168.1.$_"; 
 'netmask' = '255.255.255.0'; 
 'gateway' = '192.168.1.1';
 }

Step 6 - Lastly, it is time to run the script by issuing the following command:

.\vmfork-esxi60.ps1

instant-clone-nested-esxi-0
If everything was successful, you should see a couple of new powered on Instant Cloned Nested ESXi VMs that have been fully customized and ready for use!

instant-clone-nested-esxi-1
Note: There have been a couple of times where newly Instant Clone VMs have not been properly customized and when looking in the Instant Clone logs under /var/tmp/quiesce.log you may find "Unable to fork" error message. I usually have to re-quiesce the Parent VM which I do so by reverting back to a snapshot that captures the state after Step 4. Once I re-run the PowerCLI script, I am able to successfully deploy N-Number of Instant Clone Nested ESXi VMs. For additional best practices and tips/tricks, be sure to check out this blog post here.

Big thanks to Jim Mattson for some of his earlier research and work on this topic which made implementing these scripts much easier.

How to create a custom ESXi VIB to execute a script once?

by // 16 Comments

Earlier this week I received a question from a customer who was interested in creating a custom ESXi VIB that could execute a specific script within the payload as part of the installation and only running it once. This was a fairly common request that I have seen in the past and as far as I knew, this type of behavior was not possible. What was unique about this particular custom inquiry was that they mentioned they found some references on this being possible. Being the curious person I am, I decided to take another look and reach out to a few folks in Engineering.

After speaking with one of the Engineers familiar with our VIB specification, to my surprise I learned that this type of behavior was actually indeed possible but was not very well documented externally. The typical use case for this is usually to apply certain configurations which are not exposed through the traditional ESXi interfaces like the vSphere API/CLI which includes Host Profiles. One example that comes to mind is being able to deploy a VIB across several hundred ESXi hosts that could configure a specific ESXi Advanced Setting which might be marked hidden. Another example would be updating a configuration file like /etc/vmware/config or running a series of ESXi Shell commands which can not available in the vSphere API and require the use of SSH and the ESXi Shell. There are many other examples, but this should give you an idea of some of the common use cases that I have heard from customers.

Here is what is required to execute a specific script as part of the VIB installation:

  • Created an "init" script which needs to be stored under /etc/init.d as part of your VIB payload (e.g. /etc/init.d/lamw-special-vib)
  • You must ensure that the "live-install-allowed" param is set to true in your VIB's descriptor.xml file (this assumes the changes can be applied without requiring reboot to take affect)
  • The init script will need to parse special keywords passed as command-line arguments

Here is a table showing the pair of special keywords that are passed to the init script as command-line arguments during a VIB install, upgrade or remove:

Arg1 Arg2 VIB Operation
start install VIB install
start upgrade VIB upgrade
stop remove VIB removal

It is up to creator of the init script to handle the different VIB operations by parsing the command-line arguments which would then determine the operations that would get executed within the script. This not only allows you to control the commands that are executed during an installation of a VIB but it also allows you specify the commands to run upon the removal of a VIB which is quite handy for properly cleaning up an uninstall. In addition, since these special keywords are not passed as part of the ESXi boot up process, the commands within the script will not execute and ensures it only runs once during the install.

Here is very simple shell script (you could also do this in Python as well) which demonstrates how to handle the three different types of VIB operations and then uses the "logger" utility to write some output to syslog:

#!/bin/sh

ARG1=$1
ARG2=$2

if [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "install" ]]; then
 # commands to run go here #
 /bin/logger "William's custom script ran on start-install"
elif [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "upgrade" ]]; then
 /bin/logger "William's custom script ran on start-upgrade"
 # commands to run go here #
elif [[ "${ARG1}" == "stop" ]] && [[ "${ARG2}" == "remove" ]]; then
 /bin/logger "William's custom script ran on stop-remove"
 # commands to run go here #
fi

As you can see, this allows you to perform a variety of tasks through the use of a custom ESXi VIB which is a great way to be able to roll out a set of changes that may not be possible using either the vSphere API or Host Profiles for example. Best of all, this solution does not require the use of SSH which is great since many customers already disable this by default.

For more information on creating a custom VIB, be sure to check out my blog post here and you can even use this Docker image I created for building custom ESXi VIBs.

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

by // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI:

guest_operations_api_nested_esxi