Search Results for: guest operations

Instant Clone PowerCLI cmdlets Best Practices & Troubleshooting

08.06.2015 by William Lam // Leave a Comment

I was fortunate to have been given early access to the VMFork (Instant Clone) PowerCLI cmdlets to help provide early feedback and usability improvements before it was released to customers. Having spent some time with the Fling, I have learned a thing or two about how Instant Cloning works and some of the caveats or gotchas while creating the customization scripts that are used as part of the Instant Clone workflows. I wanted to put together a quick reference on some of my findings as well as well as other recommendations from Engineering who have worked closely with the Instant Clone feature.

The idea is to have this as a living document which I will update as new tips and tricks are identified.

Best Practices

Ensure VMware Tools is installed inside the guestOS and also good time to ensure you are running the latest
Both Pre/Post Customization scripts are uploaded to /var/tmp by the Enable-InstantCloneVM cmdlet
Do not delete Child VMs directly on ESXi, manage it through vCenter Server. There is currently a known issue in which deleting Child VMs will also delete the Parent VM's disk
Additional custom variables can be passed to the post-customization script by adding to the -ConfigParams array of variables.
- An example could be passing in two custom properties called "foo" and "bar" which would look like:
@{foo = "val1";bar ="val2"}
- To retrieve the variable "foo" and "bar" from within the post-customization script, you would issue the following commands:
vmtoolsd --cmd "info-get guestinfo.fork.foo"
vmtoolsd --cmd "info-get guestinfo.fork.bar"
A Forked Child VM will also have a duplicate MAC Address which needs to be updated as it is not automatically picked up.
modprobe -r vmxnet3;modprobe vmxnet3

A Forked Child VM may also have identical kernel entropy pools which means semi-predictable RNG, possibly including TCP sequence numbers (Submited by George Hicken)
A Forked Child VM's system clock may also be out of date (until you call hwclock --hctosys or similar) which can cause problems with ordering of file timestamps (Submited by George Hicken)
Shared host keys if you are using a PKI system or identical asset identifiers in the case of Windows and any sort of AD infrastructure would also need to be either removed prior or updated after a Child VM is created (Submited by George Hicken)
Instant Cloning Nested ESXi has been a bit tricky due to a known issue with the VMware Tools for Nested ESXi. I have found that manually preparing the guest prior to Instant Cloning has yield better results. For more information on how to Instant Clone Nested ESXi, check out the blog post here
Powering off the Parent VM means that the VM is no longer quiesced and this also means that new Child VMs can not be instantiated until all existing Forked Child VMs have been powered off and the Parent VM has been re-quiesced
If you plan on downloading or installing additional software packages on the Parent VM, it is recommended that you perform that operation directly in the VM and not within the pre-customization script. I have noticed that if pre-customization takes too long, the quiesce operation eventually fails even though the operations within the pre-customization script executed successfully.
To ensure Forked Child VMs do not contain duplicate disk ID's from Parent VM such as setting up a VSAN environment using Instant Clone Nested ESXi, add the disks after Forked Child VMs have been created.
For additional OS Customization Scripts, be sure to check out the Instant Clone community customization script repository and consider contributing back scripts that you have developed.
When you hard reset or power off on a child VM it will respawn from the parent, soft resets will not respawn (Submitted by Alan Renouf)

Troubleshooting

Instant Clone guestOS logs are stored in /var/tmp/quiesce.log
Consider enabling tracing within your customization scripts. An example of this for a shell script is using
set -x

Add additional echo or print statements like Start/Stop of certain sections like Pre/Post which can aide in reviewing the Instant Clone logs as seen in the screenshot above
For Instant Cloning Nested ESXi guestOSes, I recommend taking a snapshot after you have prepared the guest and removed any system specific information. This allows you to quickly revert back to a known state for ease of debugging. I found this to be very useful to be able to start back a known clean state while developing the customization scripts for Instant Cloning Nested ESXi
A known issue that is mentioned in the documentation of the Instant Clone cmdlets is after enabling a ParentVM for Instant Cloning, is that it is no longer available for migration to another ESXi host. The reason for this is that after powering off the VM, the "parentEnabled" boolean flag is still set to "true" which prevents the migration. Currently, there is not a work around but hopefully this will be resolved in a future update of the cmdlets. You can see this by running the following PowerCLI snippet:
(Get-VM "MyParentVM").ExtensionData.Config.ForkConfigInfo

How to VMFork aka Instant Clone Nested ESXi?

08.03.2015 by William Lam // 15 Comments

The VMware Fling's team recently released an update to the existing PowerCLI Extensions which now exposes the new VMFork aka Instant Clone capability that was introduced in vSphere 6.0. The Fling contains a set of PowerCLI Extension Modules which in turn provides new PowerCLI cmdlets for accessing the Instant Clone feature. The idea behind the Fling is to help VMware understand how customers would like to consume the Instant Clone feature not only from a CLI point of view but also from an API and UI standpoint. Prior to this, Instant Clone was only available through the use of either Horizon View or the Big Data Extensions product. I think this is a great opportunity for customers and partners to help shape how Instant Clone should be consumed more generally.

One of the use cases I had in my mind when I had first heard about the Instant Clone feature was to be able to quickly instantiate new Nested ESXi VMs. When I got the opportunity to help test out early prototypes of the Instant Clone cmdlet to help provide feedback and usability improvements, I knew I had to give Nested ESXi a try!

Requirements:

High level process:

A "preparation" script will be manually uploaded & executed within the Nested ESXi VM (Parent VM) to prep the system for Instant Cloning
As the Parent VM is quiesce, both the pre/post customization script will be uploaded to the Parent VM automatically. The "pre-customization" is also then executed within the Parent VM which properly setups the library path to the VMware Tools binary (applicable to ESXi 6.0 only) and is then placed in a ready state for creating Instant Clones
As new Instant Clone (Child VMs) are spun up, the "post-customization" script is automatically executed to add additional configurations and most importantly ensure newly created Instant Cloned Nested ESXi VMs have unique network identities

Note: For Instant Cloning regular OSes, only step 2 and 3 are really needed. Due to a known issue with VMware Tools for Nested ESXi, I have found that it is easier to prepare the Nested ESXi VM prior to quiescing and creating Instant Clones from the Parent VM.

Instructions:

Step 1 - Download and install both PowerCLI 6.0 Release 1 & Instant Clone PowerCLI Extensions Fling.

Step 2 - Perform a fresh Nested ESXi 6.0 installation in a VM, do not configure additional settings outside of enabling ESXi Shell and SSH.

Step 3 - Download the Nested ESXi 6.0 Instant Clone Scripts which contains the following four files:

prep-esxi60.sh - Prepares the Nested ESXi VM and ensures that new Child VMs will not retain the Parent VM's MAC Address which is baked in several places
pre-esxi60.sh - Pre-customization script which is used to properly setup the library paths to use the VMware Tools daemon to retrieve guest properties from PowerCLI script
post-esxi60.sh - Post-customization script which is used to apply networking configuration and hostnames for example
vmfork-esxi60.ps1 - An example PowerCLI script which issues the Instant Clone cmdlets

Note: For out of the box use, the only script that needs to be modified is the PowerCLI "vmfork-esxi60.ps1" script, the rest of the scripts should work or require very little to no modifications assuming you have followed the instruction thus far.

Step 4 - Upload the prep-esxi60.sh to Nested ESXi 6.0 VM (Parent VM) and then execute it using either the ESXi Shell over SSH or through a VMRC session. If you use SSH, you will notice that the script hangs, that is because the VMkernel interface is deleted as part of the script.

Step 5 - Next, we need to make a few edits to the vmfork-esxi60.ps1 script to update the name of your ESXi VM, along with its credentials and the full path to both the pre and post customization scripts. Below is an example of the variables that you will need to edit:

$parentvm = 'vESXi6'
$parentvm_username = 'root'
$parentvm_password = 'vmware123'
$precust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\pre-esxi60.sh'
$postcust_script = 'C:\Users\lamw\Desktop\vmfork\esxi60\post-esxi60.sh'

The section shown below will also need to be edited which contains the customization properties which are then passed down to the guestOS for configuration as part of the Instant Clone process.

 $configSettings = @{
 'hostname' = "$vmname.primp-industries";
 'ipaddress' = "192.168.1.$_"; 
 'netmask' = '255.255.255.0'; 
 'gateway' = '192.168.1.1';
 }

Step 6 - Lastly, it is time to run the script by issuing the following command:

.\vmfork-esxi60.ps1

If everything was successful, you should see a couple of new powered on Instant Cloned Nested ESXi VMs that have been fully customized and ready for use!

Note: There have been a couple of times where newly Instant Clone VMs have not been properly customized and when looking in the Instant Clone logs under /var/tmp/quiesce.log you may find "Unable to fork" error message. I usually have to re-quiesce the Parent VM which I do so by reverting back to a snapshot that captures the state after Step 4. Once I re-run the PowerCLI script, I am able to successfully deploy N-Number of Instant Clone Nested ESXi VMs. For additional best practices and tips/tricks, be sure to check out this blog post here.

Big thanks to Jim Mattson for some of his earlier research and work on this topic which made implementing these scripts much easier.

Quickly getting started with VMware AppCatalyst & AppCatalyst Vagrant Plugin

06.24.2015 by William Lam // 5 Comments

At Dockercon this week, the Cloud-Native Apps team at VMware introduced two new Tech Previews: VMware AppCatalyst and Project Bonneville. In addition, I also found out today that Fabio Rapposelli who works over in the CNA team has also released a Vagrant Plugin for VMware Catalyst as well. Having spent a couple of days playing with AppCatalyst before it was released, I thought this would be a good opportunity to show how to quickly get started with AppCatalyst but also to try out the new Vagrant Plugin that had just been released.

Before jumping in, What exactly is VMware AppCatalyst? It is a slimmed down desktop Hypervisor based on VMware Fusion that has been optimized for Developers who want to be able to quickly spin up Virtual Machines to run Docker Containers. Included with AppCatalyst is also VMware Photon, an open-source lightweight Linux container host that is used to quickly instantiate new VMs that are ready for running and building Docker Containers. Instead of a traditional GUI like VMware Fusion, AppCatalyst's is consumed completely via a REST API or CLI, which is also ideal for integrating with other 3rd Party Automation tools like Vagrant. Best of all, both VMware AppCatalyst & AppCatalyst Vagrant Plugin is completely Free!

Requirements:

Mac OS X 10.9.4 or 10.10
VMware Fusion must not be running

Quick INFO:

The AppCatalyst configuration file is located in ~/.appcatalyst.conf (if you wish to change system defaults)
The CLI to AppCatalyst is located at /opt/vmware/appcatalyst/bin/appcatalyst
The daemon for the AppCatalyst REST API is located at /opt/vmware/appcatalyst/bin/appcatalyst-daemon
SSH keys to the Photon VM is located in /opt/vmware/appcatalyst/etc/appcatalyst_insecure_ssh_key
Additional AppCatalyst Documentation can be found here

Exploring Appcatalyst CLI:

Step 1 - Download and install VMware AppCatalyst from here

Step 2 - Once AppCatalyst has been installed, you can explore the CLI and view the list of operations by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst

Step 3 - To create our first VM using AppCatalyst, run the following command:

/opt/vmware/appcatalyst/bin/appcatalyst vm create vGhetto1

Note: You will see that all VMs created by AppCatalyst will be stored in /Users/[USER]/Documents/AppCatalyst and you can change this by editing DEFAULT_VM_PATH in AppCatalyst configuration file

Step 4 - To power on our VM, run the following command:

/opt/vmware/appcatalyst/bin/appcatalyst vmpower on vGhetto1

Note: If you run into problems powering on your VM, there is a good chance that you may still have a VM that is running in VMware Fusion (also check docker-machine in case you are using that to power off any VMs provisioned using that tool)

Step 5 - Once the VM is powered on, we will probably want to retrieve the IP Address which could take a few seconds to be displayed by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst guest getip vGhetto1

Step 6 - Once we have the IP Address of our VM, we can then SSH to it using the SSH keys included with AppCatalyst by running the following command:

ssh -i /opt/vmware/appcatalyst/etc/appcatalyst_insecure_ssh_key [email protected]

At this point, you have successfully deployed a VM based on VMware Photon image using the AppCatalyst's CLI. You can now login and start running and building Docker Containers as Docker is already pre-installed on Photon. Next we will take look at using the AppCatalyst API.

Exploring Appcatalyst API:

UPDATE (01/22/16) - For the complete cheatsheet of using AppCatalyst API using cURL, check out this article here for more examples.

Step 1 - To use the AppCatalyst REST API, you will need to run the AppCatalyst Daemon by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst-daemon

Note: You can also run the AppCatalyst daemon in the background by adding '&' to the command

Step 2 - You can easily view the AppCatalyst API by opening a browser to http://localhost:8080 (assuming you did not change the port) which is built using Swagger. You can explore and even test the API using this interface if you have not worked with a Swagger interface before.

Step 3 - Lets quickly test the GET /vms operation which will list the VMs that are being managed by AppCatalyst. We will be using cURL by running the following command:

curl http://localhost:8080/api/vms

As long as the AppCatalyst daemon is running, you will be able to interact with the REST API using a variety of methods including the examples I have shown above.

Note: A known issue is that VMs powered on using the AppCatalyst API can not be managed by the CLI until they have been powered down using the API. Hopefully this issue will be resolved in a future update.

Exploring Appcatalyst Vagrant Plugin:

Step 1 - Ensure you have Vagrant already installed on your system, if not you can download it here.

Step 2 - Install the Vagrant Plugin by running the following command:

vagrant plugin install vagrant-vmware-appcatalyst

Step 3 - You can either run "vagrant init" or manually create a file name Vagrant file that contains the following:

# Set our default provider for this Vagrantfile to 'vmware_appcatalyst'
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_appcatalyst'

nodes = [
  { hostname: 'gantry-test-1', box: 'vmware/photon' },
  { hostname: 'gantry-test-2', box: 'vmware/photon' }
]

Vagrant.configure('2') do |config|

  # Configure our boxes with 1 CPU and 384MB of RAM
  config.vm.provider 'vmware_appcatalyst' do |v|
    v.vmx['numvcpus'] = '1'
    v.vmx['memsize'] = '384'
  end

  # Go through nodes and configure each of them.j
  nodes.each do |node|
    config.vm.define node[:hostname] do |node_config|
      node_config.vm.box = node[:box]
      node_config.vm.hostname = node[:hostname]
    end
  end
end

The above Vagrant file will create two VMs called gantry-test-{1,2} using the VMware Photon Box hosted on Atlas by HashiCorp.

Step 4 (Optional) - If you decide to use the example above with VMware Photon Box, you will need to install an additional Vagrant Plugin for managing Photon Guest by running the following command:

vagrant plugin install vagrant-guests-photon

Step 5 - Ensure that the AppCatalyst daemon is running before proceeding with the next step as the Vagrant Plugin uses the AppCatalyst REST API.

Step 6 - To start the deployment, run the following command:

vagrant up --provider=vmware_appcatalyst

Step 7 - To login to one of the VMs that have been created by Vagrant, simply run the following command and specify the name of the VM to SSH to:

vagrant ssh gantry-test-1

Step 8 - Finally, we can also confirm that the VMs created by the AppCatalyst Vagrant Plugin is also visible using just the AppCatalyst REST API and we can perform another "GET" operation to verify the two VMs that we had just created.

Hopefully this quick primer has been useful in getting you introduced and started with VMware AppCatalyst as well as the new AppCatalyst Vagrant Plugin. If you have any feedback or questions, feel free to leave a comment in the VMware AppCatalyst Forum and for feedback/questions on AppCatalyst Vagrant Plugin, you can file issues here.