WilliamLam.com

How to create a custom ESXi VIB to execute a script once?

by // 16 Comments

Earlier this week I received a question from a customer who was interested in creating a custom ESXi VIB that could execute a specific script within the payload as part of the installation and only running it once. This was a fairly common request that I have seen in the past and as far as I knew, this type of behavior was not possible. What was unique about this particular custom inquiry was that they mentioned they found some references on this being possible. Being the curious person I am, I decided to take another look and reach out to a few folks in Engineering.

After speaking with one of the Engineers familiar with our VIB specification, to my surprise I learned that this type of behavior was actually indeed possible but was not very well documented externally. The typical use case for this is usually to apply certain configurations which are not exposed through the traditional ESXi interfaces like the vSphere API/CLI which includes Host Profiles. One example that comes to mind is being able to deploy a VIB across several hundred ESXi hosts that could configure a specific ESXi Advanced Setting which might be marked hidden. Another example would be updating a configuration file like /etc/vmware/config or running a series of ESXi Shell commands which can not available in the vSphere API and require the use of SSH and the ESXi Shell. There are many other examples, but this should give you an idea of some of the common use cases that I have heard from customers.

Here is what is required to execute a specific script as part of the VIB installation:

  • Created an "init" script which needs to be stored under /etc/init.d as part of your VIB payload (e.g. /etc/init.d/lamw-special-vib)
  • You must ensure that the "live-install-allowed" param is set to true in your VIB's descriptor.xml file (this assumes the changes can be applied without requiring reboot to take affect)
  • The init script will need to parse special keywords passed as command-line arguments

Here is a table showing the pair of special keywords that are passed to the init script as command-line arguments during a VIB install, upgrade or remove:

Arg1 Arg2 VIB Operation
start install VIB install
start upgrade VIB upgrade
stop remove VIB removal

It is up to creator of the init script to handle the different VIB operations by parsing the command-line arguments which would then determine the operations that would get executed within the script. This not only allows you to control the commands that are executed during an installation of a VIB but it also allows you specify the commands to run upon the removal of a VIB which is quite handy for properly cleaning up an uninstall. In addition, since these special keywords are not passed as part of the ESXi boot up process, the commands within the script will not execute and ensures it only runs once during the install.

Here is very simple shell script (you could also do this in Python as well) which demonstrates how to handle the three different types of VIB operations and then uses the "logger" utility to write some output to syslog:

#!/bin/sh

ARG1=$1
ARG2=$2

if [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "install" ]]; then
 # commands to run go here #
 /bin/logger "William's custom script ran on start-install"
elif [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "upgrade" ]]; then
 /bin/logger "William's custom script ran on start-upgrade"
 # commands to run go here #
elif [[ "${ARG1}" == "stop" ]] && [[ "${ARG2}" == "remove" ]]; then
 /bin/logger "William's custom script ran on stop-remove"
 # commands to run go here #
fi

As you can see, this allows you to perform a variety of tasks through the use of a custom ESXi VIB which is a great way to be able to roll out a set of changes that may not be possible using either the vSphere API or Host Profiles for example. Best of all, this solution does not require the use of SSH which is great since many customers already disable this by default.

For more information on creating a custom VIB, be sure to check out my blog post here and you can even use this Docker image I created for building custom ESXi VIBs.

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

by // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI:

guest_operations_api_nested_esxi

Schedule automated backups of VCSA 6.0 vPostgres embedded database to Amazon S3

by // 6 Comments

A couple of weeks back, I had received a question around backing up and restoring the Embedded vPostgres Database found within the new vCenter Server Appliance (VCSA) 6.0. At the time, the only thing I had seen was KB 2110294 and vSphere 6.0 Documentation here which recommends that a full VM backup be taken for either the vCenter Server for Windows as well as the VCSA to be able to properly protect your vCenter Server.

It was just recently that I came across VMware KB 2091961 which provides some details on just backing up the individual vPostgres DB. Having said that, just having a database backup is not sufficient to perform a proper restore in the case of completely losing your vCenter Server. There are other sources of data within the vCenter Server as well as the Platform Services Controller that are required and restoring a database would only work if you still had access to the original system. This is why a full VM backup is still the recommended approach.

For those who want to be able to just restore the database, the process listed in the KB is currently a manual step which uses a Python script that is provided in the KB. I thought it would be useful to demonstrate how you could schedule continuous backups during off peak hours using a simple cronjob and more interesting to me, is the how and where of the overall process? One option would be to mount a backup NFS share directly onto the VCSA and place all backups on that volume. Another option could have the backups directly uploaded to a Storage Cloud Provider like an Amazon S3 for example. I decided to take a look into the latter option.

In searching online, I found that Amazon offers a nice CLI called AWS CLI which provides S3 functionality like the 'cp' command and I was able to install it on the VCSA without any issues. You can find the instructions for installing the AWS CLI here and I would also recommend that you create a dedicate user assigned to the S3 bucket for storing the backups and then following the steps here to configure access to the AWS CLI. When asked about the Amazon Region as part of the configuration, I found this page to be helpful in listing the region names.

Disclaimer: Installing 3rd Party tools and products on the (VCSA) is not officially supported, you may be asked by GSS to remove them during troubleshooting.

If everything is installed correct, you should be able to run the following command to ensure you can reach the S3 bucket:

aws s3 ls s3:\\[NAME-OF-YOUR-S3-BUCKET]

To tie everything together, I created a simple shell script called backup_vcsa_vpostgres_db.sh which contains a couple of variables that you will need to edit:

  • VPOSTGRES_BACKUP_SCRIPT - The path to the Python vPostgres backup script
  • AWS_CLI - The full path to the AWS CLI binary
  • AWS_S3_BUCKET - The name of the S3 bucket using syntax s3:\\NAME-OF-YOUR-S3-BUCKET

Before creating the cronjob, I would recommend that you manually run the script to ensure everything works as expected and you are able to upload to your S3 bucket. Here is an example execution of the script which is backing up to my S3 bucket which I called "vcsa-backup".

backup-vcsa-vpostgres-db-to-s3-1
You can quickly verify that the backup has been uploaded to the S3 bucket by running the "ls" command as shown earlier or you can login to the Amazon S3 console and you should be able to see the backup files as shown in the screenshot below.

backup-vcsa-vpostgres-db-to-s3-0
To schedule the script to automatically run during a certain period, you can create a cronjob by running the following command:

crontab -e

For more information about setting up a cronjob, you can take a look here or Google your favorite resource. If you plan on storing backups with a Cloud Storage Provider and do not have direct internet access like most customers do, you can configure an HTTP(S) proxy by editing /etc/sysconfig/proxy If you prefer not to install AWS CLI, you can also use this simple bash script which uses an HTTP POST to upload to Amazon S3.