WilliamLam.com

How to create a custom ESXi VIB to execute a script once?

by // 17 Comments

Earlier this week I received a question from a customer who was interested in creating a custom ESXi VIB that could execute a specific script within the payload as part of the installation and only running it once. This was a fairly common request that I have seen in the past and as far as I knew, this type of behavior was not possible. What was unique about this particular custom inquiry was that they mentioned they found some references on this being possible. Being the curious person I am, I decided to take another look and reach out to a few folks in Engineering.

After speaking with one of the Engineers familiar with our VIB specification, to my surprise I learned that this type of behavior was actually indeed possible but was not very well documented externally. The typical use case for this is usually to apply certain configurations which are not exposed through the traditional ESXi interfaces like the vSphere API/CLI which includes Host Profiles. One example that comes to mind is being able to deploy a VIB across several hundred ESXi hosts that could configure a specific ESXi Advanced Setting which might be marked hidden. Another example would be updating a configuration file like /etc/vmware/config or running a series of ESXi Shell commands which can not available in the vSphere API and require the use of SSH and the ESXi Shell. There are many other examples, but this should give you an idea of some of the common use cases that I have heard from customers.

Here is what is required to execute a specific script as part of the VIB installation:

  • Created an "init" script which needs to be stored under /etc/init.d as part of your VIB payload (e.g. /etc/init.d/lamw-special-vib)
  • You must ensure that the "live-install-allowed" param is set to true in your VIB's descriptor.xml file (this assumes the changes can be applied without requiring reboot to take affect)
  • The init script will need to parse special keywords passed as command-line arguments

Here is a table showing the pair of special keywords that are passed to the init script as command-line arguments during a VIB install, upgrade or remove:

Arg1 Arg2 VIB Operation
start install VIB install
start upgrade VIB upgrade
stop remove VIB removal

It is up to creator of the init script to handle the different VIB operations by parsing the command-line arguments which would then determine the operations that would get executed within the script. This not only allows you to control the commands that are executed during an installation of a VIB but it also allows you specify the commands to run upon the removal of a VIB which is quite handy for properly cleaning up an uninstall. In addition, since these special keywords are not passed as part of the ESXi boot up process, the commands within the script will not execute and ensures it only runs once during the install.

Here is very simple shell script (you could also do this in Python as well) which demonstrates how to handle the three different types of VIB operations and then uses the "logger" utility to write some output to syslog:

#!/bin/sh

ARG1=$1
ARG2=$2

if [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "install" ]]; then
 # commands to run go here #
 /bin/logger "William's custom script ran on start-install"
elif [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "upgrade" ]]; then
 /bin/logger "William's custom script ran on start-upgrade"
 # commands to run go here #
elif [[ "${ARG1}" == "stop" ]] && [[ "${ARG2}" == "remove" ]]; then
 /bin/logger "William's custom script ran on stop-remove"
 # commands to run go here #
fi

As you can see, this allows you to perform a variety of tasks through the use of a custom ESXi VIB which is a great way to be able to roll out a set of changes that may not be possible using either the vSphere API or Host Profiles for example. Best of all, this solution does not require the use of SSH which is great since many customers already disable this by default.

For more information on creating a custom VIB, be sure to check out my blog post here and you can even use this Docker image I created for building custom ESXi VIBs.

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

by // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI:

guest_operations_api_nested_esxi

How my daughter almost received an extra shot (vaccine) for her birthday?

by // 4 Comments

This past week, our daughter turned 4 months old and this meant another routine visit to the pediatrician office for her vaccinations. It started out like any other visit. The medical assistant took her weight and measurements, and reviewed the vaccines that were due this visit. She noted that our daughter would receive three shots: DTap/Hib/Polio, Hepatitis B, Pneumococcal, and an oral drink for Rotavirus.

Our daughter received a clean bill of health from the pediatrician and we were also pleasantly surprised to hear that she ranked in over 85th percentile for her weight, height, and head circumference. The last thing the pediatrician reviewed with us was her vaccinations. She briefly checked off on the 4-month overview sheet that she would receive the DTap/Hib/Polio, Pneumococcal, and Rotavirus. She crossed out Hepatitis B and said she does not need it this visit because she already received two doses (the schedule for Hep B is at 0, 1-2, and 6 months). The order was placed and the pediatrician quickly left the room while we waited for the medical assistant to return.

During this time, my wife realized that the medical assistant had mentioned that our daughter would be receiving 3 shots, not 2 like what the pediatrician had said. Sure enough, when the medical assistant returned with her tray of shots, there were 3 shots and a drink. My wife quickly stated to the medical assistant that the doctor said there were to be only 2 shots. The medical assistant then went over to the computer to double check the orders and stated that the doctor changed the orders (unbeknownst to her). Luckily, my wife works in healthcare and was able to catch the mistake before it had happened, making this incident a “near miss.”

While driving home, I could not help but think about the near mishap with our daughter. Some might say this was the fault of the medical assistant as the doctor had placed the correct order, but honestly I am not sure I would place the complete blame on her for the mistake. As I discussed this further with my wife thinking through some of the possible scenarios on how this could have happened

  • Maybe the medical assistant had the vaccination schedule memorized as they work with children all day long, but perhaps she had forgotten it or did not even bother to look at the actual order?
  • Maybe the medical assistant miss-read our daughters medical record and thought she was due for Hep B shot?
  • Maybe the doctor actually did change or put in the order right before she had prepared up the shots but didn’t bother to verbally tell the medical assistant?
  • Maybe the medical assistant was just tired that day?

In all the scenarios that we could come up with, it was clear to me that the mistake was due to the "human factor".

In my mind, why was there not an automated system and safety measures put into place that could alarm the administrator of the drug between what was ordered and what was being checked out, even to the point before the drugs were being administered? One might say having an Electronic Medical Record (EMR) system could have prevented this, especially if the drugs have bar codes that could be scanned during the preparation of the order. However, this particular clinic had an EMR system and even if the above process existed, I still believe this mistake could have easily been overlooked since someone would still have to remember to scan the drugs. What if the person giving the drugs was in a rush and just grabbed the drugs in advanced?

This incident actually reminded me of a local story that I had read several months back where a child at the UCSF Benioff Children's Hospital was accidentally given dosage that was 38 times over their normal dose. This incident occurred because a nurse had completely relied on the EMR system and had accepted the incorrect measurement conversion which then of course translated to a much higher dosage to be prescribed. Even with a modern hospital where an EMR system was used and was tightly integrated with patient drug orders and the distribution of the drugs, the system was still unable to prevent this mistake from happening.

I know we cannot solely rely on ourselves because of the "human factor" with its unpredictable nature and the primary reason we humans are not good at performing repetitive tasks or even some basic math for that matter. However, I also do not think we are ready (yet?) to place our complete trust onto computers and remove ourselves completely from the equation. What I do know is that whatever solution we come up with, we desperately need an automated system that is "naturally integrated" into the existing processes and workflows of our Healthcare system. The system needs to be an enabler to physicians, nurses, medical assistant, etc. but it cannot be a distraction or an inhibitor when providing patient healthcare.

Sadly, that is not the case today and with a diverse age group with varying digital literacy skills, I often hear from many of my friends who work in the Healthcare system that existing EMR systems are still as problematic as they are helpful. I think ultimately, whatever technology we use, it cannot be something that we have to think about using but it should be seamlessly integrated into the overall patient healthcare process with proper checks and balances.

Today, I learned a valuable lesson and I think that all parents should be vigilant when visiting the doctor, even if it is for a routine checkup. Technology and Healthcare has always been a topic of interest for me and I really look forward to further advancements in these fields as they intersect to better improve our lives.