WilliamLam.com

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

by // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI:

guest_operations_api_nested_esxi

How my daughter almost received an extra shot (vaccine) for her birthday?

by // 4 Comments

This past week, our daughter turned 4 months old and this meant another routine visit to the pediatrician office for her vaccinations. It started out like any other visit. The medical assistant took her weight and measurements, and reviewed the vaccines that were due this visit. She noted that our daughter would receive three shots: DTap/Hib/Polio, Hepatitis B, Pneumococcal, and an oral drink for Rotavirus.

Our daughter received a clean bill of health from the pediatrician and we were also pleasantly surprised to hear that she ranked in over 85th percentile for her weight, height, and head circumference. The last thing the pediatrician reviewed with us was her vaccinations. She briefly checked off on the 4-month overview sheet that she would receive the DTap/Hib/Polio, Pneumococcal, and Rotavirus. She crossed out Hepatitis B and said she does not need it this visit because she already received two doses (the schedule for Hep B is at 0, 1-2, and 6 months). The order was placed and the pediatrician quickly left the room while we waited for the medical assistant to return.

During this time, my wife realized that the medical assistant had mentioned that our daughter would be receiving 3 shots, not 2 like what the pediatrician had said. Sure enough, when the medical assistant returned with her tray of shots, there were 3 shots and a drink. My wife quickly stated to the medical assistant that the doctor said there were to be only 2 shots. The medical assistant then went over to the computer to double check the orders and stated that the doctor changed the orders (unbeknownst to her). Luckily, my wife works in healthcare and was able to catch the mistake before it had happened, making this incident a “near miss.”

While driving home, I could not help but think about the near mishap with our daughter. Some might say this was the fault of the medical assistant as the doctor had placed the correct order, but honestly I am not sure I would place the complete blame on her for the mistake. As I discussed this further with my wife thinking through some of the possible scenarios on how this could have happened

  • Maybe the medical assistant had the vaccination schedule memorized as they work with children all day long, but perhaps she had forgotten it or did not even bother to look at the actual order?
  • Maybe the medical assistant miss-read our daughters medical record and thought she was due for Hep B shot?
  • Maybe the doctor actually did change or put in the order right before she had prepared up the shots but didn’t bother to verbally tell the medical assistant?
  • Maybe the medical assistant was just tired that day?

In all the scenarios that we could come up with, it was clear to me that the mistake was due to the "human factor".

In my mind, why was there not an automated system and safety measures put into place that could alarm the administrator of the drug between what was ordered and what was being checked out, even to the point before the drugs were being administered? One might say having an Electronic Medical Record (EMR) system could have prevented this, especially if the drugs have bar codes that could be scanned during the preparation of the order. However, this particular clinic had an EMR system and even if the above process existed, I still believe this mistake could have easily been overlooked since someone would still have to remember to scan the drugs. What if the person giving the drugs was in a rush and just grabbed the drugs in advanced?

This incident actually reminded me of a local story that I had read several months back where a child at the UCSF Benioff Children's Hospital was accidentally given dosage that was 38 times over their normal dose. This incident occurred because a nurse had completely relied on the EMR system and had accepted the incorrect measurement conversion which then of course translated to a much higher dosage to be prescribed. Even with a modern hospital where an EMR system was used and was tightly integrated with patient drug orders and the distribution of the drugs, the system was still unable to prevent this mistake from happening.

I know we cannot solely rely on ourselves because of the "human factor" with its unpredictable nature and the primary reason we humans are not good at performing repetitive tasks or even some basic math for that matter. However, I also do not think we are ready (yet?) to place our complete trust onto computers and remove ourselves completely from the equation. What I do know is that whatever solution we come up with, we desperately need an automated system that is "naturally integrated" into the existing processes and workflows of our Healthcare system. The system needs to be an enabler to physicians, nurses, medical assistant, etc. but it cannot be a distraction or an inhibitor when providing patient healthcare.

Sadly, that is not the case today and with a diverse age group with varying digital literacy skills, I often hear from many of my friends who work in the Healthcare system that existing EMR systems are still as problematic as they are helpful. I think ultimately, whatever technology we use, it cannot be something that we have to think about using but it should be seamlessly integrated into the overall patient healthcare process with proper checks and balances.

Today, I learned a valuable lesson and I think that all parents should be vigilant when visiting the doctor, even if it is for a routine checkup. Technology and Healthcare has always been a topic of interest for me and I really look forward to further advancements in these fields as they intersect to better improve our lives.

Schedule automated backups of VCSA 6.0 vPostgres embedded database to Amazon S3

by // 6 Comments

A couple of weeks back, I had received a question around backing up and restoring the Embedded vPostgres Database found within the new vCenter Server Appliance (VCSA) 6.0. At the time, the only thing I had seen was KB 2110294 and vSphere 6.0 Documentation here which recommends that a full VM backup be taken for either the vCenter Server for Windows as well as the VCSA to be able to properly protect your vCenter Server.

It was just recently that I came across VMware KB 2091961 which provides some details on just backing up the individual vPostgres DB. Having said that, just having a database backup is not sufficient to perform a proper restore in the case of completely losing your vCenter Server. There are other sources of data within the vCenter Server as well as the Platform Services Controller that are required and restoring a database would only work if you still had access to the original system. This is why a full VM backup is still the recommended approach.

For those who want to be able to just restore the database, the process listed in the KB is currently a manual step which uses a Python script that is provided in the KB. I thought it would be useful to demonstrate how you could schedule continuous backups during off peak hours using a simple cronjob and more interesting to me, is the how and where of the overall process? One option would be to mount a backup NFS share directly onto the VCSA and place all backups on that volume. Another option could have the backups directly uploaded to a Storage Cloud Provider like an Amazon S3 for example. I decided to take a look into the latter option.

In searching online, I found that Amazon offers a nice CLI called AWS CLI which provides S3 functionality like the 'cp' command and I was able to install it on the VCSA without any issues. You can find the instructions for installing the AWS CLI here and I would also recommend that you create a dedicate user assigned to the S3 bucket for storing the backups and then following the steps here to configure access to the AWS CLI. When asked about the Amazon Region as part of the configuration, I found this page to be helpful in listing the region names.

Disclaimer: Installing 3rd Party tools and products on the (VCSA) is not officially supported, you may be asked by GSS to remove them during troubleshooting.

If everything is installed correct, you should be able to run the following command to ensure you can reach the S3 bucket:

aws s3 ls s3:\\[NAME-OF-YOUR-S3-BUCKET]

To tie everything together, I created a simple shell script called backup_vcsa_vpostgres_db.sh which contains a couple of variables that you will need to edit:

  • VPOSTGRES_BACKUP_SCRIPT - The path to the Python vPostgres backup script
  • AWS_CLI - The full path to the AWS CLI binary
  • AWS_S3_BUCKET - The name of the S3 bucket using syntax s3:\\NAME-OF-YOUR-S3-BUCKET

Before creating the cronjob, I would recommend that you manually run the script to ensure everything works as expected and you are able to upload to your S3 bucket. Here is an example execution of the script which is backing up to my S3 bucket which I called "vcsa-backup".

backup-vcsa-vpostgres-db-to-s3-1
You can quickly verify that the backup has been uploaded to the S3 bucket by running the "ls" command as shown earlier or you can login to the Amazon S3 console and you should be able to see the backup files as shown in the screenshot below.

backup-vcsa-vpostgres-db-to-s3-0
To schedule the script to automatically run during a certain period, you can create a cronjob by running the following command:

crontab -e

For more information about setting up a cronjob, you can take a look here or Google your favorite resource. If you plan on storing backups with a Cloud Storage Provider and do not have direct internet access like most customers do, you can configure an HTTP(S) proxy by editing /etc/sysconfig/proxy If you prefer not to install AWS CLI, you can also use this simple bash script which uses an HTTP POST to upload to Amazon S3.