WilliamLam.com

Using vSphere Guest Operations API on macOS Guests? 

by // 2 Comments

I have written a number of articles exploring the usage and some of the cool tricks that the vSphere Guest Operations (GuestOps) feature provides which you can be found here, here, here and here. I have been a huge fan and supporter of GuestOps since the early days where it was formally known as the VIX API. Having used GuestOps across many different GuestOS types including Nested ESXi, I have to admit, I had never tried it against an Apple macOS guests. I recently had a customer reach out who was looking to use the GuestOps API via PowerCLI (Invoke-VMScript) to automate updates against his guestOS templates that span across Windows, Linux and macOS (from 10.7 to latest). The customer was able to get all guestOSes working except for macOS.

Since I had never tried this before, I spun up my Apple Mac Mini which happen to have a macOS 10.11 (El Capitan) guests running. I tried using the vSphere API GuestOps directly to see if this was a PowerCLI and/or API issue. I too ran into issues and after enabling VMware Tools debugging on the guests (which you can find more details below), I found that it hit the following error:

[Jun 28 06:35:42.805] [   debug] [vix] >VixToolsImpersonateUser
[Jun 28 06:35:42.925] [ warning] [vmsvc] Failed to set gid for user root

Reaching out to Engineering regarding the problem, I came to learn that this particular issue was due to a syscall change made by Apple starting with macOS 10.10.3 and newer. Although the change was a positive thing from a security standpoint, it did break the GuestOps functionality. The good news was that this was already resolved with VMware Tools 10.1 or later. When I had initially provisioned the macOS guests, the latest VMware Tools at the time was 9.10.5. After I applied the latest version which is currently 10.1.7, the issue went away and I was able to successfully use the GuestOps API on my macOS guests.

Below are examples of running the system_profiler SPSoftwareDataType command using both the Invoke-VMScript cmdlet as well as the vSphere API and PowerCLI to consume the GuestOps APIs. Both approaches delivers the exact same outcome, the one benefit of using Invoke-VMScript is that if you want to easily return output from a given command, the cmdlet already does the heavy lifting. If you notice in the native vSphere API case, you do not get output but rather just the PID ID. If you want to return the output, you need to first save it into a file and then download the file to your client system, which may not be ideal for interactive usage but it all depends on your use case.

[Read more...]

How to determine when a Virtual Machine is ready for additional operations?

by // 3 Comments

This might sound like a pretty simple and basic question, right? However, the answer will actually depend on what you are trying to do. The motivation for this blog post actually stemmed from a conversation I had with one of our Engineers who works over in our core Platform Management Infrastructure which includes VMware Tools, Guest OS Customization, Guest Operations, etc. Although I was aware of some of these methods in determining when a VM was ready, I came to learn about a few new other methods that I was not aware of before. This is really one of the things I love about my job, constantly learning new things and diving deeper into our technologies and sharing that back with our customers to help enable them and their business.

So, what does it even mean for a Virtual Machine to be "ready"? Is it when the VM is powered on? Is it when the VM obtains an IP Address? Is it when the GuestOS is fully customized or is it when I can SSH or RDP to the system? Depending on what you are trying to accomplish, the answer will vary. In addition to that, there are two distinct VM states to consider. The first being the actual Virtual Machine state and the second being the GuestOS state.

Virtual Machine State

The VM state is pretty straight forward, it describes the "Hard" power state of the VM which can either be powered on, powered off or suspended. As you can probably guess, you will need to make sure the VM is powered on before you attempt to check whether the GuestOS is ready 🙂

Using the vSphere API, you can find the VM powerState under:

VirtualMachine->Runtime->powerState

Here is a PowerCLI snippet using the out of box cmdlet to get the power state for a VM named "air":

Get-VM -Name air | Select PowerState

Here is a PowerCLI snippet that uses the vSphere API via Get-View to retrieve the exact same information:

$vm = Get-View -ViewType Virtualmachine -Property Name,Runtime -Filter @{"name"="air"}
$vm.Runtime.PowerState

[Read more...]

Using PowerCLI to invoke Guest Operations API to a Nested ESXi VM

by // 1 Comment

In my opinion, the Guest Operations API in vSphere is still one of the most powerful Virtual Machine capabilities that is available to vSphere Administrators and anyone else who integrates with the vSphere Platform. The Guest Operations API allows users to perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a handy feature for troubleshooting and can even extend into application level provisioning through a single API.

Obviously, I am a huge fan of this capability and have used it myself on more than one occasion. However, one limitation that I discovered awhile back when using the Guest Operations API with Nested ESXi VMs is that it threw some very strange memory related errors. It was only recently did I find out that there was a known issue with the VMware Tools for Nested ESXi, both the installable VIB and the pre-installed version in ESXi 6.0 on how the guest operations are executed. The good news is that for now, there is a simple workaround that can be applied when using the Guest Operations API.

You will need to add the following option, which runs the command under a specific resource group in the ESXi Shell:

'++group=host/vim/tmp'

Here is an example if I were to run the 'echo' command:

/bin/echo '++group=host/vim/tmp' "hello world"

A more interesting example would be to issue ESXCLI commands using the Guest Operations API, perhaps configuring the welcome message?

/bin/python '++group=host/vim/tmp' '/bin/esxcli.py system welcomemsg set -m "vGhetto Was Here"'

Notice, we need to pass in the resource group command to the "python" binary versus ESXCLI binary. The reason for this is that /bin/esxcli is really just a symlink to /bin/esxcli.py which is just a Python wrapper. The actual command being launched is the python interpreter and the arguments to the command is /bin/esxcli.py and the ESXCLI arguments itself.

For those who prefer to consume the Guest Operations API without having to directly use the vSphere API, you can use PowerCLI and use the Invoke-VMScript cmdlet. The problem with that is due to the way the cmdlet has been abstracted, the necessary underlying API details can not be accessed due to certain assumed defaults which can not be overridden or extended. This is a general problem that I have seen in more than one occasion where the abstraction is to make the consumption of the API simpler but in certain cases, it can also inhibit the use of the underlying API feature.

In this case, we will actually need to call into the vSphere API and using PowerCLI as an example, I have created a script called runGuestOpsInNestedESXiVM.ps1 which implements the specific Guest Operations APIs to issue commands to a Nested ESXi VM.

Here is an example of running the script and configuring the welcome message using ESXCLI:

guest_operations_api_nested_esxi