WilliamLam.com

VM serial logging to the rescue for capturing Nested ESXi PSOD

by // Leave a Comment

I frequently deploy pre-releases of our software to help test and provide early feedback to our Engineering teams. One piece of software that I deploy some what frequently is our ESXi Hypervisor and the best way to deploy it, is of course inside of a Virtual Machine or commonly referred to as Nested ESXi.

Most recently while testing a new ESXi build in my lab (screenshot below is for demo purposes, not the actual PSOD image), I encountered an ESXi purple screen of death (PSOD) during the bootup of the ESXi Installer itself. Since ESXi had not been installed, there was no place for ESXi to actually store the core dumps which made it challenging when filing a bug with Engineering as screenshots may not always contain all the necessary details.

Screen Shot 2016-03-21 at 9.26.08 AM
Luckily, because we are running in a VM, a really neat feature that VMware has supported for quite some time now is configuring a virtual serial port for logging purposes. In fact, one of the neatest feature from a troubleshooting standpoint was the introduction of the Virtual Serial Port Concentrator (vSPC) feature in vSphere 5.0 which allowed a VM to log directly to a serial console server just like you would for physical servers. You of course had few other options of either logging directly to the serial port of the physical ESXi, named pipe or simply to a file that lived on a vSphere Datastore.

Given this was a home lab setup, the easiest method was to simply output to a file. To add a virtual serial port, you can either use the vSphere Web/C# Client or the vSphere APIs. Since this is not something I need to do often, I just used the UI. Below is a screenshot using the vSphere Web Client and once you have added the virtual serial port, you need to specify the filename and where to the store the output file by clicking on the "Browse" button.

vm-serial-logging
If the GuestOS which includes ESXi has been configured to output to a serial port, the next time there is an issue and you can easily captured the output to a file instead of just relying on a screenshot. One additional tip which might be useful is by default, vSphere will prompt whether you want to replace or append to the configured output file. If you wish to always replace, you can add the following VM Advanced Setting and you will not get prompted in the UI.

answer.msg.serial.file.open = "Replace"

Virtual serial ports are supported on both vSphere (vCenter Server + ESXi) as well as our hosted products VMware Fusion and Workstation.

Getting started with the new VSAN 6.2 Management API

by // 4 Comments

As I have previously written, with the release of VSAN 6.2 (vSphere 6.0 Update 2), a new VSAN Management API has been introduced which allows developers, partners and administrators to automate all aspects of VSAN functionality including: complete lifecycle (install, upgrade, patch), monitoring (including RVC and VSAN Observer capabilities), configuration and troubleshooting. Simply put, anything that you can do from the vSphere Web Client UI or the RVC CLI from a VSAN standpoint, you will be able to completely automate using one of the four new VSAN Management SDKs: Python, Ruby, Java and C#.

In this article, I will show you how to quickly get started with the new VSAN Management API by exercising two of the VSAN Management SDKs: Python and Ruby. Another must bookmark is the VSAN Management API Reference Guide which provides more details on the individual APIs and how they work.

Step 1 - Download the VSAN Management SDK of your choice. You can find the VSAN Management SDK downloads in either of two locations:

In this example, I will be using the VSAN Management for Python and Ruby.

Step 2 - Extract the VSAN Management SDK zip file which should give you a directory that contains a README on how to setup the SDK and three folders as shown in the screenshot below:

Screen Shot 2016-03-17 at 6.27.58 AM
The bindings directory contains the language specific library to the VSAN Management API. The docs folder contains the offline copy of the VSAN Management API Reference Guide and lastly the sample directory contains a basic sample to connect to VSAN Cluster as well as an individual ESXi host contributing to a VSAN Cluster.

Step 3 - Each of the VSAN Management SDKs extends the existing vSphere Management SDKs. This means that you will need to have the appropriate vSphere Management SDK installed on your system before you can proceed further. In our example, Python requires pyvmomi (vSphere SDK for Python) and Ruby requires rbvmomi (vSphere SDK for Ruby). If you are on Mac OS X, it is pretty easy to install these packages. Make sure you are running the latest version of these SDKs.

Installing pyvmomi:

pip install pyvmomi

Upgrading pyvmomi: (if you already have it installed)

pip install --upgrade pyvmomi

Installing rbvmomi:

gem install rbvmomi

Step 4 - Copy the VSAN Management SDK library file over to the samples directory.

VSAN Mgmt SDK for Python:

cp bindings/vsanmgmtObjects.py samplecode/

VSAN Mgmt SDK for Ruby:

cp bindings/vsanmgmt.api.rb samplecode/

Step 5 - At this point, we can quickly verify that everything was setup correctly by going into the samplecode directory and then run one of the following commands below. If everything is working as expected, then you should see the usage information being printed out. If you do not, the issue is most likely with vSphere Management SDK either not being the latest version or not configured in the default library path for the sample to access.

VSAN Mgmt SDK for Python:

python vsanapisamples.py

Screen Shot 2016-03-17 at 6.43.32 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb

Screen Shot 2016-03-17 at 6.43.56 AM
Step 6 - Now that we have verified our VSAN Management SDK installation was successful, we can now connect to a real VSAN Cluster. To do so, run the following command and specify your vCenter Server along with the credentials as well as the name of the VSAN Cluster. If successful, you should see the status for each of your VSAN hosts and its current state as seen in the screenshot below.

VSAN Mgmt SDK for Python:

python vsanapisamples.py -s 192.168.1.139 -u '*protected email*' -p 'VMware1!' --cluster VSAN-Cluster

Screen Shot 2016-03-17 at 6.54.38 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb -o 192.168.1.139 -u '*protected email*' -k -p 'VMware1!' VSAN-Cluster

Screen Shot 2016-03-17 at 6.56.34 AM
Step 7 - Each individual ESXi hosts that participate in the VSAN Cluster also exposes an VSAN Management API endpoint. We can use this exact same sample to connect to one of the hosts to get some additional information. To do so, run the following command and specify your ESXi hosts along with the credentials.

VSAN Mgmt SDK for Python:

python vsanapisamples.py -s 192.168.1.190 -u root -p vmware123

Screen Shot 2016-03-17 at 7.00.28 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb -o 192.168.1.190 -u root -p vmware123

Screen Shot 2016-03-17 at 6.59.46 AM
As you can see, it is pretty straight forward on getting the new VSAN Management SDK up and running. The provided sample only scratches the surface of what is possible and for a complete list of capabilities within the new VSAN Management API, be sure to check out the VSAN Management API Reference document for more information. I am really looking forward to seeing what solutions our customers and partners develop using this new API. If you would like to contribute code samples back to the community or just find new samples be sure to check out the VMware Developer Center Sample Exchange. 

Apple Mac Pro 6,1 PCIe SSD issue resolved w/ESXi 6.0 Update 2

by // 6 Comments

Early last year, the new Apple Mac Pro 6,1 (aka black can design) was certified and fully supported on vSphere 6.0 which I had blogged about here. Several months later, customers discovered that some of the newer Mac Pro 6,1 units were shipping with different model of their PCIe SSD device than what was originally released at GA. This was problematic because ESXi was not aware of this newer device and could not detect during or after installation. Although a work around was identified for customers looking to install either ESXi 5.x or 6.x on the newer Apple Mac Pros, it definitely was not ideal.

It has taken a bit longer than expected, but the issue has now been resolved with the latest release of ESXi 6.0 Update 2. A similar fix will be available for customers running ESXi 5.5 in a future update. You can find the direct download for ESXi 6.0 Update 2 in link below which includes a pointer to the release notes in case you are interested in other fixes included in this release.