WilliamLam.com

Two coredump partitions in ESXi 5.5?

by // 8 Comments

A couple of days back I had to re-install ESXi on a physical host for some troubleshooting purposes and while looking at the partitions on the disks using ESXCLI, I noticed the fresh ESXi installation had created two coredump partitions.

two-coredump-partition-0
I was quite surprised to see two, since normally you would just have one configured. I even asked a colleague if he had ever see this before and he had not, so I wanted to double check that there was in fact two coredump partitions being created which I verified by using partedUtil.

two-coredump-partition-1
As you can see from the screenshot above, there are definitely two coredump partitions. I took a look at our vSphere documentation, but did not find any mention of this. I decided to look internally and found that this is actually a new behavior that was introduced in ESXi 5.5. From what I can tell, the second coredump partition which is 2.5GB was created to ensure that there was sufficient space to handle ESXi hosts configured with a huge amount of memory (up to 4TB) if a coredump were to occur. This new coredump partition is only created on a fresh ESXi install, for upgrade scenarios the original partition structure is preserved. I suspect even on the fresh install, the original coredump partition was kept for potential backwards compatibility.

This definitely made sense given the reason. I guess this actually raises another interesting point from an operational point of view that though upgrades may be preferred, there are also good reasons to perform a fresh install over an upgrade. In this case, to ensure we do not break past requirements/assumptions, we could not just automatically expand or create a larger coredump partition to adhere to new requirements. This is actually not the first instance of this, here are two additional examples in which a fresh installation would have potentially yielded a more optimal environment:

"Community" VSAN Storage Controller Queue Depth List

by // 13 Comments

After reading this Reddit thread about a customers recent experience with VSAN, I have been thinking about how customers can actually tell what the queue depth is for a particular storage controller? Currently, the VSAN HCL for storage controllers does not provide any queue depth information and from my understanding this information may not always be easy to find or documented.

I know Duncan Epping has even "crowd source" for some of this information and currently his list seems to be the best at the moment. However, if you look through his list carefully, you will see that it only contains a very small subset of the supported storage controllers found on the VSAN HCL as it also contains non-supported storage controllers. I was thinking about how can we build a more compressive list and more importantly, one that includes ALL the storage controllers found on the VSAN HCL to help our customers?

It then hit me, why not build on top of the effort Duncan has started and create a compressive list that includes all storage controllers found within the VSAN HCL and their corresponding queue depth? For this effort, I decided to take a slightly different approach on how I gather the information. Right now, a user is asked to must manually run through a series of commands in ESXTOP and then report back the vendor, make and the queue depth of a particular storage controller that may or may not be on the VSAN HCL. My goal was to make the process as simple as possible by automating the data collection but also adding some intelligence into the script which you will see as you read further.

If you currently look at the VSAN HCL for storage controllers (as of 07/18/14), there are currently 73 supported storage controllers:

Vendor Controllers
Cisco 1
Dell 4
Fujitsu 6
HP 4
IBM 6
Intel 16
LSI 33
SuperMicro 3

Instead of asking a user to identify the proper storage controller, the make/model and the queue depth to submit, I have instead created a very simple python script that runs inside the ESXi Shell (this information is not available in the API) to help collect this information. The interesting thing about the script is not the collection itself as mentioned, but how it performs the collection. I have embedded the entire list of supported storage controllers found in the VSAN HCL and as the script scans through the storage controllers within an ESXi host, it will compare that to the list of supported controllers. If a supported controller is found, it will then display some basic information about the storage controller along with the current supported queue depth. The nice thing about this list if completed, is that when selecting a particular storage controller in the VSAN HCL, you can easily map that same device to the VSAN storage controller queue depth list and have confidence it is the same device!

To use the script, follow these 3 simple instructions:

Step 1 - Download the script here: find_vsan_storage_ctrl_queue_depth.py

Disclaimer: Please excuse my poor Python script, as a Python beginner,  I am sure it can be better written and open to any fixes/suggestions

Step 2 - SCP it to your ESXi host and make sure you set the execute permission on the script before running (chmod +x find_vsan_storage_ctrl_queue_depth.py).

Here is an example of the script running on an ESXi host with a supported VSAN storage controller:

vsan-storage-controller-queue-depth
As you can see from the screenshot above, this is an Intel controller which supports a queue depth of 600.

Step 3 - Submit the results to the "Community" VSAN Storage Controller Queue Depth List which is hosted on Google Docs and is available for everyone to contribute

The easiest way to map the output to the Google document is to find the "Identifier" ID which is actually made up of the Vendor ID (VID), Device ID (DID), Sub-Vendor ID (SVID), and Sub-Device ID (SDID) within the Google document. Once you have found the match on the document and if no one has submitted the queue depth, go ahead and edit the document with the queue depth from the script.

For those of you who would like to contribute non-supported VSAN storage controllers, there is a variable in the script called show_non_vsan_hcl_ctr that can be toggled from False to True and this will provide a much longer list of controllers and their queue depth.

In addition to the assistance from the community, I also hope to see some of the storage controller vendors participate in this effort to help build a complete list of supported queue depth for every storage controller found on the VSAN HCL. I think this will benefit everyone and I look forward to seeing the collaboration from the community! Lets see how fast we can complete the list, I have faith in our powerful community!

Extending VSAN capabilities in the vSphere Web Client using vCO

by // Leave a Comment

One of my favorite features of vCenter Orchestrator is how easy it is to extend an existing vCO workflow and making it available directly in the vSphere Web Client. I think this is still not a very well known feature of vCO, but once you realize the capability of this feature, you will see how powerful it is to be able to provide context aware workflows in the vSphere Web Client. Another thing to be aware of is that vCO also provides full access to the underlying vSphere API, this means you can easily expose new functionality that may not exists in the vSphere Web Client.

A good example of this is a recent workflow that I created to extend some additional VSAN information directly in the vSphere Web Client. I wanted to be able to easily view the number of VSAN components for each of my ESXi hosts. Since this information is available through the vSphere API which I wrote about here, I was able to create a vCO Workflow which exposed this information and then make it available in the vSphere Web Client.

To get started, you will need to download the workflow and an updated vCenter vCO Plugin as it contains a fix for leveraging the VSAN APIs:

In the example below, I am using the vCO Appliance but the steps are very similar if you are using the Windows version.

Step 1 - Upload the o11nplugin-vsphere.dar.zip to your vCO Appliance

Step 2 - Unzip the contents by running the following command:

unzip o11nplugin-vsphere.dar.zip

Step 3 - Run the following command to set the appropriate ownership and permissions:

chmod 644 o11nplugin-vsphere.dar
chown vco:vco o11nplugin-vsphere.dar

Step 4 - Backup the original vCenter vCO Plugin by running the following command:

mv /usr/lib/vco/app-server/plugins/o11nplugin-vsphere.dar /usr/lib/vco/app-server/plugins/o11nplugin-vsphere.dar.bak

Step 5 - Copy the new plugin to the plugins directory by running the following command:

mv o11nplugin-vsphere.dar /usr/lib/vco/app-server/plugins

Step 6 - Restart the vCO Service to load the new plugin

/etc/init.d/vco-server restart

Once the vCO Server is available, you can login to the vCO Client and import the new VSAN workflow. To make the workflow available in the vSphere Web Client, you will need to login to the vSphere Web Client using an account that has access to the vCO Server and the instructions below.

Step 1 - Click on the vCO icon on the home page and then select Manage and Context Actions

Step 2 - Click on the green arrow to add a new worfklow

Step 3 - Browse for the VSAN workflow and then click on the Add button and associate the workflow with a vSphere Cluster object as seen in the screenshot below:

vsan-vco-plugin-0
Once the context workflow has been added, you are now ready to run the new VSAN workflow! Right click on a VSAN enabled vSphere Cluster and under the All vCenter Orchestrator Actions, you should see our workflow:

vsan-vco-plugin-1
Go ahead and run the workflow and once it completes, you can view the results by clicking on the workflow name in the Recent Tasks:

vsan-vco-plugin-2
Under the Parameters section, we can see our input and output variables. In this workflow, I have created a String output called "count" which contains the name of each ESXi host in the VSAN Cluster along with the number of corresponding components.

As you can see, you can easily enhance the functionality of the vSphere Web Client by simply extending it with either out of the box or custom vCO Workflows that you have created. Happy workflowing!