WilliamLam.com

A kitten is also harmed, every time you edit a VMX file by hand

03.25.2014 by William Lam // 4 Comments

Following up on a recent article that I wrote a kitten dies every time you query the VCDB, I wanted to raise another topic that could also bring harm to kittens as well as making me cringe (JK on the harming of kittens). The topic, as you might have guessed is about editing a Virtual Machine's VMX configuration file by hand which comes up some what frequently from customers. There are many valid reasons on why a customer would need to perform such an operation such as enabling Change Block Tracking for efficient Virtual Machine backups to the hardening of a Virtual Machine based on the vSphere Security Hardening Guideline, the list just goes on and on. Outside of extreme troubleshooting scenarios (debatable unless directed by VMware GSS), you should be leveraging the vSphere API to automate these changes and instead of manually editing the VMX file by hand.

Disclaimer: No kittens were harmed during the writing of this article

There are several disadvantages when editing a VMX file by hand:

Very error prone, you might get it right for the first VM but what about 500 VMs with couple of settings each?
VMX editing requires the VM to be powered off
Requires ESXi host to have SSH enabled unless you plan to manually download/re-upload the edited VMX file
Changes to VMX file must also be reloaded for changes to take affect

I think there are two main reasons customers are manually editing VMX files by hand. First, it might seem to be the quickest way but looking at the list of disadvantages, you will soon realize it really is not when you need to scale this change beyond a single Virtual machine. The second reason I think is the perceived intuitiveness of the change. I mean, all configuration changes made to a Virtual Machine whether it is using the vSphere Web/C# Client or vSphere API will need to eventually persist onto disk and those changes is what is saved into Virtual Machine's VMX configuration file. However, that is not the only way nor would I say the "correct" way of modifying your Virtual Machines when it comes to these and other settings. As I have already mentioned, you can already use the vSphere Web/C# Client to make these changes which in turn leverage the vSphere API. So why would it be any different if you were to perform these operations outside of a UI?

In fact, the vSphere API abstracts away the complexity of needing to edit the VMX file and provides nice programmatic/scripting interfaces to perform some of these Virtual Machine configurations. Taking our two existing examples, Change Block Tracking is actually exposed through a simple boolean property called changeTrackingEnabled and you can just call the ReconfigVM_Task() method to enable this functionality to ensure that you have efficient Virtual Machine backups. You do not need to manually tweak each Virtual Disk since that will automatically be handled by vSphere once this has been configured.

The second example is around hardening your Virtual Machines based on the vSphere Security Hardening Guide. Luckily, to implement those settings among other Virtual Machine Advanced Settings, it just simply an array property called extraConfig that accepts a list of key/value pairs. In fact, you can easily automate the configuration of one or dozen parameters using the scripts found in this blog article called Accessing Virtual machine Advanced Settings which I wrote a couple years back. If you want to automatically harden your Virtual Machines based on a set of pre-defined settings, you can also take a look at my Automate the hardening of your Virtual Machine VMX Configurations article.

The point that I am trying to make here is that though these changes may eventually end up in the Virtual Machines VMX configuration file, there are better, easier and more efficient ways of making these changes through the use of the vSphere API and that is one of the many benefits of using an API. If you have gotten this far in the article, then I would like to conclude with some of the benefits I mentioned earlier that you will not find if you were to manually edit a VMX file by hand. The first is that the Virtual Machine Advanced Settings can be applied while the Virtual Machine is up and running, yes the changes will not go into effect, but you can at least stage the changes. If really can not take down time of the Virtual Machine to completely power off and power on for the changes to reload, another trick that I learned from a friend in the VMware Community is to vMotion the Virtual Machine from one host to another. By doing a vMotion, a new VMX will be created on the destination host and the Virtual Machine Advanced Settings will then become active as a new VMX process is spawned. This is not something you can do if you manually edited the VMX file by hand 🙂

Quick Tip - Don't always assume your local HDs will be claimed correctly

03.21.2014 by William Lam // Leave a Comment

For whatever reason I woke up super early this morning around 4am PST and since I could not go back to sleep, I figure I might as well finish re-building one of my lab environments. I was trying to create a VSAN Disk Group using ESXCLI, but ran into the following error: Stamping non-local disks requires a pre-created vsan cluster If I run the following ESXCLI command, we can clearly see the Hitachi disks that I have are not being recognized as a local device:

esxcli storage core device list

I already know that some disks may not be recognized as a local device which VSAN requires and this is nothing new, in fact Duncan Epping even shows you how to get around this in this article here. Seeing that it was so early in the US, I figure Duncan was probably awake and wanted to run a few things by him. It was while he was looking around in the system did he notice a stranger issue.

It turns out my local Hitachi disks were being claimed by the VMW_SATP_DEFAULT_AA (ALUA) SATP plugin which was kind of strange as I would have expected to be claimed by VMW_SATP_LOCAL instead. I decided to take a look at the SATP claim rules to see why this was the case by running the following ESXCLI command:

esxcli storage nmp satp rule list

It turns out, for identifying Hitachi storage devices, the SATP rules is quite generic and keys off of the vendor name only and hence the assignment of the ALUA SATP plugin is choosen.

VMW_SATP_DEFAULT_AA HITACHI system inq_data[128]={0x44 0x46 0x30 0x30} VMW_PSP_RR
VMW_SATP_DEFAULT_AA HITACHI system

In my opinion this can be problematic as the plugin can potentially cause strange IO or pathing behaviors as it expects one thing but is getting something completely different. I suspect this plugin was probably meant for an Hitachi storage array like an HDS instead of a local device. Once I was able to narrow this down, I just needed to create a new rule that would override the system defaults.

To verify this, I created a new rule based on the vendor name just for testing purposes which will have priority over the default system rules. When creating custom SATP rules, you can filter on a variety of attributes such as the model, transport, controller, target, adapter, etc. Depending on your use case, you may want to be generic or quite specific. The following ESXCLI command will create the new rule and assign it the VMW_SATP_LOCAL plugin:

esxcli storage nmp satp rule add -V HITACHI -P VMW_PSP_FIXED -s VMW_SATP_LOCAL

Once that rule has been added, I can now finish my original task which is to mark my Hitachi drives as "local" and reclaim these devices by running the following ESXCLI commands:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -o "enable_local" -d [DEVICE]
esxcli storage core claiming reclaim -d [DEVICE]

If we now take a look at our device, we can see that are rule was activated and if we perform the following ESXCLI command, we can now see the device is "local":

esxcli storage core device list

The lesson here, do not assume your local devices and potentially even remote devices will be claimed correctly. There maybe vendor best practices that differ from the out out box rules and you should always do your due diligence to either verify yourself or work with your vendors to confirm the configurations.

Quick Tip - Increasing capacity on a Nested VSAN Datastore

03.21.2014 by William Lam // 2 Comments

The other day I needed to increase the capacity on one of my Nested VSAN Datastores as one of our users required a larger VSAN datastore than it was initially configured for. I was expecting to be able to just increase the size of the underlying VMDKs like I would for a traditional Nested ESXi environment and rescan in ESXi to pick up the new capacity without any downtime. It turns out, this is was not exactly the case for a Nested VSAN environment.

Disclaimer: Nested Virtualization is not officially supported by VMware

When you first setup VSAN, regardless of how the disks were claimed, VSAN will consume the entire device (SSD or MD). The capacity that VSAN initially detects will then be used to create the necessary partition as part of the VSAN Disk Group creation. VSAN assumes that the capacity for the underlying devices would never change as in the "real" world, disks do not auto-magically get larger 🙂 and this is a valid assumption. In a Nested ESXi environment however, it can auto-magically get larger but VSAN was not built for this use case. What ends up happening is that the underlying devices can be "hot-extended" but the existing VSAN Disk Group can not detect this new capacity.

Having said that, there are two ways you can increase your VSAN datastore:

Option 1 - If you wish to preserve your VSAN Datastore, you can hot-add additional VMDK(s) to your existing VSAN Disk Group or if it is full, you can create a new disk group and add additional VMDK(s). This will modify your setup slightly if you wanted a particular set of disk groups but will allow you to preserve your data.

Option 2 - The latter option requires the deletion and re-creation of the VSAN Datastore which is not ideal if you already have data on it. You will need to increase the capacity of the underlying VMDKs and then re-create your VSAN Datastore, but this way you can keep the existing number of disks and disk groups you initially created your Nested ESXi environment with.

In my scenario, I could not destroy the VSAN Datastore as I had someone using it and so I opted for option #1. Here is what my configuration looked like before which was a single VSAN Disk Group with 1xSSD and 1xMD:

I then added an additional 10GB VMDK to each of my Nested ESXi hosts and issue a rescan so the ESXi host would pickup the new device:

In just a few seconds, I can see my new storage device. I can now head over to the VSAN management page which is located at the vSphere Cluster and once I refresh, I can see that VSAN has automatically added the new "MD" into the existing disk group and my storage has automatically expanded!