vm storage policy

SPBM (Storage Policy Based Management) MOB in vSphere 5.5

11.27.2013 by William Lam // 8 Comments

A handy little tool that was useful for me which might come in handy for others too while working on my VSAN VM Storage Policy recovery article is the SPBM (Storage Policy Based Management) MOB which works similar to the vSphere MOB and FDM MOB for those of you who have used this interface before. The MOB stands for Managed Object Browser and simply put is an interface that allows you to browse the objects and properties of your vSphere environment by just using a web browser. You can also interact with the MOB by executing the same set of vSphere API methods as you would if you were to write a script or application which is useful for quickly getting a sense of what a certain property might look like or the output of an API method without writing a single line of code.

In vSphere 5.5, there is now an SPBM API which is available as a separate API endpoint on the vCenter Server. This new API allows you to manage the full lifecycle of a VM Storage Policy formally known as VM Storage Profiles from a programmatic standpoint which is very important when working with VSAN as everything is policy driven. For more more details about the new SPBM API, please take a look here. As mentioned earlier, one great way to learn about the API is by exploring the MOB and luckily the SPBM API includes one!

To access the SPBM MOB in vSphere 5.5, you will need to point your browser to your vCenter Server as that is where the endpoint is exposed using the following URL format:

http://[VC-IP-OR-HOSTNAME]:8190/mob

To access the SPBM MOB in vSphere 6.0, you will need to point your browser to your vCenter Server as that is where the endpoint is exposed using the following URL format:

http://[VC-IP-OR-HOSTNAME]/pbm/mob

You will be prompted for your vCenter Server credentials before the MOB will allow you to login. Once you are logged in you will be brought to the main service content of the SPBM endpoint similar to the vCenter Server service content and you can then click on content link to explore the various sub-managers that are available.

Note: You will be able to get more details on each of these sub-managers by taking a look at the VM Storage Policy Programming Guide and VM Storage Policy API reference guide.

To quickly show you around, I will provide a couple of examples using the ProfileManager and I am sure you can probably guess what type of functionality it provides :). The first method that we will take a look at is the PbmQueryProfile which will return the list of available VM Storage Policies that have been defined. You will need to set the resourceType property to "STORAGE" and remove the profileCategory and then click on "Invoke Method".

If you are using VSAN and you do not have any VM Storage Policies defined, there will still be two default VM Storage Policies that is automatically created when VSAN is enabled. What you will see are the internal identifiers for each of the VM Storage Policy and as you can see from the output I have 5 VM Storage Policies.

You will notice that the output does not contain the human readable display name for each VM Storage Policy, to retrieve that information we will need to use the PbmRetrieveContent which accepts a list of VM Storage Profile ID's and in return provide the human readable name as well as other properties such as the initial creation date and last modified date. Using the pre-canned input form, you can specify one or more VM Storage Profile IDs from the previous step and then click on "Invoke Method".

In my example, I specified two of my VM Storage Policies and I can see they map to the names "Aluminum" and "Copper" which is what I named them when I first created the policies.

From here on out, we will be using the VM Storage Policy ID as that is what is used to uniquely identify a VM Storage Policy and input for majority of the SPBM API methods. Now if we want to see what objects (VM Home directory or VMDKs) are associated with a particular VM Storage Policy we can use the PbmQueryAssociatedEntity method. You will need to provide the VM Storage Policy ID and remove the entityType and then click on the "Invoke Method".

As you can see from the output this a virtualMachine object type which tells us this VM Storage Policy is used for the VM Home. Lets go ahead and specify a VM Storage Policy that is used for a Virtual Machine's VMDK and see what that looks like.

We now see the object type is virtualDiskId and you can see the particular VMDK and the associated Virtual Machine by looking at the key which has the format of vm-mo-ref:vmdk-key. Now what if we wanted to perform the reverse look up, by providing only a Virtual Machine or VMDK as input? Well, we can easily do this lookup by using the PbmQueryAssociatedProfiles method. This API method requires you to specify three parameters: objectType, key and serverUuid (technically speaking the serverUuid can be left out).

From the above examples you will get an idea of what the expected input format is for either a Virtual Machine or VMDK query.

Here is an example of a Virtual Machine query:

Here is an example of a VMDK query:

Hopefully this quick introduction of the SPBM MOB will give you a good idea on how you can leverage this interface, especially if you plan on using the new SPBM API to automate and manage your VM Storage Policies.

Restoring VSAN VM Storage Policy without vCenter Part 2: Using vSphere API

11.25.2013 by William Lam // 3 Comments

Last week I demonstrated a manual method of recovering a VSAN VM Storage Policy when vCenter Server is no longer available by using a VSAN command-line utility found in the ESXi Shell called cmmds-tool. Though this approach works, it can also be quite tedious and error prone as you have to manually go through various configuration files and extract out the individual VSAN Object UUIDs. Luckily, one can automate the process outlined in the previous article by leveraging the vSphere API to connect directly to an ESXi host and access the VSAN Internal CMMDS system (Clustering Monitoring, Membership and Directory Services).

Disclaimer: This script is provided as a sample, please ensure it is properly tested before using it in a production environment.

I have created a sample vSphere SDK for Perl script called queryVSANVMStoragePolicyMapping.pl to demonstrate the vSphere API functionality. To use this script, you will need to ensure you have either the vCLI or vSphere SDK for Perl 5.5 installed on a system or you can use the vMA 5.5 appliance. You will also need to install an additional Perl module called JSON which is used in the script.

In my environment, I have a Virtual Machine called VSAN-VM-1 which has the following VM Storage Policies assigned to it.

Lets say our vCenter Server is now gone, how do we go about recovering the VM Storage Policy configurations to rebuild it on our new vCenter Server? With this script it is quite simple to recover the information by simply connecting to the ESXi host and specifying the name of the Virtual Machine.

Here is a sample execution of the script for my VSAN-VM-1:

From the output we can see that it automatically identifies the VSAN Object UUID for the VM Home directory as well as all VMDKs that are associated with that Virtual Machine. The script then uses that information to pass to the QueryCmmds API method which is part of the vsanInternalSystem manager to perform the query. The output is then returned as a JSON string which is parsed by the script to display the VM Storage Policy MoRef ID for each corresponding Virtual Machine component along with their configured VSAN policies.

The VM Home directory maps to our "Copper" policy which looks like the following in the vSphere Web Client:

The first VMDK maps to our "Platinum" policy which looks like the following in the vSphere Web Client:

The final VMDK maps to our "Aluminum" policy which looks like the following in the vSphere Web Client:

Restoring VSAN VM Storage Policies without vCenter Part 1: Using cmmds-tool

11.22.2013 by William Lam // 5 Comments

A scenario that I have been been looking into recently while testing VSAN in my lab is what happens when vCenter Server is no longer available and the impact that might have on your environment. We know that VSAN from a configuration perspective works very similiar to vSphere HA where vCenter Server is only required for the initial VSAN Cluster configuration. Once the ESXi hosts has been added to the VSAN Cluster, vCenter Server is no longer part of the picture from a functional perspective and the ESXi hosts will know how to communicate with each other within the VSAN Cluster. We can even build a single VSAN node to help bootstrap vCenter Server itself for greenfield deployments.

So what does that leave us with? Well, the Virtual Machines of course. The Virtual Machines will continue to run without any impact whether or not vCenter Server is available. VSAN will continue to govern and maintain compliance for the VM Storage Policies that have been assigned to each and every Virtual Machine. However, in the scenario where you can not restore vCenter Server which is primarily where the VM Storage Policies are stored and you need to build out a new environment, how do you go about restoring the VM Storage Policies?

Well it turns out that vCenter Server is not the only place where the VM Storage Policies are stored at. To ensure that VSAN can continue enforcing the policies that have been assigned to each Virtual Machine and their associated VMDKs, there is a copy of the VM Storage Policies that is distributed amongst all the ESXi hosts within the VSAN Cluster. In this first first article I will demonstrate how to recover the VM Storage Policies for a particular Virtual Machine running on an ESXi host where vCenter Server is no longer available using a utility located in the ESXi Shell called cmmds-tool. In part two of the article I will demonstrate the same recovery process but leveraging the vSphere API which will be more user friendly.

Disclaimer: The cmmds-tool is not meant for troubleshooting, you should only use under VMware GSS/Engineering supervision. If you choose to use it, do so at your own risk.

In the ESXi Shell, there is a nifty little VSAN utility called cmmds-tool which stands for Clustering Monitoring, Membership and Directory Services. This tool allows you to perform a variety of operations and queries against the VSAN nodes and their associated objects. One interesting command is the "find" operation which will allow us to lookup a specific VM Storage Policy, a bit more on this later.

Lets say we have a Virtual Machine called VSAN-VM-1 and it is associated with three VM Storage Policies called Copper, Aluminum and Platinum. We have one for the VM Home and one for each of the two VMDKs. Here is a screenshot of what that looks like in the vSphere Web Client:

Now lets say vCenter Server is some how lost or unrecoverable for whatever reason, but we still have access to the ESXi host and the running Virtual Machine. Lets go ahead and recover the VM Storage Policies so we can then rebuild a new vCenter Server and re-create the policies.

Step 1 - We need to first identify a couple of pieces of information. The first is going to be the UUID of the VM Home directory (VSAN uses with UUIDs for all its objects). Login to ESXi Shell of the ESXi host that is currently hosting the Virtual Machine and run the following command:

vim-cmd vmsvc/getallvms | grep [DISPPLAY-NAME-OF-YOUR-VM]

The VM Home directory UUID will be part of the Virtual Machine directory name which can be seen in the screenshot above highlighted in green. Make a note of that UUID as you will need it in a later step. You should also make a note of the Virtual Machine MoRef ID which is the first numeric value on the left hand side of the output. In this example, I have 1 as the MoRef ID

Step 2 - Next we need to identify the UUID for each of the VMDKs for that given Virtual Machine. To do so, we need to take a look at the descriptor file for each of the VMDKs in the Virtual Machine home directory. You can use vim-cmd vmsvc/get.filelayout [VM-MOREF-ID] to get the VMDK paths or you can change into the Virtual Machine directory and cat out the files. In my example I have the following two VMDK descriptor files:

/vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1.vmdk
/vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1_1.vmdk

You can just grep for the keyword "vsan" by using the following command (replacing the path of your VMDKs):

grep "vsan" /vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1.vmdk

From the output you will see vsan:// and UUID associated with each VMDK, please make a note of the UUID for each VMDK. We are now ready to query the VM Storage Policy configuration which will help us rebuild the policy in our new vCenter Server.

Step 3 - To look up the VM Home VM Storage Policy, run the following command and specify the UUID of the VM Home in Step 1:

cmmds-tool find -t POLICY -u 51108952-6e91-b30b-a5ab-005056ad9acf -f json

The VM Storage Policy configurations is stored in the "content" field and you will need to translate the properties back to the VSAN policy you have defined. As part of the output you will also see a property called spbmProfileId which is the unique identifier for VM Storage Policy which you can query if you are using the VM Storage Policy APIs that were introduced in vSphere 5.5.

Here is a table that will help you translate the keys to the apporopirate VSAN Policies:

VSAN Capability Description	VSAN Capability Key
Number of failures to tolerate	hostFailuresToTolerate
Number of disk stripes per object	stripeWidth
Force provisioning	forceProvisioning
Object space reservation	proportionalCapacity
Flash read cache reservation	cacheReservation

Step 4 - To lookup the VMDK VM Storage Policies, we will perform the same command and just replace the UUID with our VMDK UUIDs.

Once you have recorded the configurations for each of the VM Storage Policy, you can then head over to your new vCenter Server and re-create the VM Storage Policies and then re-associate the policy with the Virtual Machines.

As you can see the steps to recover a VSAN VM Storage Policy is not too difficult but can be a bit tedious. In the next article, we will simplify this by leveraging the vSphere API which has access to the same CMMDS system but make querying the VM Storage Policy super easy by only requiring the user to provide the name of the Virtual Machine.