WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

Restoring VSAN VM Storage Policies without vCenter Part 1: Using cmmds-tool

11.22.2013 by William Lam // 5 Comments

A scenario that I have been been looking into recently while testing VSAN in my lab is what happens when vCenter Server is no longer available and the impact that might have on your environment.  We know that VSAN from a configuration perspective works very similiar to vSphere HA where vCenter Server is only required for the initial VSAN Cluster configuration. Once the ESXi hosts has been added to the VSAN Cluster, vCenter Server is no longer part of the picture from a functional perspective and the ESXi hosts will know how to communicate with each other within the VSAN Cluster. We can even build a single VSAN node to help bootstrap vCenter Server itself for greenfield deployments.

So what does that leave us with? Well, the Virtual Machines of course. The Virtual Machines will continue to run without any impact whether or not vCenter Server is available. VSAN will continue to govern and maintain compliance for the VM Storage Policies that have been assigned to each and every Virtual Machine. However, in the scenario where you can not restore vCenter Server which is primarily where the VM Storage Policies are stored and you need to build out a new environment, how do you go about restoring the VM Storage Policies?

Well it turns out that vCenter Server is not the only place where the VM Storage Policies are stored at. To ensure that VSAN can continue enforcing the policies that have been assigned to each Virtual Machine and their associated VMDKs, there is a copy of the VM Storage Policies that is distributed amongst all the ESXi hosts within the VSAN Cluster. In this first first article I will demonstrate how to recover the VM Storage Policies for a particular Virtual Machine running on an ESXi host where vCenter Server is no longer available using a utility located in the ESXi Shell called cmmds-tool. In part two of the article I will demonstrate the same recovery process but leveraging the vSphere API which will be more user friendly.

Disclaimer: The cmmds-tool is not meant for troubleshooting, you should only use under VMware GSS/Engineering supervision. If you choose to use it, do so at your own risk.

In the ESXi Shell, there is a nifty little VSAN utility called cmmds-tool which stands for Clustering Monitoring, Membership and Directory Services. This tool allows you to perform a variety of operations and queries against the VSAN nodes and their associated objects. One interesting command is the "find" operation which will allow us to lookup a specific VM Storage Policy, a bit more on this later.

Lets say we have a Virtual Machine called VSAN-VM-1 and it is associated with three VM Storage Policies called Copper, Aluminum and Platinum. We have one for the VM Home and one for each of the two VMDKs. Here is a screenshot of what that looks like in the vSphere Web Client:

Now lets say vCenter Server is some how lost or unrecoverable for whatever reason, but we still have access to the ESXi host and the running Virtual Machine. Lets go ahead and recover the VM Storage Policies so we can then rebuild a new vCenter Server and re-create the policies.

Step 1 - We need to first identify a couple of pieces of information. The first is going to be the UUID of the VM Home directory (VSAN uses with UUIDs for all its objects). Login to ESXi Shell of the ESXi host that is currently hosting the Virtual Machine and run the following command:

vim-cmd vmsvc/getallvms | grep [DISPPLAY-NAME-OF-YOUR-VM]

The VM Home directory UUID will be part of the Virtual Machine directory name which can be seen in the screenshot above highlighted in green. Make a note of that UUID as you will need it in a later step. You should also make a note of the Virtual Machine MoRef ID which is the first numeric value on the left hand side of the output. In this example, I have 1 as the MoRef ID

Step 2 - Next we need to identify the UUID for each of the VMDKs for that given Virtual Machine. To do so, we need to take a look at the descriptor file for each of the VMDKs in the Virtual Machine home directory. You can use vim-cmd vmsvc/get.filelayout [VM-MOREF-ID] to get the VMDK paths or you can change into the Virtual Machine directory and cat out the files. In my example I have the following two VMDK descriptor files:

/vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1.vmdk
/vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1_1.vmdk

You can just grep for the keyword "vsan" by using the following command (replacing the path of your VMDKs):

grep "vsan" /vmfs/volumes/vsanDatastore/51108952-6e91-b30b-a5ab-005056ad9acf/VSAN-VM-1.vmdk

From the output you will see vsan:// and UUID associated with each VMDK, please make a note of the UUID for each VMDK. We are now ready to query the VM Storage Policy configuration which will help us rebuild the policy in our new vCenter Server.

Step 3 - To look up the VM Home VM Storage Policy, run the following command and specify the UUID of the VM Home in Step 1:

cmmds-tool find -t POLICY -u 51108952-6e91-b30b-a5ab-005056ad9acf -f json

The VM Storage Policy configurations is stored in the "content" field and you will need to translate the properties back to the VSAN policy you have defined. As part of the output you will also see a property called spbmProfileId which is the unique identifier for VM Storage Policy which you can query if you are using the VM Storage Policy APIs that were introduced in vSphere 5.5.

Here is a table that will help you translate the keys to the apporopirate VSAN Policies:

VSAN Capability Description VSAN Capability Key
Number of failures to tolerate hostFailuresToTolerate
Number of disk stripes per object stripeWidth
Force provisioning forceProvisioning
Object space reservation proportionalCapacity
Flash read cache reservation cacheReservation

Step 4 - To lookup the VMDK VM Storage Policies, we will perform the same command and just replace the UUID with our VMDK UUIDs.

Once you have recorded the configurations for each of the VM Storage Policy, you can then head over to your new vCenter Server and re-create the VM Storage Policies and then re-associate the policy with the Virtual Machines.

As you can see the steps to recover a VSAN VM Storage Policy is not too difficult but can be a bit tedious. In the next article, we will simplify this by leveraging the vSphere API which has access to the same CMMDS system but make querying the VM Storage Policy super easy by only requiring the user to provide the name of the Virtual Machine.

Categories // VSAN, vSphere 5.5 Tags // cmmds-tool, ESXi 5.5, Virtual SAN, vm storage policy, vm storage profile, VSAN, vSphere 5.5

VM Storage Policy APIs aka Storage Profile APIs will be available in vSphere 5.5

09.12.2013 by William Lam // 17 Comments

A frequently requested feature from customers and partners have been around the Storage Profile APIs and with the upcoming vSphere 5.5 release, it will now be possible to automate the management and consumption of Storage Profiles. In vSphere 5.5, Storage Profiles has been renamed to VM Storage Policy and they have been enhanced from the previous version of Storage Profile. VM Storage Policy introduces new concept of a rule set also known as a sub-profile or sub-policy from an API perspective.

A VM Storage Policy can contain multiple rule sets which describes a requirement for a virtual machine storage resource. Each rule can either be an underlying storage capability or a user defined vSphere Tag.

One important thing to note about the VM Storage Policy API (SOAP API), is that it is exposed as a separate API endpoint (similar to how the SMS API is exposed) on vCenter Server and it will not be accessible through the normal vSphere API. To consume this API, you will need to connect to the PBM (Policy Based Management) Server which requires an authenticated vCenter Server session. A great way to learn and explore the new SPBM API is to check out the SPBM MOB.

Here is the high level workflow for connecting to the PBM Server:

  1. Login to vCenter Server
  2. Extract the session cookie
  3. Add vCenter Server session cookie & connect to PBM Server

Once connected to the PBM Server, you will have access to PBM ServiceInstance with following three managed objects:

    • ProfileProfileManager (not a typo, repeat of Profile for some reason)
    • PlacementSolver
    • ComplianceManager

As mentioned earlier, a VM Storage Policy can be made up of several rule sets and each rule set contains a property rule. Here is the specification for what the VM Storage Policy looks like from an API perspective:

For managing and creating VM Storage Policies, you will need use the new VM Storage Policy API and for consuming and assigning VM Storage Policies to a virtual machine, you will need to use the vSphere API. When provisioning or cloning a virtual machine, there is a new profile property that denotes the MoRef ID for a particular VM Storage Policy.

As part of the VM Storage Policy API, there will be a Java SDK that includes a programming guide that goes over the VM Storage Policy API in greater detail as well as several sample programs exercising the various API methods. Since the VM Storage Policy API is a SOAP API similar to the vSphere API, the WSDL will also be available if you wish to generate your own language binding to the API.

Here is a screenshot of the available sample programs leveraging the new VM Storage Policy API:

Here is an example of one of the sample programs which lists all the VM Storage Policies for a given vCenter Server:

run.bat ListProfiles --vcurl https://[VC-IP]/sdk/vimService --ssourl https://[VC-IP]:7444/ims/STSService --spbmurl https://[VC-IP]/pbm --username *protected email* --password vmware --ignorecert

In the screenshot, you will see four VM Storage Policy being shown, one which I had created earlier and there others which are VM Storage Policies created by VSAN. You will notice that you will need three pieces of information when connecting: vCenter Server endpoint, SSO Server endpoint and PBM Server endpoint. You can find more details by referring to the VM Storage Policy Programming Guide and VM Storage Policy API reference.

Categories // vSphere 5.5 Tags // spbm, vm storage policy, vm storage profile, vSphere 5.5

Automate Enabling VM Storage Profiles Capability in vSphere

08.06.2013 by William Lam // 1 Comment

I recently had to rebuild one of my lab environments that consisted of a vCenter Server and several ESXi hosts and one of the capabilities I required was VM Storage Profiles. In a brand new vSphere environment, VM Storage Profiles is disabled by default, presumably due to licensing as this feature is not available in every SKU. To enable VM Storage Profiles, you will need to connect to your vCenter Server using either the vSphere Web Client or the legacy vSphere C# Client as seen in the screenshot below.

Another way of enabling the VM Storage Profiles feature is through an automated fashion using the vSphere APIs where this functionality is exposed.

Note: To manage and consume VM Storage Profiles, you will still need to use the vSphere Web Client or vSphere C# Client as the APIs for this functionality is not yet available.

VM Storage Profiles is enabled on a per vSphere Cluster basis and there is a boolean property called spbmEnabled which determines whether this feature is enabled or disabled. To update this property, you will need to use the ReconfigureComputeResource_Task method.

Here is are two examples of implementing the API both in a PowerCLI script as well as a vSphere SDK for Perl script:

PowerCLI:

In my lab environment, I have a vSphere Cluster called "Cluster" and you can use PowerCLI to check whether VM Storage Profiles is enabled or not by running this command (replace Cluster with the name of your vSphere Cluster):

(Get-Cluster -Name Cluster).extensionData.ConfigurationEx.SpbmEnabled

To enable VM Storage Profiles, you can then call the Enable-StorageProfile function along with the name of the vSphere Cluster which is just called "Cluster":

 

vSphere SDK for Perl:

To enable VM Storage Profiles using the vSphere SDK for Perl, I wrote a quick script called storageProfileMgmt.pl which allows you to query whether a vSphere Cluster has VM Storage Profiles as well as enabling and disabling the feature. Here is a screenshot demonstrating how the script works:

Categories // Uncategorized Tags // PowerCLI, spbm, Storage Policy Based Management, vm storage profile, vsphere sdk for perl

  • « Previous Page
  • 1
  • 2
  • 3
  • 4

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025