WilliamLam.com

VM Storage Policy APIs aka Storage Profile APIs will be available in vSphere 5.5

by // 17 Comments

A frequently requested feature from customers and partners have been around the Storage Profile APIs and with the upcoming vSphere 5.5 release, it will now be possible to automate the management and consumption of Storage Profiles. In vSphere 5.5, Storage Profiles has been renamed to VM Storage Policy and they have been enhanced from the previous version of Storage Profile. VM Storage Policy introduces new concept of a rule set also known as a sub-profile or sub-policy from an API perspective.

A VM Storage Policy can contain multiple rule sets which describes a requirement for a virtual machine storage resource. Each rule can either be an underlying storage capability or a user defined vSphere Tag.

One important thing to note about the VM Storage Policy API (SOAP API), is that it is exposed as a separate API endpoint (similar to how the SMS API is exposed) on vCenter Server and it will not be accessible through the normal vSphere API. To consume this API, you will need to connect to the PBM (Policy Based Management) Server which requires an authenticated vCenter Server session. A great way to learn and explore the new SPBM API is to check out the SPBM MOB.

Here is the high level workflow for connecting to the PBM Server:

  1. Login to vCenter Server
  2. Extract the session cookie
  3. Add vCenter Server session cookie & connect to PBM Server

Once connected to the PBM Server, you will have access to PBM ServiceInstance with following three managed objects:

    • ProfileProfileManager (not a typo, repeat of Profile for some reason)
    • PlacementSolver
    • ComplianceManager

As mentioned earlier, a VM Storage Policy can be made up of several rule sets and each rule set contains a property rule. Here is the specification for what the VM Storage Policy looks like from an API perspective:

For managing and creating VM Storage Policies, you will need use the new VM Storage Policy API and for consuming and assigning VM Storage Policies to a virtual machine, you will need to use the vSphere API. When provisioning or cloning a virtual machine, there is a new profile property that denotes the MoRef ID for a particular VM Storage Policy.

As part of the VM Storage Policy API, there will be a Java SDK that includes a programming guide that goes over the VM Storage Policy API in greater detail as well as several sample programs exercising the various API methods. Since the VM Storage Policy API is a SOAP API similar to the vSphere API, the WSDL will also be available if you wish to generate your own language binding to the API.

Here is a screenshot of the available sample programs leveraging the new VM Storage Policy API:

Here is an example of one of the sample programs which lists all the VM Storage Policies for a given vCenter Server:

run.bat ListProfiles --vcurl https://[VC-IP]/sdk/vimService --ssourl https://[VC-IP]:7444/ims/STSService --spbmurl https://[VC-IP]/pbm --username *protected email* --password vmware --ignorecert

In the screenshot, you will see four VM Storage Policy being shown, one which I had created earlier and there others which are VM Storage Policies created by VSAN. You will notice that you will need three pieces of information when connecting: vCenter Server endpoint, SSO Server endpoint and PBM Server endpoint. You can find more details by referring to the VM Storage Policy Programming Guide and VM Storage Policy API reference.

My top 5 favorite enhancements to the new vSphere Web Client 5.5

by // 10 Comments

I have been using the vSphere Web Client more and more lately and though transitioning away from the familiar legacy vSphere C# Client is not the easiest thing to do or always possible for every single operation, there are definitely some nice benefits when using the vSphere Web Client. With the upcoming vSphere 5.5 release, there is even more cool new features in the vSphere Web Client!

Here are my top 5 favorite enhancements in the new vSphere Web Client 5.5 in no particular order. For a complete list of new features in the vSphere Web Client, I recommend you take a look at the What's New in vSphere 5.5 whitepaper.

Mac OS X Support for vSphere Web Client

Being a web application, the vSphere Web Client has always worked on a Mac OS X system, however you may have noticed a couple of things did not work such as OVA/OVF upload, remote device management such as mounting an ISO/Floppy and the biggest one of all is virtual machine console access! This has been one of the most requested feature that I can think of and I am personally excited to see this finally come to fruition. In addition to to the native VM console support (HTML5/WebSockets), there is also now a vSphere Client Integration package for Mac OS X that provides both OVA/OVF upload and remote device management support. This alone is enough for me to upgrade my vCenter Server to 5.5 to get these new feature!

Recently Visited & Created Objects

The recently visited objects is a pretty handy feature that came in vSphere 5.1 which allows you to see what objects you have been recently working with. However, this feature may not have been very well known due to its tiny icon. I am glad to see this feature get its own icon and is now located at the top of the vSphere Inventory Navigator between the navigator and pin icon. In addition to this change, it also now includes a list of the recently created vSphere objects which can come in handy when you are doing something new for the first time and would like a quick way to view the sequence of objects created.

vSphere Inventory Navigator History + Back/Forward Navigation

I am pretty sure our vSphere UE engineers have a more elegant name for this awesome feature, but  you can now view the history as you traverse through the vSphere Inventory Navigator and navigate both backwards as well as forward (which is new in vSphere 5.5). To view your current history, you simply just right click on the navigator bar at the top and you will get a drop down list of your history. You can go move forwards or backwards through your history which is a great if you are still getting familiar with the vSphere Web Client and forgot how you got to a particular object.

Deploy vCenter Operations from vSphere Web Client

I thought this was a pretty cool enhancement by allowing you to deploy vCenter Operations Management from within the vSphere Web Client. You will notice a new vC Ops icon on the main dashboard and on the Getting Started page, there is a link at the bottom that will allow you to deploy the vC Ops appliance by first logging into your MyVMware account. I wonder if we will are going to start doing this for other VMware solutions and just making it easier to deploy the latest version without having to first download it onto your local system.

Configure Auto-Refresh & Disable Inventory Navigator Animation

A common piece of feedback that I have heard regarding the vSphere Web Client experience is that it does not automatically refresh the screen. This is a change from the vSphere C# Client where it will automatically refresh the inventory, but of course there is some overhead associated with this refresh as it needs to pull the latest data from the vCenter Server. However, with the latest vSphere Web Client 5.5, you can now enable auto-refresh using an advanced configuration (by default it is disabled). Before you enable this, do note that this can alter the performance of your environment and be aware this will prevent the session from automatically logging out if you have configured an idle session timeout.

UPDATE: (03/11/16) - In vSphere 6.0, the path to webclient.properties has changed to /etc/vmware/vsphere-client/webclient.properties

To enable auto-refresh, you will need to locate the following configuration file /var/lib/vmware/vsphere-client/webclient.properties on the VCSA (there should also be an equivalent on Windows version of vSphere Web Client Server)

By default the auto-refresh is disabled, to enable it, you will need to un-comment the following configuration parameter and set the number of seconds to auto-refresh:

refresh.rate = # of seconds

Another feature that I found interesting that can also be controlled in this configuration file is the sliding animation shown when clicking on the vSphere Inventory Navigator. This I assume is to reduce the amount of resources loading the animation, unless the animation was bothering some folks?

By default this is now disabled in vSphere 5.5 and if you wish to see that animation (default in vSphere 5.1), you can re-enable by un-commenting the following configuration parameter:

navigator.disableAnimation = true or false

There are few other settings that you can control in the webclient.properties, you can take a look at the file for more details.

There are definitely a few more new features in the vSphere Web Client 5.5 that I have not mention, but these were my my top five favorite enhancements. One more thing I would like to also mention is that vSphere Web Client in vSphere 5.5 release definitely feels much snappier than previous releases and this has made for a much better user experience in my opinion. When you get your hands on the new vSphere Web Client, what will be your favorite new feature?

How to bootstrap vCenter Server onto a single VSAN node Part 2?

by // 47 Comments

In this article, I will provide a step by step walk through on how to setup and configure single VSAN node that will allow you to deploy a vCenter Server onto a VSAN datastore. This initial "bootstrapping" can help when initially building out your VSAN cluster and can come in handy for greenfield deployments and potentially for brownfield deployments as well. Before getting started, make sure you have taken a look at How to bootstrap vCenter Server onto a single VSAN node Part 1.

Environment:

  • 3 physical host
  • Each host as a small iSCSI boot LUN for ESXi installation (this can be another local disk or USB/SD card)
  • Each host has single SSD and SATA disk (minimum)

Step 1 -  Install ESXi 5.5 onto your physical hosts, we technically only need one host to begin the process but you will probably want to have two additional hosts ready unless you do not care about your vCenter Server being able to recover if there is any hardware issues.

Step 2 - You will need to modify the default VSAN storage policy on the ESXi host in which you plan to provision your vCenter Server. It looks like this behavior changed during the VSAN beta and when VSAN was GA'ed yesterday with vSphere 5.5 Update 1. You will need to run the following two ESXCLI commands to enable "force provisioning":

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"

You can confirm you have the correct VSAN default policy by running the following ESXCLI command:

~ # esxcli vsan policy getdefault
Policy Class  Policy Value
------------  --------------------------------------------------------
cluster       (("hostFailuresToTolerate" i1))
vdisk         (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmnamespace   (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmswap        (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))

We start off with our first ESXi host and as you can see from the screenshot below, we do not have additional datastores that we can use to provision our vCenter Server.

Step 3 - You will need to identify the disks that you will be using on the first ESXi host to contribute to the VSAN datastore. You can do so by running the following ESXCLI command:

esxcli storage core device list

To get specific details on a particular device such as identifying whether it is an SSD or regular HDD, you can specify the -d option and the device name.

Once you have identified the disks you will be using, make a note of the the disks names as they will be needed in the upcoming steps. As mentioned in my environment, I only have a single SSD and single HDD and their respective device names are naa.50026b72270126ff & naa.5000c500331bca77

Step 4 - Before we can create our VSAN datastore, we need to first create a VSAN cluster. One of the parameters that is needed when going through this "bootstrapping" method without a vCenter Server is a unique UUID to identify the VSAN cluster. The UUID is in the format of "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn" where n is a hexidecimal value. You can easily generate one within the ESXi Shell by leveraging the following Python snippet

python -c 'import uuid; print(str(uuid.uuid4()));'

Step 5 - To create a VSAN cluster, we will use the following ESXCLI command and specify the UUID from the previous step for the -u option:

esxcli vsan cluster join -u UUID

UPDATE (02/11/15) - In vSphere 6, you no longer have to perform step 4 to generate a UUID. There is now a new ESXCLI command which will automatically create a VSAN Cluster and generate a UUID automatically by running the following command:

esxcli vsan cluster new

Once the VSAN cluster has been created, you can retrieve information about the VSAN cluster by running the following ESXCLI command:

esxcli vsan cluster get

Step 6 - Next we need to add the disks from our ESXi host to create our single node VSAN datsatore. To do so, we will need the disk device names from our earlier step for both SSD and HDDs and run the following ESXCLI command:

esxcli vsan storage add -s SSD-DISK-ID -d HDD-DISK-ID

The -d option specifies regular HDD disks and the -s option specifies an SSD disk. If you have more than one HDD disk, you will need to specify multiple -d entries. You can also take a look at the disks being contributed to the VSAN datatore by running the following ESXCLI command:

esxcli vsan storage list

Step 7 - To save us one additional step, you can also enable the VSAN traffic type on the first ESXi host using ESXCLI and you can also do this for the other two hosts in advance. This step does not necessary have to be done now as it can be done later when the vCenter Server is available and using the vSphere Web Client. You will need to either create or select an existing VMkernel interface to enable the VSAN traffic type and you can do so by running the following ESXCLI command:

esxcli vsan network ipv4 add -i VMK-INT

At this point, you now have a valid VSAN datastore for your single ESXi host! You can verify this by logging into the vSphere C# Client and you should see the VSAN datastore mounted to your ESXi host.

At this point, you are now ready to deploy your vCenter Server 5.5 onto the VSAN datastore. The next series of steps outline the deployment of the VCSA for completeness of the article.

Step 8 - Deploy the VCSA 5.5 OVA/OVF onto the VSAN datastore and power on the VM.

UPDATE: You skip Steps 9-11 by leveraging ovftool 4.0 to inject the required OVF properties when deploying the VCSA, take a look at this article for more details.

Step 9 - Since you can not configure the OVF properties for the VCSA, you will notice that networking is not configured (unless you happen to have DHCP on the network). If you are like most Enterprise customers, you will not have DHCP running in your environment and you will need to configure a static IP.

Step 10 - Login to the VCSA console and we will use the following VAMI CLI /opt/vmware/share/vami/vami_set_network to configure the IP Address for the VCSA. Here is an example of what that command would look like:

/opt/vmware/share/vami/vami_set_network eth0 STATICV4 172.24.68.14 255.255.255.0 172.24.68.1

For more details on the syntax, you can refer to this blog article here. At this point, you should be able to ping your VCSA and verify connectivity.

Step 11 (Optional) - In addition to IP connectivity, you may also want to configure your DNS Server and DNS search domain before configure the VCSA application. You can also do this by using the following VAMI CLI /opt/vmware/share/vami/vami_set_dns and for search domain, you would need to add the entry to /etc/resolve.conf

Step 12 - You now are ready to configure the VCSA. Open a browser and connect to https://[VCSA-IP]:5480 and proceed through the VCSA setup wizard.

Step 13 - Once the VCSA has been configured, you can now login to the vSphere Web Client and create a Datacenter object and then a vSphere Cluster and enable VSAN. Make sure you also enter your VSAN beta license key under the "Manage" section of the vSphere Cluster before you can use VSAN.

Step 14 - Add all three of your ESXi hosts to the vSphere Cluster. If you recall earlier we had enabled the VSAN traffic type on our first ESXi host and if you did not run the command on the remainder ESXi hosts, you will need to do so using the vSphere Web Client under the "Networking" section of each ESXi host

Step 15 - Once all three ESXi hosts have been added to the vSphere Cluster, we should now see their local storage contributed to the VSAN datastore under the "General" tab

Step 16 (Optional) - If for whatever reason the disks do not get automatically claimed, you can click on "Disk Management" and manually claim them. If you selected "Automatic" mode when enabling VSAN, the disks on each ESXi host should automatically be handled by VSAN. However, they may not be claimed if the disks are being seen as "remote" versus "local" devices.

Step 17 - The final thing I would recommend is to configure the VCSA to automatically startup and shutdown when the ESXi host reboots. To do so, login to the ESXi host using the vSphere C# Client and click on "Virtual Machine Startup/Shutdown" under the Configuration tab.

So there you have it! You are now running the vCenter Server on top of the VSAN datastore without having to initially setup a local VMFS or rely on an external NFS volume to deploy your vCenter Server and build up to the full VSAN cluster. By leveraging this bootstrap method, you can easily standup a fully self contained storage and compute cluster which is ideal for an SMB or ROBO environment. The best part of about this setup is that the VCSA will use the default VSAN storage policy which is to tolerate at least one failure and as you add your 2nd and 3rd ESXi host, you will automatically have resiliency for the VCSA.