WilliamLam.com

How to backup VMs in ESX onto Google Storage

by // 4 Comments

Over the weekend I received an email invite for Google Storage for Developers, which I had applied for a month or two back. I did not think I would get into the limited beta and at the time I was just curious about the offering from Google. Back in January I wrote an article How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud which uses Amazon's S3 storage offering and their s3cmd to upload a VM into the cloud. I spent a little time over the weekend playing with Google's storage offering and was able to install Google's python gsutil utility on to an ESX 4.1 host and backup a VM to Google Storage.

Before you begin, you will to have access to Google Storage for Developers and you will need to download the gsutil and transfer the tarball to your ESX host. The test was performed on an ESX 4.1 host, but I do not see why it will not work on ESX 4.0.

1. Extract the contents of gsutil.tar.gz

tar -zxvf gsutil.tar.gz

2. Edit .bashrc

vi .bashrc

Add the following two lines:

By default, gsutil requires you to be running python 2.5.1 or greater. ESX 4.1 only comes with python 2.4.3. What I found is that for the basic operations, you actually do not need 2.5.1 and you can edit the gsutil utility to not check for the version.

3. Edit gsutil located in /root/gsutil/gsutil and change the following on line 1137
From:
To: 4. Login to Google Storage for Developers, you will need to setup key management under "Google Storage Manager" tab. This will be necessary for the next step.

5. For the first time, you will need to run gsutil to setup boto configuration file which will contain your Google access key ID and secret.

[root@esx4-1 ~]# gsutil ls -l
You have no boto config file. This script will create one at /root/.boto containing your credentials, based on your responses to the following questions. What is your Google access key ID? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX What is your Google secret access key? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Configuration file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. Please try running gsutil again now.

Once you have successfully provided the correct keys, you are now ready to use Google Storage from your ESX host.

To start using Google Storage, you will need to first create a bucket:

[root@esx4-1 ~]# gsutil mb gs://ghettovcb

Note: The bucket name must be all lower case, else you will get an error

We can now list the bucket that was just created:

[root@esx4-1 ~]# gsutil ls -l gs://ghettovcb/: ACL: [Owner:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, [UserById: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY]: u'FULL_CONTROL']

We can also see the new bucket using Google Storage Manager browser:

Let's say we have a Virtual Machine on the ESX host called "dummyVM" and we wanted to upload to Google Storage. We will want to compress the directory. [root@esx4-1 esx4-1-local-storage-1]# ls dummyVM/ dummyVM-flat.vmdk dummyVM.vmdk dummyVM.vmsd dummyVM.vmx dummyVM.vmxf [root@esx4-1 esx4-1-local-storage-1]# tar -zcf dummyVM.tar.gz dummyVM/
Upload to Google Storage:

[root@esx4-1 esx4-1-local-storage-1]# gsutil cp dummyVM.tar.gz gs://ghettovcb

We can verify by listing the contents of the bucket:

[root@esx4-1 esx4-1-local-storage-1]# gsutil ls gs://ghettovcb gs://ghettovcb/dummyVM.tar.gz

We can also verify by viewing the bucket on Google Storage Manager:

As you can see, it is pretty easy to manage your Google Storage using gsutil utility. For more information on other commands, take a look at Getting Started Guide for Google Storage. I was not able to get gsutil working on ESXi 4.1, because of missing python modules that were needed. I spent some time trying to copy existing modules from class ESX 4.1, but it did not resolve all dependencies.

VMware API Related Acronyms

by // 3 Comments

Duncan Epping recently wrote an article about VMware related acronyms and shared with the community  the history and origins of some of the acronyms (e.g. ESX, GSX). After reading Duncan's article, I realized the acronym list primarily focused on product names and features. They did not cover some of the API related acronyms that I was curious about. If you have worked with the VMware APIs, SDKs or vmkernel, vmkwarning, hostd, vpxa, vpxd, etc. logs you may have noticed some of these acronyms and wondered what they actually stood for.

Here is my compiled list of VMware API related acronyms that I have been able to dig up by looking at these various locations:

DMS: Data Management Service
EAM: ESX Agent Manager
FDM: Fault Domain Manager
HMO: Host Managed Object
LS: License Service
MOB: Managed Object Browser
MOR: Managed Object Reference
OMS:  Operation Management Service
RBD: Rule Based Deployment (Auto Deploy)
SMS: Storage Monitoring Service
SPS:  Storage Policy Service
STOREMON: Storage Utilization Monitoring
VCI: (vci-integrity deals with VMware Update Manager, but no info on acronym)
VMAM: VM Application Monitoring
VMODL: VMware Managed Object Design Language
VMOMI: VMware Managed Object Management Interface
VOD: VPX Operational Dashboard
VOB/VOBD: VMkernel Observation 
VORB: VMOMI Object Request Broker Library/Application
VSM: vService Manager
VWS: VMware Web Services
WOE: Workflow Orchestration Engine

Disclaimer: These namespaces may change without notice as they are part of the internal VMware APIs. These are not publicly documented by VMware and should not be relied on other than for informative purposes.

There was also a recent VMTN question regarding the following two licensed features that is displayed for a given ESX(i) host and it was not clear what they were used for:

Here are the actual names and purposes:

dpvmotion: Direct Path vMotion (VM DirectPath I/O)
dvfilter: vNetwork Appliance API (dvfilter was an internal name)

The following are not VMware acronyms, but you may see these from time to time if you work with the vSphere API/SDKs:

API: Application Programming Interface
SDK: Software Developement Kit
WSDL: Web Services Definition Language
SOAP: Simple Object Access Protocol

I would also like to thank both Steve Jin and Reuben Stump from VMware in helping me track down the last few acronyms that I could not figure out.

If there are any additional acronyms that I missed or ones I should add to the list, please feel free to leave me a comment.

Upcoming vSphere.Next Features Hinted in vSphere 4.1 APIs?

by // 2 Comments

Back in December, I created a VTMN document - How to browse the internal vSphere APIs which shows you how to access some of the undocumented vSphere API properties and methods. This was done prior to vSphere 4.1 release, but by looking at the python SDK stubs located on an ESX or ESXi host, you can see some of the some of the upcoming features that would eventually be released in vSphere 4.1.

Here is a look at some of these features found in the vSphere 4.0 SDK stubs:

Features known:

API STRING FEATURE
vim.version.drs Distributed Resource Scheduler
vim.version.dvs Distributed Virtual Switch
vim.version.ft Fault Tolerance
vim.version.ipv6 IPv6

Features eventually known:

API STRING FEATURE
vim.version.iorm I/O Resource Management what is now known as Storage I/O Control or SIOC

Features unknown:

API STRING FEATURE
vim.version.fed Unknown - (Federation?)
vim.version.lc Unknown - (Related properties in API referenced Linked Clones)
vim.version.logan Unknown - (Related properties in API referenced MediaAnalysisManager
vim.version.policy Unknown - (Rrelated properties in API referenced PolicyManager
vim.version.svm Unknown - (Rrelated properties in API referenced svmVmxDiskCopy)
vim.version.uber Unknown

Now, looking at vSphere 4.1 SDK stubs, I noticed a slew of new "potential" features that VMware is or maybe working on:

vim.version.cvp - ?unknown? (Client Virtualization Platform? below are properties and methods associated with this feature)

  • SetDisplayTopologyModes
  • VirtualMachinePowerPolicyPowerMode
  • VirtualMachinePowerPolicyCpuMode
  • requestedReplicationCanBeDeferred
  • userAllowedToDeferReplication
  • policyCacheLifetime

vim.version.dev - ?device? (below are properties and methods associated with this feature which seem to be related to FCoE support with vSphere)

  • DiscoverFcoeHbas
  • FcoeConfigFcoeSpecification
  • removeFcoeHba
  • userAllowedToDeferReplication

vim.version.h20 - (unknown, but related properties in API referenced PropertyProviderManager)

  • InternalPropertyProviderManager

vim.version.hbr - ?unknown? (Host Based Replication? below are properties and methods associated with this feature)

  • hostBasedReplicationSupported

vim.version.vcp - ?unknown? (VM Component Protection? below are properties and methods associated with this feature)

  • ClusterVmComponentProtectionSettings
  • ClusterVmComponentProtectionSettingsFtVmReaction

What you see here are not just string text found in the SDK stubs, these features are actually implemented in the current release of vSphere 4.1. Though undocumented, if you can reverse engineer where these methods and properties are located, you can actually execute them.

Here is an example of executing discoverFcoeHbas method via the MOB which is part of the Host's StorageSystem:

I am no fortune teller, but I suspect we may see some of these features much sooner than later 😉