How to backup VMs in ESX onto Google Storage

Over the weekend I received an email invite for Google Storage for Developers, which I had applied for a month or two back. I did not think I would get into the limited beta and at the time I was just curious about the offering from Google. Back in January I wrote an article How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud which uses Amazon's S3 storage offering and their s3cmd to upload a VM into the cloud. I spent a little time over the weekend playing with Google's storage offering and was able to install Google's python gsutil utility on to an ESX 4.1 host and backup a VM to Google Storage.

Before you begin, you will to have access to Google Storage for Developers and you will need to download the gsutil and transfer the tarball to your ESX host. The test was performed on an ESX 4.1 host, but I do not see why it will not work on ESX 4.0.

1. Extract the contents of gsutil.tar.gz

tar -zxvf gsutil.tar.gz

2. Edit .bashrc

vi .bashrc

Add the following two lines:

By default, gsutil requires you to be running python 2.5.1 or greater. ESX 4.1 only comes with python 2.4.3. What I found is that for the basic operations, you actually do not need 2.5.1 and you can edit the gsutil utility to not check for the version.

3. Edit gsutil located in /root/gsutil/gsutil and change the following on line 1137
From:
To: 4. Login to Google Storage for Developers, you will need to setup key management under "Google Storage Manager" tab. This will be necessary for the next step.

5. For the first time, you will need to run gsutil to setup boto configuration file which will contain your Google access key ID and secret.

[root@esx4-1 ~]# gsutil ls -l
You have no boto config file. This script will create one at /root/.boto containing your credentials, based on your responses to the following questions. What is your Google access key ID? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX What is your Google secret access key? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Configuration file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. Please try running gsutil again now.

Once you have successfully provided the correct keys, you are now ready to use Google Storage from your ESX host.

To start using Google Storage, you will need to first create a bucket:

[root@esx4-1 ~]# gsutil mb gs://ghettovcb

Note: The bucket name must be all lower case, else you will get an error

We can now list the bucket that was just created:

[root@esx4-1 ~]# gsutil ls -l gs://ghettovcb/: ACL: [Owner:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, [UserById: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY]: u'FULL_CONTROL']

We can also see the new bucket using Google Storage Manager browser:

Let's say we have a Virtual Machine on the ESX host called "dummyVM" and we wanted to upload to Google Storage. We will want to compress the directory. [root@esx4-1 esx4-1-local-storage-1]# ls dummyVM/ dummyVM-flat.vmdk dummyVM.vmdk dummyVM.vmsd dummyVM.vmx dummyVM.vmxf [root@esx4-1 esx4-1-local-storage-1]# tar -zcf dummyVM.tar.gz dummyVM/
Upload to Google Storage:

[root@esx4-1 esx4-1-local-storage-1]# gsutil cp dummyVM.tar.gz gs://ghettovcb

We can verify by listing the contents of the bucket:

[root@esx4-1 esx4-1-local-storage-1]# gsutil ls gs://ghettovcb gs://ghettovcb/dummyVM.tar.gz

We can also verify by viewing the bucket on Google Storage Manager:

As you can see, it is pretty easy to manage your Google Storage using gsutil utility. For more information on other commands, take a look at Getting Started Guide for Google Storage. I was not able to get gsutil working on ESXi 4.1, because of missing python modules that were needed. I spent some time trying to copy existing modules from class ESX 4.1, but it did not resolve all dependencies.