WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How to backup VMs in ESX onto Google Storage

08.08.2010 by William Lam // 4 Comments

Over the weekend I received an email invite for Google Storage for Developers, which I had applied for a month or two back. I did not think I would get into the limited beta and at the time I was just curious about the offering from Google. Back in January I wrote an article How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud which uses Amazon's S3 storage offering and their s3cmd to upload a VM into the cloud. I spent a little time over the weekend playing with Google's storage offering and was able to install Google's python gsutil utility on to an ESX 4.1 host and backup a VM to Google Storage.

Before you begin, you will to have access to Google Storage for Developers and you will need to download the gsutil and transfer the tarball to your ESX host. The test was performed on an ESX 4.1 host, but I do not see why it will not work on ESX 4.0.

1. Extract the contents of gsutil.tar.gz

tar -zxvf gsutil.tar.gz

2. Edit .bashrc

vi .bashrc

Add the following two lines:

By default, gsutil requires you to be running python 2.5.1 or greater. ESX 4.1 only comes with python 2.4.3. What I found is that for the basic operations, you actually do not need 2.5.1 and you can edit the gsutil utility to not check for the version.

3. Edit gsutil located in /root/gsutil/gsutil and change the following on line 1137
From:
To: 4. Login to Google Storage for Developers, you will need to setup key management under "Google Storage Manager" tab. This will be necessary for the next step.

5. For the first time, you will need to run gsutil to setup boto configuration file which will contain your Google access key ID and secret.

[root@esx4-1 ~]# gsutil ls -l
You have no boto config file. This script will create one at /root/.boto containing your credentials, based on your responses to the following questions. What is your Google access key ID? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX What is your Google secret access key? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Configuration file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. Please try running gsutil again now.

Once you have successfully provided the correct keys, you are now ready to use Google Storage from your ESX host.

To start using Google Storage, you will need to first create a bucket:

[root@esx4-1 ~]# gsutil mb gs://ghettovcb

Note: The bucket name must be all lower case, else you will get an error

We can now list the bucket that was just created:

[root@esx4-1 ~]# gsutil ls -l gs://ghettovcb/: ACL: [Owner:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, [UserById: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY]: u'FULL_CONTROL']

We can also see the new bucket using Google Storage Manager browser:

Let's say we have a Virtual Machine on the ESX host called "dummyVM" and we wanted to upload to Google Storage. We will want to compress the directory. [root@esx4-1 esx4-1-local-storage-1]# ls dummyVM/ dummyVM-flat.vmdk dummyVM.vmdk dummyVM.vmsd dummyVM.vmx dummyVM.vmxf [root@esx4-1 esx4-1-local-storage-1]# tar -zcf dummyVM.tar.gz dummyVM/
Upload to Google Storage:

[root@esx4-1 esx4-1-local-storage-1]# gsutil cp dummyVM.tar.gz gs://ghettovcb

We can verify by listing the contents of the bucket:

[root@esx4-1 esx4-1-local-storage-1]# gsutil ls gs://ghettovcb gs://ghettovcb/dummyVM.tar.gz

We can also verify by viewing the bucket on Google Storage Manager:

As you can see, it is pretty easy to manage your Google Storage using gsutil utility. For more information on other commands, take a look at Getting Started Guide for Google Storage. I was not able to get gsutil working on ESXi 4.1, because of missing python modules that were needed. I spent some time trying to copy existing modules from class ESX 4.1, but it did not resolve all dependencies.

Categories // Uncategorized Tags // ESX 4.0, ESXi 4.1, gsutil, vSphere 4.1

esxcli Part3 - Automating esxcli using PowerShell

06.18.2010 by William Lam // Leave a Comment

In this final three part esxcli series, I will show you how to automate esxcli operations in a Windows environment using PowerShell. There was a recent question on the VMTN forums regarding this very topic. As a simple solution to the problem, a community member provided a snippet of PowerShell code. Of course the snippet of PowerShell code originated from the great PowerCLI Master - Luc Dekens.

I have slightly modified the original version of the script as I am not a fan of hard coding passwords within a script. This version of the script requires plink which is a command-line interface to the free Windows SSH PuTTY back end. The script performs an SSH connection to either an ESX or ESXi host to run esxcli command locally, with the specified parameters as you would if you were on the Service Console, or the unsupported Busybox console.

Copy the snippet of code below and create a new script called esxcli-automation1.ps1 and replace all the variables matching your environment including the $cmd which specifies the exact esxcli command you are running on the remote host.

esxcli-automation1.ps1

$User = "root"
$Computer = "esxi4-1.primp-industries.com"

#prompt user for password
$passwordInput = Read-Host -AsSecureString:$true "Please enter password for $Computer"

#convert secure password to normal string
$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)
$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)
[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

$plink = "C:\plink.exe"
$esxcli = "esxcli"
$plinkoptions = " -v -batch -pw '$Password'"
$cmd = 'nmp device list'
$remoteCommand = $esxcli + " " + '"' + $cmd + '"'
$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + $remoteCommand
Invoke-Expression -command $command

Here is a sample execution listing devices:

There are a few issues with this script:

  • Assumes SSH connectivity is open to the host, which on classic ESX w/Service Console is probably a valid reason but what about ESXi? We know that we should not be using the unsupported Busybox console on ESXi and we can not make the assumption that it will be opened
          Here is the same example when executing against an ESXi host:

  • The script can only execute against one host at a time.

In a perfect world, the esxcli API would be properly exposed and integrated into the vSphere API. PowerCLI and other vSphere SDKs would be able to automate these configurations without this hack, but unfortunately this is not the case. However, we can leverage the esxcli utility that is bundled up with the vCLI which can be installed on either a Linux or Windows platform.

With this information, I decided to write my own PowerShell script (yes you heard right, it was slightly difficult without knowing the syntax and format) matching the capabilities similar to that of the vSphere SDK for Perl implementation running on vMA. So here is the second revision of the script called esxcli-automation2.ps1:

esxcli-automation2.ps1

$Username = "root"
$esxcli = "C:\Program Files\VMware\VMware vSphere CLI\bin\esxcli.exe"
#$Params = "swiscsi nic list -d vmhba33"

#list of ESX and ESXi host to perform operation against
$vihosts = @("esxi4-1.primp-industries.com","esxi4-3.primp-industries.com")

#prompt user for password
$passwordInput = Read-Host -AsSecureString:$true "Please enter your common password for hosts"

#convert secure password to normal string
$Ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToCoTaskMemUnicode($passwordInput)
$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringUni($Ptr)
[System.Runtime.InteropServices.Marshal]::ZeroFreeCoTaskMemUnicode($Ptr)

#loop through array of hosts
for($i = 0; $i -le $vihosts.length -1; $i++) {
 write-Host Executing operation on $vihosts[$i]
 #could not find a way to store the params in a variable you must
 #insert esxcli parameters at the very end of the statement below
 & $esxcli --server $vihosts[$i] --username $Username --password $Password swiscsi nic list -d vmhba33
}

Note: PowerCLI is not required, just PowerShell.

This new version of the script allows you specify a list of ESX or ESXi host to perform a common esxcli operation. The script also prompts for the password as the previous script, so you do not have to hard code the password to the host. You will need to update the script with the path to the plink.exe and the list of hosts you want to perform the operation against.

Here is an example of verifying iSCSI multipathing configuration against two hosts (1 ESX and 1 ESXi) without the use of SSH:

As you can see, you can easily integrate this with other existing PowerCLI scripts that perform configuration changes across a given set of host(s). Until VMware provides access to the esxcli APIs, these are some of the methods/hacks that need to be implemented when automating configuration changes using esxcli.

Here are some PowerShell references that were used in creating the two scripts:

  • http://technet.microsoft.com/en-us/library/ee176935.aspx
  • http://www.rickcable.com/blog/default.asp?Display=33
  • http://bsonposh.com/archives/247

Categories // Uncategorized Tags // api, ESX 4.0, esxcli, vSphere

esxcli Part2 - Automating esxcli using vMA

06.17.2010 by William Lam // 3 Comments

In the previous article, esxcli Part1, we discussed what esxcli is and what it can be used for. In this article, we will show you how you can automate and perform a common esxcli operation against a set of ESX and/or ESXi hosts using vMA. As mentioned in the previous article, esxcli needs to be executed against a specific host, it is not vCenter aware and cannot connect to vCenter to perform an operation against a host.

One of the pre-requisites before starting is that all ESX and ESXi hosts need to be managed by vMA and accessible using vi-fastpass authentication. Please refer to the vMA documentation on configuring this on vMA.

Download:esxcli-automation.pl

The following script is based off of useVIFastpassOnvMAToRunPerlScriptWithoutClearTextPassword.pl. The modified script utilizes the vi-fastpass library to properly access a set of target(s) and performs the specified esxcli operation without having to provide credentials to each and every host.

The following example will demonstrate iSCSI multipath configuration on two hosts using the method described by Duncan Epping in this article.

1) Create a file containing the list host to perform the operations against:

[vi-admin@scofield ]$ cat hosts
esxi4-1.primp-industries.com
esxi4-3.primp-industries.com

2. Run the script with the specific esxcli parameters as you would normally by passing it into the --cmd_option argument (double quote the arguments):

Binding iSCSI interface to vmk0

[vi-admin@scofield ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk0 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...
Binding iSCSI interface to vmk1

[vi-admin@scofield ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic add -n vmk1 -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
Executing script on "esxi4-3.primp-industries.com" ...

3. Verify iSCSI multipathing has been properly configured:

[vi-admin@scofield ]$ ./esxcli-automation.pl --hostlist hosts --cmd_option "swiscsi nic list -d vmhba33"
Executing script on "esxi4-1.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.183
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:69:7a
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 167051261
packets sent: 232132
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.184
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:68:6c
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 54502
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
Executing script on "esxi4-3.primp-industries.com" ...
vmk0
pNic name: vmnic0
ipv4 address: 172.30.0.233
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:33:95
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 138535
packets sent: 8026
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A
vmk1
pNic name: vmnic1
ipv4 address: 172.30.0.234
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:50:56:92:27:65
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 145924
packets sent: 112
NIC driver: e1000
driver version: 8.0.3.1-NAPI
firmware version: N/A

Now you will be able to automate any of the supported esxcli operations against multiple hosts!

In Part3, we will take a look at automating esxcli operations in a Windows environment using Powershell.

Categories // Uncategorized Tags // api, ESX 4.0, esxcli, vSphere

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025