WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How to kill a stuck VM on ESX(i) 4.1 using esxcli

08.14.2010 by William Lam // 1 Comment

With so many new features in vSphere 4.1, you might have not noticed the new additions to the esxcli utility. One new feature is the ability to remotely kill a stuck or hung virtual machine running on ESX or ESXi. Prior to vSphere 4.1, the process of killing a stuck VM meant you had to use a variety of methods and tools that included vmware-cmd, vm-support, vmkload_app and ps/kill and required the use of either Service Console on ESX or Tech Support Mode for ESXi. Depending on how "stuck" your VM was, you would start from the most gentle method to killing your VM with prejudice (kill -9 [pid]).

A new namespace called "vms" has now been added and there are two available operations: list running VMs on a host or killing a specific VM. To kill a VM, you will need to specify the kill type and VM's world id.

Here is an example of listing all running VMs on an ESXi host:

When a VM gets into a stuck state in which you have no other option than to forcefully stop the VM, you will have three types of options to kill a VM: soft, hard and force. Per the help command, you should always perform the kill operations in the following order:

soft - Will give the VMX process a chance to shutdown cleanly (like kill or kill -SIGTERM)
hard - Will shutdown the process immediately (like kill -9 or kill -SIGKILL)
force - Should be used as a last resort attempt to kill the VM

The final resort, is to reboot the host if none of the above work.

Here is an example of various kill types:

Here is an example of killing a VM using the "soft" method, you should see the value of "true" if the operation was successful.

This was one feature users have been asking for with regards to managing VMs on ESXi, where the Service Console was no longer available. You do not need to login to Tech Support Mode to forcefully kill a VM and from what I have been told, if the above does not work, it will not work in Tech Support Mode either. I am hoping a public API for esxcli operations will be exposed which can be leverage by the various vSphere SDK's (e.g. vSphere SDK for Perl, PowerCLI, VI Java, etc.)

Categories // Uncategorized Tags // esxcli, vSphere 4.1

Why you should upgrade from vMA 4.0 to vMA 4.1

08.10.2010 by William Lam // 5 Comments

As you may know, with the release of vSphere 4.1, core vSphere components such as vCenter and ESX(i) all received significant feature updates. This was no exception with VMware vMA, though the number of updates were minimal but still signifcant.

Here is the list of new features:

  • Support for Windows Active Directory Integration using Likewise Open
  • Offline bundle support for upgrading vMA
  • Resolved major security flaw with vi-fastpass component

1. Support for Windows Active Directory Integration using Likewise Open
For those that use Active Directory, you now have the capability of centralizing your user management on vMA. Using Likewise's Open software, users can now login to vMA using their AD credentials. Not only can they login, but with the enhancement of the new vi-fastpass, you can now add a target and configure it to support AD authentication versus normal fastpass authentication. This means that the user who logs into vCenter can log into vMA to perform an operation such as adding a vSwitch or powering on a Virtual Machine without providing further credentials using this new AD authentication passthrough. For more details, take a look at this article vMA 4.1 Authentication Policy - fpauth vs adauth.

2. Offline bundle support for upgrading vMA
Prior to vMA 4.1, to upgrade the software within vMA, you had to use vima-update (now renamed to vma-update) which connects to VMware's online repository. This was an issue for envrionments that did not have direct internet or proxy access to VMware's website and there were no supported work arounds for this. You now can download an offline bundle just like you can for  ESX or ESXi and perform an offline upgrade.

3. Resolved major security flaw with vi-fastpass component
Saving the best for last. This has been a security issue that has existed since the release of VIMA 1.0 which was not considered an offiical release by VMware until vMA 4.0. The issue was with the use of vifpinit (now renamed to vifptarget) and when a target was initialized using the vi-fastpass library, the username and password was viewable in plain text.

When you add a target to vMA's management using vi-fastpass, two accounts are created on the target host: vi-adminXX and vi-userXX. The password for these two accounts are obfuscated using a simple XOR cipher which can be cracked in days if not hours and the obfuscated form is stored in a credential store located in /home/vi-admin/.vmware/credstore. Prior to vMA 4.1, when you initialize a target from this credential store, you are allowed to execute any vCLI or vSphere SDK for Perl script without having to provide additional credentials to the host. The authentication is basically bypassed as it uses the vi-adminXX credentials to login and perform the operation. The side effect of initializing the fastpass target, was that the password to the vi-adminXX was exposed as plain text via an environmental variable.

Here is an example of of initializing a target on vMA 4.0 with the password exposed:

As you can see from above, both the username and password is exposed as a simple environmental variable entry. You can now take those credentials and login to an ESX(i) host and perform an administrative operation as if you were root.

Here is an example of using these credentials to create a new vSwitch on an ESX host:

Before you jump and upgrade vMA 4.1, note that the security issue has been resolved but it does NOT make the system any more secure than it was in vMA 4.0. The vi-fastpass is actually available as a library module that supports both Perl and Java. There is available sample code under /opt/vmware/vima/samples/ for vMA4.0 and /opt/vmware/vma/samples/ for vMA 4.1. Using this library, you can access the targets under vMA management which also includes both the username and password in its unencrypted form.

Here is a vMA 4.0 script I wrote called enumeratevifp40.pl that demonstrates the above:

Here is a vMA 4.1 script I wrote called enumeratevifp41.pl that demonstrates the above:

As you can see, even with the security issue fixed vMA 4.1, a user can still access the entire credential store and obtain both the username and password to any of the targets. The only real way to protect against this issue is to use adauth when adding your targets and forcing users to authenticate against your AD server when connecting and performing an operation on vCenter or an ESX(i) host. For those that must use standard fpauth, VMware has released a KB1017669 article for vMA 4.1 on securing the credential store by creating an encrypted volume in which the obfuscated file is stored in. This ensures that if you are transferring your vMA image, someone can not just mount the VMDK and extract the credential store file and try to decrypt the contents.

Now, that you understand the security implication and features available with vMA 4.1, off you go to upgrade 🙂

Here are additional links that may also be useful:

  • http://www.virtuallyghetto.com/2010/07/resxtop-vi-fastpass-downgraded-feature.html
  • http://www.virtuallyghetto.com/2010/07/vma-41-active-directory-intergration.html

Categories // Uncategorized Tags // vma, vSphere 4.1

How to backup VMs in ESX onto Google Storage

08.08.2010 by William Lam // 4 Comments

Over the weekend I received an email invite for Google Storage for Developers, which I had applied for a month or two back. I did not think I would get into the limited beta and at the time I was just curious about the offering from Google. Back in January I wrote an article How to install Amazon s3cmd utility on ESX(i) 4.0 & Backup VMs to the Cloud which uses Amazon's S3 storage offering and their s3cmd to upload a VM into the cloud. I spent a little time over the weekend playing with Google's storage offering and was able to install Google's python gsutil utility on to an ESX 4.1 host and backup a VM to Google Storage.

Before you begin, you will to have access to Google Storage for Developers and you will need to download the gsutil and transfer the tarball to your ESX host. The test was performed on an ESX 4.1 host, but I do not see why it will not work on ESX 4.0.

1. Extract the contents of gsutil.tar.gz

tar -zxvf gsutil.tar.gz

2. Edit .bashrc

vi .bashrc

Add the following two lines:

By default, gsutil requires you to be running python 2.5.1 or greater. ESX 4.1 only comes with python 2.4.3. What I found is that for the basic operations, you actually do not need 2.5.1 and you can edit the gsutil utility to not check for the version.

3. Edit gsutil located in /root/gsutil/gsutil and change the following on line 1137
From:
To: 4. Login to Google Storage for Developers, you will need to setup key management under "Google Storage Manager" tab. This will be necessary for the next step.

5. For the first time, you will need to run gsutil to setup boto configuration file which will contain your Google access key ID and secret.

[root@esx4-1 ~]# gsutil ls -l
You have no boto config file. This script will create one at /root/.boto containing your credentials, based on your responses to the following questions. What is your Google access key ID? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX What is your Google secret access key? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Configuration file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. Please try running gsutil again now.

Once you have successfully provided the correct keys, you are now ready to use Google Storage from your ESX host.

To start using Google Storage, you will need to first create a bucket:

[root@esx4-1 ~]# gsutil mb gs://ghettovcb

Note: The bucket name must be all lower case, else you will get an error

We can now list the bucket that was just created:

[root@esx4-1 ~]# gsutil ls -l gs://ghettovcb/: ACL: [Owner:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, [UserById: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY]: u'FULL_CONTROL']

We can also see the new bucket using Google Storage Manager browser:

Let's say we have a Virtual Machine on the ESX host called "dummyVM" and we wanted to upload to Google Storage. We will want to compress the directory. [root@esx4-1 esx4-1-local-storage-1]# ls dummyVM/ dummyVM-flat.vmdk dummyVM.vmdk dummyVM.vmsd dummyVM.vmx dummyVM.vmxf [root@esx4-1 esx4-1-local-storage-1]# tar -zcf dummyVM.tar.gz dummyVM/
Upload to Google Storage:

[root@esx4-1 esx4-1-local-storage-1]# gsutil cp dummyVM.tar.gz gs://ghettovcb

We can verify by listing the contents of the bucket:

[root@esx4-1 esx4-1-local-storage-1]# gsutil ls gs://ghettovcb gs://ghettovcb/dummyVM.tar.gz

We can also verify by viewing the bucket on Google Storage Manager:

As you can see, it is pretty easy to manage your Google Storage using gsutil utility. For more information on other commands, take a look at Getting Started Guide for Google Storage. I was not able to get gsutil working on ESXi 4.1, because of missing python modules that were needed. I spent some time trying to copy existing modules from class ESX 4.1, but it did not resolve all dependencies.

Categories // Uncategorized Tags // ESX 4.0, ESXi 4.1, gsutil, vSphere 4.1

  • « Previous Page
  • 1
  • …
  • 542
  • 543
  • 544
  • 545
  • 546
  • …
  • 560
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025