vmdumper

https://twitter.com/virtualdominic/status/488570929383890945

While catching up on my Twitter feed this morning, I saw an interesting question from Dominic Rivera. I remember a few years back I was troubleshooting an issue and required to panic a Virtual Machine, I did not recall the steps that GSS had walked me through but I figure if anyone would know, it would be my buddy Paudie who used to work in GSS. Sure enough, he had the answer which was using a nifty little tool on the ESXi Shell called vmdumper.

UPDATE: It turns out there is an even simpler thanks to Frank Buechsel who is also from GSS. Frank mentioned you can send an NMI request simply using either the vSphere Web/C# Client. In the vSphere Web Client, you just need to right click on the VM and go to All vCenter Actions->Export Systems Logs. To be able to perform this action, you will need the Global->Diagnostic privilege for the the given user.

Next, expand the menu and select Send_NMI_To_Guest option.

If you wish to perform this operation using the vSphere C# Client, you just need to select the VM and then at the top menu and click File->Export System Logs and you will have the option of sending the NMI request this way.

Using the vmdumper utility, you can create a BSOD (Blue Screen of Death) for a Windows or Kernel Panic for Linux Virtual Machines by sending an NMI (Non-Maskable Interrupt) request.

To send an NMI or any other request, you need to first identify the VM's World ID, this can be done using either ESXCLI or vmdumper utility itself.
Using ESXCLI:

esxcli vm process list | grep -A1 '^[VMNAME]'

Using vmdumper:

vmdumper -l | grep [VMNAME]

In the example above, I have a VM called vcenter55-2 running Windows 2008 R2 and the World ID is 23100118. To send the NMI request, you will now run the following command:

vmdumper [WORLD-ID] nmi

If we now take a look at the VM Console, we will see that it successfully BSOD the Windows VM:

Here is a screenshot for sending an NMI request to a Linux VM. It seems the Linux system handles the NMI in a much more graceful manner and the system was still responsive 🙂

After learning about the vmdumper utility, I also performed a quick search online and found VMware KB 2005715 that provides some additional information. It looks like the KB only references ESXCLI as a way to locate the World ID, I will see if we can get the KB updated to also include vmdumper method of identifying the VM's World ID.

Last week I remember seeing a tweet from Duncan Epping regarding the use of VMware vsish to obtain the VMX Cartel ID, also known as the LWID (Leader World Id) for a virtual machine world. This VMX Cartel ID is then used to obtain the GID (Group Id) which was documented by Duncan on his esxtop page to limit the view in esxtop to a particular VM of interest:

VMWID=`vm-support -x | grep |awk '{gsub("wid=", "");print $1}'`
VMXCARTEL=`vsish -e cat /vm/$VMWID/vmxCartelID`
vsish -e cat /sched/memClients/$VMXCARTEL/SchedGroupID

As you can see from the above, Duncan utilizes the vm-support to obtain the WID (World ID) and then using vsish to query for the VMX Cartel ID (LWID). Finally, using LWID, he was able to obtain the GID (Group ID). This example only applies to ESXi, since classic ESX does not include vsish tool.

Here is an example of where to locate the values in esxtop under CPU section:

I remember during some of my skunk work adventures, there were other methods of obtaining these IDs. Due to my VCAP-DCD cramming last week, I did not get a change to further investigate. Now that I have some time, I thought share some of these other methods for obtaining the GID and LWID in both ESX and ESXi.

ESXi

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

~ # /sbin/vmdumper -l
wid=16881 pid=-1 cfgFile="/vmfs/volumes/4cdeeb09-1ad4c18a-5ff9-003048d9586a/scofield/scofield.vmx" uuid="42 34 e6 bc f1 83 c2 db-fb fb 08 73 b7 0c 26 40" displayName="scofield" vmxCartelID=16880

ps - The VMX Cartel ID is actually both the PID and CID within process status in ESXi

~ # ps -Cc | grep "scofield" | grep -v grep | awk '{print $2}'
16880

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

~ # /sbin/vscsiStats -l | grep "scofield" | awk '{print $4}' | sed 's/,//g'
16880

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

~ # /sbin/esxcli vms vm list | grep -A3 "scofield" | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
16880

Obtaining GID

Method1 - Using vscsiStats and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method2 - Using esxcli and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
4639

Method3 - Using esxcli and vsish to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');/sbin/vsish -e cat /sched/memClients/${VMX_CARTEL}/SchedGroupID
4639

Note: You just need to substitute VM_NAME variable with the display name of of the virtual machine you are interested in. There are actually multiple commands being executed in this one line. If your VM has spaces, make sure you put quotes around it

ESX

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

[root@himalaya ~]# /usr/lib/vmware/bin/vmdumper -l
wid=29250 pid=-1 cfgFile="/vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/STA202G/STA202G.vmx" uuid="42 30 d1 75 c5 3e 81 2a-14 15 1f 86 bb 5b b9 e5" displayName="STA202G" vmxCartelID=29249

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

[root@himalaya ~]# /usr/lib/vmware/bin/vscsiStats -l | grep "STA202G" | awk '{print $4}' | sed 's/,//g'
29249

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

[root@himalaya ~]# /usr/sbin/esxcli vms vm list | grep -A3 STA202G | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'
29249

Obtaining GID

Method1 - Using esxcli and esxcfg-info to extract VM's GID

[root@himalaya ~]# VM_NAME=STA202G ;VMX_CARTEL=$(/usr/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'
197

Method2 - Using sched-stats to extract GID

[root@himalaya ~]# VM_NAME=STA202G;VMX_CARTEL=$(/usr/lib/vmware/bin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');/usr/bin/sched-stats -t groups | grep "vm.${VMX_CARTEL}" | awk '{print $1}'
197

Method3 - Using /proc information from sched

[root@himalaya ~]# VM_NAME=STA202G ;grep "${VM_NAME}" /proc/vmware/sched/drm-stats | awk '{print $1}'
197

As you can see, there are more than one way to obtain the same exact values and I am sure there are probably a few others. For command simplicity, I would probably recommend method #3 for ESXi and method #3 for ESX. As with anything, be careful when you using these methods as they are not really supported by VMware unless directed by their support organization.

Quick Tip - How to BSOD/Panic a Virtual Machine in ESXi?

How to obtain GID and LWID from esxtop?