ESXi – Page 129

Quick Tip - How to BSOD/Panic a Virtual Machine in ESXi?

07.14.2014 by William Lam // Leave a Comment

https://twitter.com/virtualdominic/status/488570929383890945

While catching up on my Twitter feed this morning, I saw an interesting question from Dominic Rivera. I remember a few years back I was troubleshooting an issue and required to panic a Virtual Machine, I did not recall the steps that GSS had walked me through but I figure if anyone would know, it would be my buddy Paudie who used to work in GSS. Sure enough, he had the answer which was using a nifty little tool on the ESXi Shell called vmdumper.

UPDATE: It turns out there is an even simpler thanks to Frank Buechsel who is also from GSS. Frank mentioned you can send an NMI request simply using either the vSphere Web/C# Client. In the vSphere Web Client, you just need to right click on the VM and go to All vCenter Actions->Export Systems Logs. To be able to perform this action, you will need the Global->Diagnostic privilege for the the given user.

Next, expand the menu and select Send_NMI_To_Guest option.

If you wish to perform this operation using the vSphere C# Client, you just need to select the VM and then at the top menu and click File->Export System Logs and you will have the option of sending the NMI request this way.

Using the vmdumper utility, you can create a BSOD (Blue Screen of Death) for a Windows or Kernel Panic for Linux Virtual Machines by sending an NMI (Non-Maskable Interrupt) request.

To send an NMI or any other request, you need to first identify the VM's World ID, this can be done using either ESXCLI or vmdumper utility itself.
Using ESXCLI:

esxcli vm process list | grep -A1 '^[VMNAME]'

Using vmdumper:

vmdumper -l | grep [VMNAME]

In the example above, I have a VM called vcenter55-2 running Windows 2008 R2 and the World ID is 23100118. To send the NMI request, you will now run the following command:

vmdumper [WORLD-ID] nmi

If we now take a look at the VM Console, we will see that it successfully BSOD the Windows VM:

Here is a screenshot for sending an NMI request to a Linux VM. It seems the Linux system handles the NMI in a much more graceful manner and the system was still responsive 🙂

After learning about the vmdumper utility, I also performed a quick search online and found VMware KB 2005715 that provides some additional information. It looks like the KB only references ESXCLI as a way to locate the World ID, I will see if we can get the KB updated to also include vmdumper method of identifying the VM's World ID.

Does reinstalling ESXi with an existing VSAN Datastore wipe your data?

07.11.2014 by William Lam // 16 Comments

The quick answer is no, reinstalling ESXi on a host that contained VSAN data will not be touched or destroyed. This, of course, assumes you do not touch any of the disks that contains your VSAN data. To further assist administrators, you can also see which disks are claimed by VSAN during the the disk selection of a reinstall/upgrade of ESXi, so the proper disk is selected.

This to me is a very important fact and probably a very conscious design decision by the VSAN Engineers to ensure that you do not accidentally wipe your data because ESXi needed to be reinstalled. The topic has been one I have been wanting to write about for awhile but just never found the time. The reason I am bringing this up is that I saw an interesting tweet the other day from fellow VMware colleague, Geordy Korte who works over in the NSBU that had the following request:

NEED HELP: Anyone have any clue on how to recreate a #VSAN using esxcli and not wiping the data. My Vcenter is on the VSAN. RT PLZ

— Geordy Korte (@gekort) July 2, 2014

Geordy needed to reinstall ESXi but he did not want to lose any of the VSAN data residing on the disks within the ESXi host which also contained his vCenter Server among other Virtual Machines. Fortunately, reinstalling ESXi is a very safe operation and it will not touch any of the VSAN partitions. In fact, after the installation, if ESXi detects there are partitions that contains VSAN data, it will automatically claim the devices for you. You may have even seen this as part of the ESXi boot process:

Going back to Geordy's original question, how do you re-create the VSAN Datastore after reinstalling ESXi? Well, the process is actually pretty straight forward and is quite similar to how you bootstrap vCenter Server onto a VSAN Datastore. To demonstrate this process, I decided to take a fully functional 3-Node VSAN setup that contained a Virtual Machine and reinstall ESXi on all three nodes. I will now take you through the process of restoring the VSAN Datastore and at the end I will also talk about an alternative approach depending on the situation you might be in.

Step 1 - You can easily check whether you have an existing VSAN Disk Group by running the following ESXCLI command:

esxcli vsan storage list

Step 2 - You will need to enable VSAN traffic type for the VMkernel interface you wish to run VSAN on by running the following command and specifying the interface (this will need to be done on ALL ESXi hosts):

esxcli vsan network ipv4 add -i vmk0

Step 3 - You will need to re-create a new VSAN Cluster and you can do so by running the following command which will generate a UUID and create a VSAN Cluster from that:

esxcli vsan cluster join -u $(python -c 'import uuid; print(str(uuid.uuid4()))')

Step 4 - Once the VSAN Cluster has been created, you will need to make a note of the UUID that was generated so you can join the remaining ESXi hosts to the same VSAN Cluster. To do so, you will need to run the following command:

esxcli vsan cluster get

You will need to look for "Sub-Cluster UUID" property as seen in the screenshot above highlighted in green.

Step 5 - On the remainder ESXi hosts, you can now join the the VSAN Cluster using the following command and specifying the UUID from the previous step:

esxcli vsan cluster join -u [UUID]

Step 6 - Once all ESXi hosts have re-joined the VSAN Cluster, you can now go into the VSAN Datastore and re-register all the Virtual Machines using either the vSphere C# Client or from the CLI on the ESXI Shell.

As you can see from the screenshot above, I was able to recover my Virtual Machine and there was no data loss!

If you recall earlier, I also mentioned there was an alternative approach that you could take depending on the situation. If you need to deploy a new vCenter Server for what ever reason, you can easily restore your existing VSAN Cluster by simply creating a new VSAN Cluster in the new vCenter Server. You would then add each ESXi hosts to the VSAN Cluster and the VSAN Datastore and its contents will automatically be restored without any issues. Once you have your VSAN Datastore, you can then re-create the missing VM Storage Policies and re-apply them to the respective Virtual Machines.

In this article I have described two potential scenarios when working with VSAN and in both cases you can safely recover your data without any problems. This is one of the things I have come to appreciate about VSAN is the amount of engineering effort to make it super simple to use but also very resilient!

Quick Tip - Steps to shutdown/startup VSAN Cluster w/vCenter running on VSAN Datastore

07.08.2014 by William Lam // 11 Comments

I know Cormac Hogan already wrote about this topic awhile ago, but there was a question that was recently brought up that included a slight twist which I thought it would be useful to share some additional details. The question that was raised: How do you properly shutdown an entire VSAN Cluster when vCenter Server itself is also running on the VSAN Datastore? One great use case for VSAN in my opinion is a vSphere Management Cluster that would contain all your basic infrastructure VMs including vCenter Server which can be bootstrapped onto a VSAN Datastore. In the event that you need to shutdown the entire VSAN Cluster which may also include your vCenter Server, what is the exact procedure?

To help answer this question, I decided to perform this operation in my own lab which contains a 3-Node (physical) VSAN Cluster that had several VMs running on the VSAN Datastore including the vCenter Server VM that was managing the VSAN Cluster.

Below are the steps that I took to properly shutdown down a VSAN Cluster as well as powering everything back on.

UPDATE (4/27) - Added instructions for shutting down a VSAN 6.0 Cluster when vCenter Server is running on top of VSAN.

Shutdown VSAN Cluster (VSAN 6.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM, that will be the last VM you shutdown.

Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

Step 3 - Ensure that there are no vSAN Components being resync'ed before proceeding to the next step. You can find this information by going to the vSAN Cluster and under Monitor->vSAN->Resyncing Components as shown in the screenshot below.

Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

Step 5 - Next, you will need to place ALL ESXi hosts into Maintenance Mode. However, you must perform this operation through one of the CLI methods that supports setting the VSAN mode when entering Maintenance Mode. You can either do this by logging directly into the ESXi Shell and running ESXCLI locally or you can invoke this operation on a remote system using ESXCLI.

Here is the ESXCLI command that you will need to run and ensure that "No Action" option is selected when entering Maintenance Mode:

esxcli system maintenanceMode set -e true -m noAction

Step 5 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Shutdown VSAN Cluster (VSAN 1.0)

Step 1 - Shutdown all Virtual Machines running on the VSAN Cluster except for the vCenter Server VM.

Step 2 - To help simplify the startup process, I recommend migrating the vCenter Server VM to the first ESXi host so you can easily find the VM when powering back on your VSAN Cluster.

Step 3 - Place all ESXi hosts into Maintenance Mode except for the ESXi host that is currently running the vCenter Server. Ensure you de-select "Move powered-off and suspend virtual machines to other hosts in the Cluster" as well as selecting the "No Data Migration" option since we do not want any data to be migrated as we are shutting down the entire VSAN Cluster.

Note: Make sure you do not shutdown any of the ESXi hosts during this step because the vCenter Server VSAN Components are distributed across multiple hosts. If you do this, you will be unable to properly shutdown the vCenter Server VM because its VSAN components will not available.

Step 4 - Shutdown the vCenter Server VM which will now make the vSphere Web Client unavailable.

Step 6 - Finally, you can now shutdown all ESXi hosts. You can login to each ESXi hosts using either the vSphere C# Client / ESXi Shell or you can also perform this operation remotely using the vSphere API such as leveraging PowerCLI as an example.

Startup VSAN Cluster

Step 1 - Power on all the ESXi hosts that is part of the VSAN Cluster.

Step 2 - Once all the ESXi hosts have been powered on, you can then login to the ESXi host that contains your vCenter Server. If you took my advice earlier from the shutdown procedure, then you can login to the first ESXi host and power on your vCenter Server VM.

Note: You can perform steps 2-4 using the vSphere C# Client but you can also do this using either the API or simply calling vim-cmd from the ESXi Shell. To use vim-cmd, you need to first search for the vCenter Server VM by running the following command:

vim-cmd vmsvc/getallvms

You will need to make a note of the Vmid and in this example, our vCenter Server has Vmid of 6

Step 3 - To power on the VM, you can run the following command and specify the Vmid:

vim-cmd vmsvc/power.on [VMID]

Step 4 - If you would like to know when the vCenter Server is ready, you can check the status of VMware Tools as that should give you an indication that system is up and running. To do so, you can run the following command and look for the VMware Tools status:

vim-cmd vmsvc/get.guest [VMID]

Step 5 - At this point, you can now login to the vSphere Web Client and take all of your ESXi hosts out of Maintenance Mode and then power on the rest of your VMs.

As you can see the process to shutdown an entire VSAN Cluster even with vCenter Server running on the VSAN Datastore is fairly straight forward. Once you are comfortable with the procedure, you can even automate this entire process using the vSphere API/CLI, so you do not even need a GUI to perform these steps. This might even be a good idea if you are monitoring a UPS and have an automated way of sending remote commands to shutdown your infrastructure.