WilliamLam.com

How to create vCenter Alarm to alert on ESXi 5.5u1 NFS APD issue?

by // 14 Comments

As some of you may have heard, there is currently a known issue with NFS based datastores (includes VSA NFS datastores) after upgrading to vSphere 5.5 Update 1. The issue causes NFS datastores to disconnect and go into an APD (All Paths Down) state. VMware is currently aware of the problem and you can follow KB 2076392 for the latest updates.

While going through my Twitter stream this morning, I noticed an interesting question from fellow Blogger and friend Jase McCarty who asked the following:

vsphere55u1-nfs-apd-alarm-2
I was quite surprised to hear that there were no vCenter Alarms being triggered for this issue. I decided to take a look at the KB to better understand the symptoms and see if there was anything I could do to help. From what I can tell, the only way to identify this particular problem is by looking at the logs which the KB has an example of what you would see.

Once I took a look at the logs, I knew there was at least two methods in which one could get alerts. One option would be to leverage vCenter Log Insight and create a query based on the particular string but no every customer is using Log Insight and it does require a bit of setup. The second more obvious option for me would be to key off of the VMkernel VOBs that are being generated which I have written about in the past for detecting duplicate IP Addresses for ESXi and VSAN component threshold count.

Here are the steps to create vCenter Alarm:

Step 1 - Create a new vCenter Alarm and give it a name. Select "Hosts" for Monitor and "Specific event occurring ..." for Monitor for

vsphere55u1-nfs-apd-alarm-0
Step 2 - For the Trigger, you will add the following VOB entries (just copy/paste them in)

  • esx.problem.storage.apd.start
  • esx.problem.vmfs.nfs.server.disconnect
  • esx.problem.storage.apd.timeout

Note: The alarm will activate if ANY of the VOBs are seen since it is an OR statement. It would have been nice to be able to group these together to generate the alarm

vsphere55u1-nfs-apd-alarm-1
Once the alarm has been created, you will at least have a way to get notified if you are potentially affected by this problem. I would still highly recommend you subscribe to KB 2076392 for all the latest updates.

Running Nested Xen Hypervisor with VMware Tools on ESXi

by // 1 Comment

I recently saw an interesting tweet from former VMware colleague Steve Muir who use to run our VMware Academic Program (VMAP). From the looks of things, Steve has been exploring the virtual rabbit hole by performing some cool multi-level vInception by running Xen in a Nested ESXi Virtual Machine running on top of VMware Fusion 🙂 I wonder if he has tried to run a VM in the Xen instance yet?

xen-vmware-tools-nested-4
Anyway, the thing that caught my eye was not the fact that you could run a Xen Hypervisor within ESXi, this has actually been done before in the past. What was interesting to me was the fact that he could not get VMware Tools to show up when Xen is installed and running. I was actually curious to see why this was the case and while waiting for some of my physical hosts to finish rebooting, I figure I give this a go myself. I was leveraging the Wiki page that Steve had initially provided but that gave me some issues as Xen would kernel panic upon a reboot.

In talking to Steve about this, he provided me with another link to a QuickStart guide for Xen which simplifies the setup and I was able to finally get it booted up (definitely not as easy as an ESXi install) . Afterwards, I just installed VMware Tools as I normally would and you can even use this one-liner script here that will automate the installation:

Funny enough, just by using this latest guide for the Xen installation, I was able to get VMware Tools to show up in the vSphere Web/C# Client without any additional tweaks. Here is a screenshot to prove this works:

xen-vmware-tools-nested-1
If you are interested in setting this up, here are the high level steps:

  1. Install latest CentOS which is 6.5 using either the full ISO installer or netinstall which is what I prefer to use
  2. Install Xen per the Wiki
  3. Install VMware Toosl (this step can be swapped with Step 2 if you, it works either way)

Thanks Steve for sharing your notes on getting Xen running on ESXi and I guess you ended up getting an answer to your question 😉

Quick Tip - How to snapshot & revert a physical ESXi host

by // 5 Comments

Nested environment, which is something I did quite a bit of as a customer and still continue to do so today. I could easily snapshot my Nested ESXi environment, perform my tests and then quickly rollback to my original starting state. However, when it comes to testing a physical ESXi host, it is a bit more challenging as there is no "quick" snapshot functionality as far as I was aware of. It was only until recently did I have a use case for this and picked up a nice tidbit from one of our engineers on the team. It turns out you could "snapshot" a physical or even virtual ESXi host by just backing up the state.tgz file and then restoring it. As the name suggest, the state.tgz file contains all the configurations of your ESXi host. The process is pretty straight forward:

  1. SCP /bootbank/state.tgz and back that up to your local system or shared storage
  2. Perform your tests or make changes to the system
  3. When you are ready to restore, copy the state.tgz back into /bootbank folder
  4. Login to ESXi Shell and run reboot -f which will ensure no changes are saved to our state.tgz

Once the ESXi host reboots, it will use the restored state.tgz file and your system will be back at its original state. This process is actually not new, ESXi already provides a way to backup/restore