WilliamLam.com

How to run Nested RHEV Hypervisor on ESXi?

by // 6 Comments

I have written a number of articles about VMware Nested Virtualization and even today, I am still surprised at how easy it is to virtualize not only our own hypervisor but other vendor's hypervisors as well. This week I received an interesting question from my old Technical Marketing colleague Rawlinson Rivera who wanted to run a nested RHEV (Red Hat Enterprise Virtualization) Hypervisor on ESXi. This was not something I had done before nor had any interest in doing and I told Rawlinson that it should technically work as long as the guestOS is enabled with VHV.

Rawlinson's attempt at installing RHEV resulted in the VM hanging after boot up. After a bit of research, it turns out some additional tweaks are required to get RHEV running on ESXi. I would like to give a huge thanks to Jim Mattson, one of the VMware developers who help made Nested Virtualization possible, for his assistance.

Disclaimer: This is not officially supported by VMware. Please use at your own risk.

Here are the instructions on creating a virtual machine that can be used to install RHEV (make sure you follow these exact steps, the VM must be created with these settings or it will not work):

Step 1 - Download RHEV 6.3 or 6.4 from Red Hat's website

Step 2 - Create a new Virtual Machine (vHW9) and when you get to the OS selection, you will need to select the following:

Guest Family - Other
Guest Version - Other (64-bit)

Step 3 - When you get to the virtual hardware customization, make sure you select LSI Logic SAS for the SCSI controller and also enable VHV under the CPU option.

Step 4 - Finally, you will need to add the following two advanced virtual machine settings:
vcpu.hotadd = false
apic.xapic.enable = false

Step 5 - Mount the RHEV ISO and once the VM starts to boot up, when you are presented with install/upgrade options, hit the TAB key. This will allow you to change the boot parameters and you will need to move your cursor to the left and remove "quiet" from the command-line which is right after the install keyword and then hit enter.

Note: This is required due to a known issue from Red Hat.

Step 6 - If everything was successful, you should be prompted with RHEV installer:

Step 7 - Once the installer has completed and you reboot, you now have nested RHEV running on ESXi!

Now it is time to delete the VM 😉

Quick Tip - Correlating VMFS Datastore to Storage Device Using ESXCLI

by // 1 Comment

There was a question on Twitter this morning from AJ Kuftic on whether it is possible to display the mapping of a VMFS Datastore to its respective storage device using ESXCLI. Josh Coen beat me to the answer this morning, but yes it is possible using ESXCLI. I thought I still share this quick tip as it may not be obvious, especially when you need this information while performing a storage maintenance or troubleshooting with your storage administrator.

For those of you who are familiar with the legacy esxcfg-* commands, this information can be retrieved using the following command:

esxcfg-scsidevs -m

You can also retrieve this same information by using the following ESXCLI command (can be executed remotely as well):

esxcli storage vmfs extent list

As you can see from both screenshots, we can easily identify the name of the VMFS datastore and the specific storage device it is mapped along with other pieces of information. I prefer the ESXCLI method as it is nicely formatted along with the title header for each property.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 2

by // 7 Comments

In Part 1 I showed how you can forward virtual machine logs to ESXi syslog using an advanced virtual machine setting that was introduced in vSphere 5.1. A caveat with this solution is that the ESXi syslog file contains both system logs as well as virtual machine logs which is not very ideal from an isolation perspective. With virtual machine logs being quite verbose, if you are not forwarding logs to a remote syslog server, important system events can easily be rotated out of the local logs.

To work around this caveat, we can create a new logger specifically for handling virtual machine logs within the ESXi syslog client. You can view the existing logger types by looking in /etc/vmsyslog.conf.d directory. You will need to create a new logger configuration file which I named vmx.conf and it should contain the following:

[vmsyslog-logger]
# unique id for this logger
id = vmx
# description of this logger
descr = VMX Logs
# idents this logger is interested in
idents = vmx
# output file (e.g. foo == /var/log/foo.log)
file = vmx
# file logger class
fclass = FileLoggerSyslog
# network logger class
nclass = NetworkFilterSyslogTimestamp

Here is a screenshot of of my configuration file and noticed the highlighted text in yellow is what needs to be modified:

Note: Ensure that idents property matches the vmx.log.syslogID string specified for your virtual machines. This also means you will not be able to specify the virtual machine's name for the advanced setting, but will need to keep it generic so it can be filtered by the logger.

Once you have saved the vmx.conf configuration file, you will need to reload the ESXi syslog client for the changes to go into effect by running the following ESXCLI command:

esxcli system syslog reload

You now should see a new log file in /var/log called vmx.log which will contains only virtual machine logs:

If your ESXi host is forwarding its logs to vCenter Log Insight, you can easily create a filter for the keyword "vmx" in the log source or whatever string you decided to set it to if you are not using the default.

One final caveat to be aware of now is that the custom syslog logger (vmx.conf) will not persist after a system reboot. To preserve this file, you can either automatically re-create the file during bootup and reload syslog client using this article here OR create a custom VIB using this article here.