WilliamLam.com

Quick Tip - Useful ovftool debugging options

by // 7 Comments

This morning I needed perform several OVF uploads using ovftool and provide that information to engineering to investigate some performance issues. I tend to error on the side of providing more information than requested. The ovftool provides some really useful debugging options that are really handy in these situations but are un-documented. I can never seem to remember the exact syntax and hence I am documenting them here. I will also file a documentation bug to ensure these get added 🙂

UPDATE (08/19/13) - Thanks to one of the OVF engineers, it turns out you can see all the debug options and their definitions by running ovftool --help debug

The two options that I am referring to are:

--X:logFile=
--X:logLevel=

The first option allows you to log the entire ovftool session to a file which you can then send off to someone and the second option allows you to control the verbosity of the logs which I normally set to use verbose.

Here is an example of how you would use these ovftool options:

/Applications/VMware\ OVF\ Tool/ovftool --X:logFile=upload.log --X:logLevel=verbose -ds=mini-local-datastore-2 '--net:Network 1=VM Network' VMware-vCenter-Server-Appliance-5.1.0.10200-1235310_OVF10.ova vi://root@mini

Once the ovftool has completed its operation, you can take a look at the log and you will see quite a bit of information including some additional ovftool options that can be specified on the command-line which start with /X:

--> /X:httpTimeout = "0"
--> /X:imageReadSize = "262144"
--> /X:logFile = "upload.log"
--> /X:logLevel = "verbose"
--> /X:maxNumberOfTermSignals = "5"
--> /X:maxRedirects = "256"
--> /X:maximalDeltaConfSize = "8"
--> /X:maximalDeltaTreeSize = "6"
--> /X:progressSmoothing = "60"
--> /X:useMacNaming = "true"
--> /X:vCloudEnableGuestCustomization = "false"
--> /X:vCloudKeepTemplate = "true"
--> /X:vCloudTimeout = "3600"
--> /X:vimSessionTimeout = "600"

Note: I would not recommend tweaking the other options as the defaults should be sufficient, but logging to a file or upping the verbosity can be useful for troubleshooting

Quick Tip - Marking an HDD as SSD or SSD as HDD in ESXi

by // 7 Comments

This was a neat little trick that I picked up in one of our internal storage email distribution groups which I thought was quite interesting. Some of you may recall an article I wrote a few years back on how to trick ESXi 5 in seeing an SSD device which relied on adding an SATP rule for a particular storage device. The actual use case for this feature was that not all real SSD devices would automatically be detected by ESXi and this allowed a user to manually mark it as an SSD.

The other "non-official" use case for this feature allows a user to basically "simulate" an SSD by marking a regular HDD as an SSD and I this actually helped me test the new Host Cache (Swap-to-SSD) feature which was part of the vSphere 5 release. Recently there was a customer inquiry asking for the complete reverse, in which you could mark an SSD as an HDD. I am not sure what the use case was behind this request but I did learn it was actually possible using a similar method of adding a SATP rule to a device.

Note: If you are running Nested ESXi, a much simpler solution for simulating an SSD is to use the following trick noted here.

Before you begin, you will need to identify the storage device in which you wish to mark as an SSD or HDD. Use the following ESXCLI command to do so:

esxcli storage core device list

In the screenshot above, we can see for our device mpx.vmhba1.C0:T2:L0 shows "Is SSD" parameter as false. After running two commands below, we should then see that property change to true.

Marking HDD as SSD:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d mpx.vmhba1:C0:T2:L0 -o enable_ssd
esxcli storage core claiming reclaim -d mpx.vmhba1:C0:T2:L0

 

Marking SSD as HDD:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d mpx.vmhba1:C0:T1:L0 -o disable_ssd
esxcli storage core claiming reclaim -d mpx.vmhba1:C0:T1:L0

To perform the opposite, you simply just need to add the disable_ssd option. If you receive an error regarding a duplicate rule, you will need to first remove the SATP rule and then re-create with the appropriate option.

Another useful tidbit is that if you are running Nested Virtualization and the virtual disk of that VM is stored on an actual SSD, that virtual disk will automatically show up within the guestOS as an SSD so no additional changes are required.

A Hidden vSphere 5.1 Gem - Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1

by // 17 Comments

Using the new vCenter Log Insight product, you can easily forward application logs from various products within the vCloud Suite for easy analysis and troubleshooting. However, one very important set of logs that we have not been able to collect in the past is the virtual machine logs (vmware.log) which are stored in the working directory of a virtual machine. These logs can be extremely useful from a VMware GSS perspective such as when a virtual machine panics, or if you need to rebuild the .VMX configuration file using these logs or for even general VM auditing purposes.

A recent conversation that I had with Daniel de Sao Jose, who works in our VMware GSS organization reminded of a neat little vSphere 5.1 feature that Daniel had shared with me awhile back. The feature allows you to configure a virtual machine to forward its vmware.log to ESXi's syslog file as well as storing them in the virtual machine's working directory. At the time, there were still a few open questions that required some additional testing and I made a note of this on my ever growing to-do list. I finally around to this and finish up the testing.

UPDATE 1 (04/25/18) - In ESXi 6.7, the ability to forward a VM's vmware.log to an external syslog server also been restored and along with the change, enabling this configuration has been simplified. Instead of having multiple entries to enable the feature and specifying a unique string, you now only have to add a single entry which is vmx.log.syslogID to your VM. The value should be a unique string identifier that the VMX associates with the VM in the syslog. For example, if I use the value of "foo", then the VMX ID will be replaced with "foo" when searching through your syslog entries.

UPDATE 2 (05/04/18) - In ESXi 6.5, 6.5 Update 1 & 6.5 Update 2, the ability to forward a VM's vmware.log to an external syslog server has also been restored and along with the change, enabling this configuration has also been simplified. Simliar to ESXi 6.7, you now only have to add a single entry which is vmx.log.syslogID to your VM. The only difference is that the unique string provided WILL NOT replace the VMX ID in the syslog entry. If you desire the original behavior, you will need to use vSphere 6.7.

To enable this feature, you will need to add the following advanced virtual machine setting:

vmx.log.destination = "syslog-and-disk"

This of course can be enabled using either the vSphere Web Client or vSphere C# Client as well as automated, take a look at this article for more details.

Here is a screenshot showing showing the contents of the vmware.log in the ESXi host's syslog which is located in /var/log/syslog:

Note: The vmware.log is only generated when a virtual machine is powered on.

You also have the option of disabling the local vmware.log from being created in the virtual machine's working directory and only forwarded to ESXi host's syslog. To do so, you would change the advanced virtual machine setting to the following:

vmx.log.destination = "syslog"

By default, the log entries will be identified by the keyword vmx and the specific virtual machine's process ID such as vmx[5313]. However, this is not very user friendly and would still require you to query the VM PID to get the virtual machine name. This can be a challenge if you are viewing the logs from a centralized syslog server such as vCenter Log Insight where you potentially could have logs being forwarded from hundreds if not thousands of ESXi hosts.

To help with this, you can specify the string in which the virtual machine will identify itself when forwarding its logs using the following advanced virtual machine setting:

vmx.log.syslogID = SOME STRING

It made the most sense to me to set this to the name of the virtual machine, so you can easily identify the source of the logs. Here is a screenshot showing the name of the virtual machine instead of the generic "vmx" string.

If you have configured your ESXi host to forward its logs to vCenter Log Insight, you can see how easy it is to view individual virtual machine logs with a click of a button isolating on the syslog source.

One caveat that I would like to mention with this solution is that you are now storing all virtual machine logs in the ESXi hosts syslog file which is also logging other things about the ESXi host. This would cause the local logs to rotate much more frequently on the ESXi host due to the verbosity when powering on and off a virtual machine. This may not be an issue if you are forwarding to a remote syslog server, but ideally it would be nice to have separate log file primarily for the virtual machine logs. In Part 2 of this article, we will take a look at how we can accomplish this by extending ESXi's logger component.