WilliamLam.com

Hidden HA and VPXA Configurations

by // Leave a Comment

Applying the strings method as described in my last article, I decided to also take a look at /opt/vmware/vpxa/vpx/vpxa binary to see if there were anymore hidden goodies. To my surprise, I was able to locate additional HA and VPXA advanced configuration options. While going through and testing some of the HA advanced options, I found that only 19 out of 47 have not been documented and much of the documentation I found online was from Mr. Duncan Epping's blog. This really shows how open Duncan has been around the advanced options with VMware HA, if only VMware as a whole could be so open with the other advanced options that are used throughout VMware but left undocumented.

I do have to stress, these are configurations that are not documented and probably not supported unless directed by VMware. You should be very careful if you decide to play with some of these options and ensure you do not test on a production environment, don't say I did not warn you 🙂

Here is the list of 47 HA Advanced Configurations, the ones with links have been publicly documented by either Duncan or VMware. The rest have not been documented and I have done some testing which will be discussed further.

das.allowNetwork
das.allowVMotionNetworks
das.appMonFailureInterval - #
das.bypassNetCompatCheck
das.checkVmStateDelay - # (default 120)
das.consoleNode - [fqdn]
das.consolePerm - [PERM_USER,PERM_OPER,PERM_ALL]
das.consoleUser - [username]
das.defaultFailoverHost
das.disableUWSwapRequirement
das.failureDetectionInterval
das.failureDetectionTime
das.failureInterval
das.ignoreRedundantNetWarning
das.includeFTcomplianceChecks
das.iostatsInterval
das.isolationAddress
das.isolationShutdownTimeout
das.maxFailures
das.maxFailureWindow
das.maxftvmrestartcount - # (default 5)
das.maxFtVmsPerHost
das.maxvmrestartcount
das.minUptime
das.perHostConcurrentFailoversLimit
das.perHostVMLimit - #
das.powerOffOnIsolation
das.preferredPrimaries
das.primaryCount - [2,3,4,5] (default 5)
das.restartPriority
das.sensorPollingFreq
das.sim.enabled
das.sim.numVms
das.sim.powerOpsPerMinute
das.sim.resourceOpsPerMinute
das.sim.sendInterval
das.slotCpuInMHz
das.slotMemInMB
das.slotNumVcpus
das.trace - [on,off] (default on)
das.traceLevel - [0,1,2,3,4,functiontracing] (default 3)
das.traceOutput - [stdout, eventLog, file] (default file)
das.useDefaultIsolationAddress
das.vmCpuMinMHz
das.vmFailoverEnabled
das.vmMemoryMinMB

You can actually use AAM's (VMware HA) cli utility to view some of these hidden configurations, by using /opt/vmware/aam/bin/Cli

Here is a screenshot of the default VMware HA configuration using the getrule VMWareClusterManager command to list cluster rules, make a note of the parameters with red arrows:

Note: I found it interesting that the "VMWareClusterManager" had an upper case "W" in VMware. It looks like this name was not vetted by VMware marketing 🙂

Next, we add some of the advanced HA configurations using the vSphere Client under Advanced Options (HA) section:

Note: das.primaryCount only supports values 2 through 5, if you try to specify any other value, you will get an error. VMware has selected 5 primary nodes for a specific reason, you should not try to change it from the default.

We reconfigure the cluster with these new HA options and re-run getrule VMWareClusterManager and now we see the values get updated along with some new ones being added:

Note: You can also see some of these changes in /var/log/vmware/aam/aam_config_util_addnode.log

You can also use AAM cli to set some of these configurations:

Usage: setRuleVar|srv
AAM> setRuleVar VMWareClusterManager ft_Trace on

The advanced VPXA configurations applies to the vCenter Agent configuration file located in /etc/opt/vmware/vpxa/vpxa.cfg on an ESX(i) host. There was a total of 47 with only 8 being used in the default vpxa.cfg file.

Here is the list of 47 VPXA Configurations, the ones listed in blue are used by default, the ones with links are documented by VMware and the rest are undocumented:

vpxa.alarm.checkFrequency
vpxa.alarm.powerOnSilence
vpxa.bundleVersion
vpxa.cfg
vpxa.checkAAMPeriod
vpxa.checkNodeResourcePeriod
vpxa.connectTo
vpxa.disableOldQuickStats
vpxa.editionCheckTime
vpxa.ft.cleanupTimeout
vpxa.getchanges.delay
vpxa.getchanges.envbrowserRefreshRate
vpxa.getchanges.timeout
vpxa.healthSystem.throttleInterval
vpxa.heartbeat.interval
vpxa.HostdCnxBypassProxy
vpxa.hostdMonitorPeriod
vpxa.hostIp
vpxa.hostKey
vpxa.hostPort
vpxa.ignoreAamErrorCount
vpxa.ignoreAamErrorTime
vpxa.ignoreAamShutdownCount
vpxa.licenseExpiryNotificationThreshold
vpxa.maxEventReportingDelaySeconds
vpxa.maxFds
vpxa.memoryCheckerTimeInSecs
vpxa.mmapThresholdInKB
vpxa.nodeResourceThreshold
vpxa.partialVmUpdates
vpxa.preserveServerIp
vpxa.respool.changeThreshold
vpxa.respool.changeThresholdInMB
vpxa.respool.enableOverheadLimitTightLoop
vpxa.serverIp
vpxa.serverPort
vpxa.soapAdapterOverNamedPipe
vpxa.soapAdapterPort
vpxa.sslVersion
vpxa.StopMemInMB
vpxa.timeDiffThreshold
vpxa.useSsl
vpxa.vmapTimeoutCountThreshold
vpxa.vmapTimeoutTimeThreshold
vpxa.vmImages
vpxa.vmotion.vmIdAcquireTimeout
vpxa.WarnMemInMB

Note: The VPXA parameters is pretty difficult to reverse engineer and figure out what values each parameter can accept. I did notice the "periods" within each key actually represents the XML node separation within the vpxa.cfg.

Taking vpxa.vmotion.vmIdAcquireTimeout would translate to the following:

<vpxa>
    <vmotion>
         <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
</vpxa>

Again, please use extreme caution if you decide to experiment with these parameters and happy hacking 🙂

1200+ undocumented .vmx parameters

by // 5 Comments

Recently while performing some skunkworks testing in my personal lab, I came across a slew of documented and undocumented virtual machine .vmx configuration parameters. Using one of my favorite UNIX/Linux utility strings, I was able to uncover some interesting things in the /usr/lib/vmware/bin/vmware-vmx binary which is used to load a virtual machines configuration file.

Here are some of the interesting observations I have made:


vSphere is hypervisor aware?

%s: %s detected by CPUID
%s: VMware detected
Microsoft Hyper-V
%s: Xen detected by hypercall
Xen detected but hypervisor unrecognized (Xen variant?)

I noticed the following strings around detecting certain guest hypervisors, is this a hint that VMware is going to support other virtual "hypervisors", specifically Microsoft and Xen?

vSphere to support Mac OSX?

Linux Host
Windows Host
Mac OS Host

There were some text that listed the various types of host, including Mac OSX.

Make sure that you have installed all available Mac OS X software updates.
@&!*@*@(msg.cdrom.darwindisconnect)Your Mac OS guest is using this CD-ROM device. The safest way to disconnect this virtual CD-ROM is by pressing %s, then ej
ecting the media from inside the guest%s. To continue anyway, press %s.%s
@&!*@*@(msg.Backdoor.OsNotMacOSXServer)The guest operating system is not Mac OS X Server.
@&!*@*@(msg.cpuid.darwinWithBTHV)Mac OS X is not supported with software virtualization. Change the execution mode to automatic.
@&!*@*@(msg.cpuid.darwinWithBT)Mac OS X is not supported with software virtualization. To run Mac OS X you need a host on which %s supports hardware virtuali
zation.
isolation.bios.IsGOS.Darwin

There were some text that listed various messages regarding Mac OSX

sbios
vbios
bios440
efi32
efi64
nvram
lsibios
nbios
nxbios
nx3bios
e1000bios
vmibios
vmmmods
sas1068bios
pvscsibios

As you can see, there is mention of EFI support which is required to boot Mac OSX. Does this mean future version of vSphere will support virtualizing Mac OSX?

New guestOS types?

darwin10
darwin10-64
darwin-64
mandrake-64
opensuse
opensuse-64
winServer2008Cluster-32
winServer2008Cluster-64
winServer2008Datacenter-32
winServer2008Datacenter-64
winServer2008DatacenterCore-32
winServer2008DatacenterCore-64
winServer2008Enterprise-32
winServer2008Enterprise-64
winServer2008EnterpriseCore-32
winServer2008EnterpriseCore-64
winServer2008SmallBusiness-32
winServer2008SmallBusiness-64
winServer2008SmallBusinessPremium-32
winServer2008SmallBusinessPremium-64
winServer2008Standard-32
winServer2008Standard-64
winServer2008StandardCore-32
winServer2008StandardCore-64
winServer2008Web-32
winServer2008Web-64
XenVMMXenVMM

There was a section that I came across which listed all supported guestOS types, here you can see there have been a few more that were added between vSphere 4.0 and 4.1. One interesting thing that I am not sure if a lot of people have noticed, is the VirtualMachineGuestOsIdentifier in the vSphere API. This basically provides the guestos identifier that is supported in each release of VI/vSphere. Interesting enough, a darwin guestos support has been documented as of vSphere 4.0:

Though we all know we can not run Mac OSX on ESX(i) ... at least not just yet from what the above is hinting at.

These were just a few of the interesting things I found while parsing through the strings output when looking at the ESX 4.1's vmware-vmx binary.

Here is a collection of over 1200+ documented and undocumented .vmx configuration parameters.

**** These are not documented by VMware, use at your own risk! ****

http://https://s3.amazonaws.com/virtuallyghetto-download/hidden_vmx_params.html

**** These are not documented by VMware, use at your own risk! ****

Some of these hidden .vmx entries have been shared by the VMware and the community, here are just a few:

How to control maximum number of VMware snapshots

by // 21 Comments

There are currently no methods of controlling the number of VMware snapshots using vCenter or ESX(i) permissions today, you either provide the snapshot privilege or you deny it all together. I recently discovered an undocumented .vmx entry that allows you to control the maximum number of VMware snapshots for a given virtual machine. By default, a virtual machine can have a snapshot tree depth of 31, in the worse case scenario supporting up to a maximum of 496 snapshots.

Here is what a VM looks like with 496 snapshots (unexpanded):

 

Note: If you are interested in what this looks like fully expanded, take a look at the screenshot at the very bottom of this post.

If you like to prevent the above or at least control the maximum number of snapshots for a given virtual machine, you can add the following into a VM's .vmx configuration file. Ideally, this is deployed using vSphere API as there is no need to directly edit the VMX's file and this can also be applied to a live running VM (another benefit of using the vSphere API).

Here is an example using PowerCLI:

$vm = Get-VM -Name TestVM
New-AdvancedSetting -Name snapshot.maxSnapshots -Value 1 -Entity $vm

For those that prefer using another vSphere SDK, you just need to use the ReconfigVM_Task() to add the VM Advanced Setting. Please take a look at this sample for here for how to add/update VM Advanced Settings.

snapshot.maxSnapshots = "n"

where n = max number of snapshots and n <= 496

Here is a screenshot of adding this .vmx parameter using the vSphere Client:

The virtual machine above already has one snapshot and per the configuration change, we should not be able to take any additional snapshots:

Next, we will try to take a second snapshot:

As you can see, an error is thrown that we have reached the maximum number of permitted snapshots. If you would like to disable snapshots all together, you can set the value to be 0 and this will prevent anyone from taking snapshots, including administrators.

Here is a an screenshot of the expanded view of a VM with 496 snapshots:

Note: These snapshots were created with a VM running in an vESXi host and script to exhaust the maximum snapshot depth of 31. Each snapshot level was also exhausted with the maximum number of snapshots. Starting from level-1: it was the maximum depth minus 1, level-2: it was maximum depth minus 2, and so fourth. This was just a test to see what the system could handle, you should not try this a home or on a production VM 😉 Use at your own risk