WilliamLam.com

New way of enabling and disabling services using vSphere 4.1

by 2 Comments

While checking out the PlanetV12n feed, I noticed a new video from David Davis about the new vSphere 4.1 Tech Support Mode. In the short video, David goes over the new method of enabling "hidden" unsupported Busybox Console, also known as Tech Support Mode. In the past, you had to be on the console of your ESXi host, type ALT+F1, and then "unsupported" to gain access. Once in, to enable remote SSH access or Remote Tech Support Mode, you had to edit /etc/inetd.conf and restart inetd service. This was pretty tedious if you needed access for a short period of time. In the video, David goes over the new method showing how it can be done using the DCUI and the old method is no longer required.

What surprised me after watching the video was that he did not mention the other method of enabling and disabling Tech Support Mode both local and remote. One issue I had with past releases of ESXi is that you could restart some services such as ntp or vmware-vpxa via the vSphere API, but others were just not available. In vSphere 4.1, VMware introduces a few new services around their Likewise Active Directory integration but also includes controlling both local and remote Tech Support Mode as well as DCUI itself.

These services can be enabled and disabled using the vSphere Client, here is a screenshot:

To enable or disable TSM, just click on the service and then click on options:

You will then have the option to configure the startup policy including enabling or disabling the service:

If you needed to perform this operation against one or two host, it is not that big of a deal. Though if you needed to enable remote Tech Support Mode (SSH access) across few dozen hosts, then can still be tedious. Luckily I wrote a script (hostServiceMangement.pl) last year that allowed you to enable and disable supported services using the vSphere API. Without any modifications, it supports vSphere 4.1 and can take advantage of the new services that are available for control.

Here is an example of listing the services on an ESXi 4.1 host:

Here is an example enabling remote Tech Support Mode:

Here is an example of disabling remote TSM:

The script can be executed on a host that has vCLI 4.1 installed or on vMA 4.1 and can bulk update a list of ESX/ESXi host or individual host. For more details, please check out the documentation for hostServiceManagement.pl.

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 2

by 1 Comment

Continuing from Part1 of new vSphere 4.1 CLI Utilities, here are few more:

1. vmkload_app64 is a 64bit version of the VMkernel application loader.

[[email protected] bin]# /usr/lib/vmware/bin/vmkload_app64 -h

Usage: vmkload_app [options]
--nocore|-n Disable core dump for this world
--enableDebug|-d Enable userworld debug wait after core
--break|-b [wid] Break a userworld into the debugger
(only specify world id when not creating
or connecting to a userworld)
--kill|-k Send signal 'signum' to a running userworld
(only works with vmx worlds)
--ouputCartelID|-o Output cartel ID and other info to
--setsid|-S Run vmkload_app in a new session
--workingdir|-w Use as userworld working directory (defaults to current)
--env|-v Add environment variable to
userWorldApp's environment
--ipSocketType|-i Set the type for IPv4 sockets, where
is one of 'proxied', 'vmktcp', or 'costcp'
--sched.group= Set scheduler group
--sched.cpu.units= Set units (mhz or pct) for cpu rates
--sched.cpu.min= Set minimum cpu rate
--sched.cpu.max= Set maximum cpu rate
--sched.cpu.minLimit= Set upper bound for minimum cpu rate
--sched.cpu.shares= Set cpu share allocation
--sched.cpu.affinity= Set affinity towards pcpus
--sched.mem.min= Set minimum memory allocation (MB)
--sched.mem.max= Set maximum memory allocation (MB)
--sched.mem.minLimit= Set upper bound for minimum memory growth (MB)
--sched.mem.shares= Set memory share allocation
--sched.swap.scope= Set swap scope (none, private, system)
--sched.swap.dir=[dir] Set directory for swap file
--sched.swap.file= Set swap file name
--sched.memSizeLimit= Set upper bound on address-space size (MB)
--help|-h This message

2. vmware-usbarbitrator is a utility for allowing VMs to connect to the host's USB devices.
[[email protected] bin]# /usr/lib/vmware/bin/vmware-usbarbitrator

3. vprobed looks to be a utility for running the vProbe daemon.

[[email protected] bin]# /usr/lib/vmware/bin/vprobed -h

Error: vprobed takes no arguments.
4. vmkiscsiadm was a pretty well known tool used to configure and troubleshoot iSCSI on ESX. However, it has been removed with the release of vSphere 4.1.

5. vmkiscsi-tool has been updated with some new options:

[[email protected] ~]# /usr/sbin/vmkiscsi-tool -h
vmkiscsi-tool -h --help
-A --Authentication
-C --connection
-D --discovery
-E --session
-I --iSCSIname
-L --Lun
-M --MTU
-N --Network: network properties
-O --Associate: Associate/Disassociate route.
-P --Phba
-R --discoveryStatus : Print discovery status.
-S --static: Static Discovery Targets
-T --Target
-U --Route: Route table.
-V --Nic
-W --Parameter
-X --Reset
-Y --SetNicList
-c --ipconfig: enable/disable DHCP, ARP redirect
-d --dnsserver
-e --ethernet: Link Status
-g --gateway
-i --ipAddress
-k --Alias
-n --iSNS
-p --Pnp: Physical Network Portal properties (pnic)
-q --Lnp: Logical Network Portal properties (vmknic)
-s --subnetmask
-v --version
Subcommands
-l --list
-r --remove
-a --add
-m --authMethod : specify method for add/remove
-b --mutual CHAP
-j --persist changes
-y --per dynamic discovery
-x --per static discovery
-z --per network portal (binded vmknic)
-t --per target
-u --per isid(session)
-f --flag: set a discovery or authentication flag
adapterName
Combine -l with an option to display the current information.

6. vmkiscsi-test is a new iSCSI utility to test various components and provides a summary at the end with the checks that ran, failed and passed:

[[email protected] ~]# /usr/sbin/vmkiscsi-test

CUnit - A Unit testing framework for C - Version 1.1-1
http://cunit.sourceforge.net/

Suite: general - General Tests
Test: IMA_NullParameters ... passed
Test: IMA_VMW_NullParameters ... passed
Test: IMA_BadOids ... passed
Test: IMA_VMW_BadOids ... passed
Suite: Info - Library and Plugin Tests
Test: IMA_GetLibraryProperties ... passed
Test: IMA_GetPluginOidList ... passed
Test: IMA_GetPluginProperties ... passed
Suite: Adapter - Adapter Tests
Test: IMA_GetPhbaOidList ... passed
Test: IMA_GetPhbaProperties ... passed
Test: IMA_VMW_GetPhbaProperties ... passed
Test: IMA_GetLhbaOidList ... passed
Test: IMA_VMW_GetLhbaOidList ... passed
Test: IMA_GetLhbaProperties ... passed
Test: IMA_VMW_GetLhbaProperties ... passed
Suite: Portals - Portals Tests
Test: IMA_GetNetworkPortalOidList ... passed
Test: IMA_VMW_GetNetworkPortalOidList ... passed
Test: IMA_GetPnpOidList ... passed
Suite: Discovery - Discovery Tests
Test: IMA_GetDiscoveryProperties ... passed
Test: IMA_SetIsnsDiscovery ... passed
Test: IMA_SetSlpDiscovery ... passed
Test: IMA_SetStaticDiscovery ... passed
Test: IMA_SetSendTargetsDiscovery ... passed
Test: Static Discovery ... passed
Test: Dynamic Discovery ... passed
Suite: Properties - Standard Properties Tests
Test: IMA_GetDataPduInOrderProperties ... passed
Test: IMA_GetDataSequenceInOrderProperties ... passed
Test: IMA_GetDefaultTime2RetainProperties ... passed
Test: IMA_GetDefaultTime2WaitProperties ... passed
Test: IMA_GetErrorRecoveryLevelProperties ... passed
Test: IMA_GetFirstBurstLengthProperties ... passed
Test: IMA_GetIFMarkerProperties ... passed
Test: IMA_GetImmediateDataProperties ... passed
Test: IMA_GetInitialR2TProperties ... passed
Test: IMA_GetMaxBurstLengthProperties ... passed
Test: IMA_GetMaxConnectionsProperties ... passed
Test: IMA_GetMaxOutstandingR2TProperties ... passed
Test: IMA_GetMaxRecvDataSegmentLengthProperties ... passed
Test: IMA_GetOFMarkerProperties ... passed
Suite: VMW Properties - VMware Properties Tests
Test: IMA_VMW_ArpRedirectProperties ... passed
Test: IMA_VMW_MtuProperties ... passed
Suite: Auth - Auth Tests
Test: IMA_GetInUseInitiatorAuthMethods ... passed
Test: IMA_VMW_GetInUseInitiatorLocalAuthMethods ... passed
Test: IMA_SetInitiatorAuthMethods ... passed
Test: IMA_VMW_SetInitiatorLocalAuthMethods ... passed
Test: IMA_GetInitiatorAuthParms ... passed
Test: IMA_VMW_GetInitiatorAuthParms ... passed
Test: IMA_SetInitiatorAuthParms ... passed
Test: IMA_VMW_SetInitiatorAuthParms ... passed
Test: IMA_VMW_GetMutualAuthParms ... passed
Test: IMA_VMW_SetMutualAuthParms ... passed
Test: IMA_GetLhbaMutualAuthParmsList ... passed
Test: IMA_GetMutualLocalAuthParms ... passed
Test: IMA_GetMutualLocalAuth ... passed
Test: IMA_RemoveLhbaMutualAuthParms ... passed
Test: IMA_AddLhbaMutualAuthParms ... passed

--Run Summary: Type Total Ran Passed Failed
suites 8 8 n/a 0
tests 55 55 55 0
asserts 495 495 495 0
Asserts Skipped: 0
Tests Skipped: 0
Ignore Flags:

7. vmfs-support is a new shell script that uses vmkfstools -D and recursively dumps information about all files given a VMFS folder or file (This utility is only found on ESXi 4.1).

~ # ash /sbin/vmfs-support
Usage: vmfs-support /

~ # ash /sbin/vmfs-support /vmfs/volumes/iSCSI-1/

vmkfstools -D /vmfs/volumes/iSCSI-1/.fbb.sf
Lock [type 10c00001 offset 4294656 v 3, hb offset 3866624
gen 73, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2656]
Addr <4, 0, 1>, gen 1, links 1, type sys, flags 0, uid 0, gid 0, mode 400
len 98304, nb 1 tbz 0, cow 0, zla 1, bs 1048576
vmkfstools -D /vmfs/volumes/iSCSI-1/.fdc.sf
Lock [type 10c00001 offset 4296704 v 1, hb offset 0
gen 0, mode 0, owner 00000000-00000000-0000-000000000000 mtime 16020]
Addr <4, 0, 2>, gen 1, links 1, type sys, flags 0, uid 0, gid 0, mode 400
len 31129600, nb 30 tbz 0, cow 0, zla 1, bs 1048576
vmkfstools -D /vmfs/volumes/iSCSI-1/.pbc.sf
.....
.....
.....

8.  The well known nc utility has been added to unsupported Busybox console of ESXi 4.1.

~ # /bin/nc
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port]
[-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version]
[-x proxy_address[:port]] [hostname] [port[s]]

9. vdu is a new utility that provides disk utilization information, similar to that of UNIX/Linux du (This utility is only found on ESXi 4.1).

~ # /bin/vdu
For './bin':
tardisk SYS1: 28694376 ( 47 inodes)
tardisk SYS2: 135168 ( 18 inodes)
heap : 208 ( 16 inodes)
For './etc':
tardisk SYS1: 4630016 ( 220 inodes)
heap : 65184 ( 35 inodes)
tardisk SYS2: 10790 ( 22 inodes)
tardisk vpxa.vgz: 6144 ( 5 inodes)
tardisk aam.vgz: 4096 ( 2 inodes)
ramdisk MAINSYS: 65536 ( 4 inodes)
For './lib':
tardisk SYS1: 41208050 ( 146 inodes)
ramdisk MAINSYS: 16384 ( 1 inodes)
tardisk SYS2: 29168640 ( 95 inodes)
For './lib64':
tardisk SYS1: 6955539 ( 23 inodes)
For './opt':
tardisk SYS1: 0 ( 1 inodes)
tardisk vpxa.vgz: 44070912 ( 24 inodes)
tardisk aam.vgz: 12457984 ( 188 inodes)
For './productLocker':
tardisk SYS1: 24
For './sbin':
tardisk SYS1: 87665431 ( 145 inodes)
heap : 8192 ( 4 inodes)
ramdisk MAINSYS: 106496 ( 10 inodes)
tardisk SYS2: 151552 ( 3 inodes)
For './tmp':
heap : 34816 ( 33 inodes)
ramdisk updatestg: 8192 ( 1 inodes)
For './usr':
tardisk SYS1: 36028127 ( 381 inodes)
tardisk SYS2: 252928 ( 39 inodes)
ramdisk MAINSYS: 53248 ( 6 inodes)
heap : 23 ( 1 inodes)
For './var':
tardisk SYS1: 2137 ( 29 inodes)
tardisk SYS2: 28048896 ( 51 inodes)
ramdisk MAINSYS: 3850240 ( 56 inodes)
heap : 63758 ( 58 inodes)
ramdisk hostdstats: 2023424 ( 2 inodes)
tardisk vpxa.vgz: 0 ( 2 inodes)
For './vmfs':
tardisk SYS1: 6 ( 2 inodes)
For './vmimages':
tardisk SYS1: 47 ( 3 inodes)
For './vmupgrade':
tardisk SYS1: 19
For './.ssh':
heap : 2048 ( 2 inodes)
ramdisk MAINSYS: 8192 ( 1 inodes)
For './.rnd':
heap : 2048
For './bootbank':
heap : 50
For './altbootbank':
heap : 50
For './store':
heap : 50
For './scratch':
heap : 50
For './locker':
heap : 7
For '.':
heap : 176484 ( 156 inodes)
tardisk SYS1: 205183772 ( 999 inodes)
tardisk SYS2: 57767974 ( 228 inodes)
tardisk vpxa.vgz: 44077056 ( 31 inodes)
tardisk aam.vgz: 12462080 ( 190 inodes)
ramdisk MAINSYS: 4100096 ( 78 inodes)
ramdisk updatestg: 8192 ( 1 inodes)
ramdisk hostdstats: 2023424 ( 2 inodes)

vSphere 4.1 was not always 4.1?

by Leave a Comment

During the BETA of vSphere 4.1, there were some inconsistencies in some of the build numbers which was eventually resolved. It looks like something was still missed in the final RTM release of vSphere 4.1 ....

Here is a screenshot of the python libraries on classic ESX 4.1 used for esxupdate. At some point, it looks like this release was going to be named vSphere 4.5 as most people had anticipated, but then was revised to 4.1.

I'm sure there will be few more easter eggs, happy hunting 🙂

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 3

by Leave a Comment

Following part1 and part2 of new vSphere 4.1 CLI Utilities, here are some new vimsh commands:

For ESX use /usr/bin/vmware-vim-cmd
For ESXi use /bin/vim-cmd

1. vmsvc/power.suspendResume is used for vMotion and sVMotion tasks before switching over to the new VM.

[[email protected] ~]# vmware-vim-cmd vmsvc/power.suspendResume
Insufficient arguments.
Usage: power.suspendResume vmid

Suspend & resume the specified virtual machine.

2. vmsvc/queryftcompat allows you to query a given Virtual Machine to check for Fault Tolerance compatibility.

[[email protected] ~]# vmware-vim-cmd vmsvc/queryftcompat
Insufficient arguments.
Usage: queryftcompat vmid

Query FT compatibility for a VM.

3. vimsvc/auth/lockdown_is_enabled allows you to query if the host has the lockdown feature enabled, this is only applicable to ESXi.

~ # vim-cmd vimsvc/auth/lockdown_is_enabled
false

4. vimsvc/auth/lockdown_is_possible allows you to check if you can put the host into lockdown mode.

~ # vim-cmd vimsvc/auth/lockdown_is_possible
true

5. vimsvc/auth/lockdown_mode_enter allows you to enter lockdown mode on an ESXi host.

~ # vim-cmd vimsvc/auth/lockdown_mode_enter

6. vimsvc/auth/lockdown_mode_exit allows you to exit lockdown mode on an ESXi host.

~ # vim-cmd vimsvc/auth/lockdown_mode_exit

The next two commands refer to the following services which can also be configured using the vSphere Client:

7. hostsvc/start_local_tsm allows you to enable Local Tech Support Mode on an ESXi host.

~ # vim-cmd hostsvc/start_local_tsm

8. hostsvc/start_remote_tsm  allows you to enable Remote Tech Support Mode (SSH access) on an ESXi host.

~ # vim-cmd hostsvc/start_remote_tsm

9. hostsvc/stop_local_tsm allows you to disable Local Tech Support Mode on an ESXi host.

~ # vim-cmd hostsvc/stop_local_tsm

10. hostsvc/stop_remote_tsm allows you to disable Remote Tech Support Mode (SSH access) on an ESXi host.

~ # vim-cmd hostsvc/stop_remote_tsm

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 1

by Leave a Comment

With the new release of vSphere 4.1, there are new additions to the CLI utilities that administrators can leverage for configurations and troubleshooting. Although, not all were treated equally from an announcement and documentation standpoint. 


Here are some of the new and/or subtle changes with the CLI utilities:

1. vmkfstools undocumented "-D" option now outputs to console along with an entry in /var/log/vmkernel, whereas in the past, to locate the entry within the vmkernel logs for identifying locked files by a particular host as documented in this article. 

Here is an example of the old version of vmkfstools and -D option:

Here is an example of the new version of vmkfstools and -D option:

 2. storageRM is a debugging utility for Storage I/O Control at the host level.


[[email protected] ~]# /usr/lib/vmware/bin/storageRM -h

Default Values:
SleepTime: 4000
Threshold: 35
Gamma: 0.25
Beta-per-host: 4.00
LowerBound: 4
UpperBound: 64
Storage I/O Control-- This tool does flow control at the host
in order to maintain disk I/O latency close to a
threshold and queue sizes converge at values
proportional to the beta parameter.

The following histogram related options are available:
-a, --print the list of all luns, their latency threshold,
queue depths and if Storage I/O Controlis set/unset
-b, --Beta per host value
-d - put debugging info in the given file
-f, --force the run without checking version or checksum
-g, --defGamma value for use in control equation
-h, --help will print the usage
-l, --lower bound on the length of lun queue, (default 4)
-n, --no anomaly detection is done
-r, --reset issue queue for all luns to default
-s, --sleep time in ms for periodic execution
-t, --threshold on the latency (in ms), for rate control
-u, --upper bound on the length of lun queue (default 64)
-v, --debug log level value
storageRM Usage:
storageRM [options]

3. net-lbt is a debugging utility for the new Load-Based Teaming feature.

[[email protected] ~]# /usr/lib/vmware/bin/net-lbt -h

Usage: [-d] [-t time] [-v] [-s threshold]
-d run in daemon mode
-t daemon sleep period in seconds, minimum 10 seconds
-v run with verbose logging
-s saturate threshold [10, 100], i.e. 60 for 60% of line rate

4. net-dvs is a debugging utility for Distributed vSwitch. 

[[email protected] ~]# /usr/lib/vmware/bin/net-dvs -h

Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -T
net-dvs -v "vlanID[;t|p[0-7][;min-max,min-max...]]
net-dvs -V "primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p..."
net-dvs -m "sid;dname;snaplen;

[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,...;srcInPort1,srcInport2,...;srcOutPort1,srcOutPort2,...;:sid2;dname2..."
net-dvs dvswitch -k "respool1_id;respool2_id;..."
net-dvs dvswitch -p dvport -K "respool1_id:shares:limit;respool2_id:shares:limit;..."
net-dvs dvswitch -p dvport -z "respool_id"
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,...] -t team_policy_type -p port switch_name
net-dvs dvswitch -H "red|yellow|green:some message" switch_name
net-dvs -o "depth,param|classname;depth,param|classname;... -p port|globalPropList switch_name
net-dvs --mtu mtu_value [-p dvport] switch_name
net-dvs --x 0|1 -p dvport switch_name
net-dvs --vlan vlanID -p dvport switch_name
net-dvs --reset -p dvport switch_name
net-dvs --cap cap_value -p dvport switch_name
net-dvs --states -p dvport switch_name

5. remoteDeviceConnect is a new utility that allows you to mount various remote devices including floppy and USB. 

/usr/lib/vmware/bin/remoteDeviceConnect: option requires an argument -- h

VMware remote Device Connect
Usage: /usr/lib/vmware/bin/remoteDeviceConnect OPTIONS filename
Options:
-h : hostname (localhost)
-p : Port to connect to (63079, 902 for authd)
-t : (Req.) cd-raw, cd-iso, cd-normal, cd-raw-ex, floppy
-d : (Req.) Device node to connect to (floppy0, ideX:Y)
-n : the VM's floppy drive number
-f : fileType
-A : use Authd to connect
-U : username for authd (your username)
-V : the VM to use. (NULL)
-P : password.
Examples:
remoteDeviceConnect -t floppy -d floppy0 -f device /dev/fd0
remoteDeviceConnect -t floppy -d floppy0 -f file image.flp
remoteDeviceConnect -t usb "path:2/1 vid:0x0547 pid:0x2131"

6. sensorD looks to be a debugging utility that can connect to an ipmi device.

[[email protected] bin]# /usr/lib/vmware/bin/sensord

sensord: failed to open ipmi device: No such file or dir
sensord: unsupported hardware

7. statedumper looks to be a debugging utility for output information about the system and its states.

[[email protected] bin]# /usr/lib/vmware/bin/statedumper -h

statedumper [-f filename] [-s off] [-e off] [-b] [-o] [-r] [-x]

The following options are supported:
-e - end dump at offset
-f - use filename rather than the default state.log
-o - output entry offsets
-r - output all registers, output 64 bits with -r64
-s - start dumping at offset
-b - filter on branch count, use -s and -e for start/end
-x - dump extra debug data

8. vmkeventd looks to be a utility for capturing VMkernel events

[[email protected] bin]# /usr/lib/vmware/bin/vmkeventd -h

/usr/lib/vmware/bin/vmkeventd: invalid option -- h
Usage: /usr/lib/vmware/bin/vmkeventd [-d]

9. analyze-esx-init-boot.py looks like a debugging utility to analyze the COS boot up logs

[[email protected] ~]# /usr/sbin/analyze-esx-init-boot.py -s -S /vmfs/volumes/datastore1/esxconsole-4c27dd75-a38d-5044-5670-005056927558/logs/sysboot.log -V /var/log/messages
ERROR: Could not find 'cpu 0: early measured tsc speed' in the log
This is one of the very first log messages after boot
POST boot times will not be relevant
Unable to find vsish. Please ensure that you have debugging tools installed and your PATH is correctly setup.
VMKernel: 0.0000 secs
POST Tests: 0.0000 secs
Init Scripts: 0.0000 secs

ESX Boot Time: 0.0000 secs
Hardware/BIOS: 0.0000 secs

Total Boot Time: 0.0000 secs

Serial port output was on