WilliamLam.com

How to Persist Configuration Changes in ESXi 4.x/5.x Part 1

by // 12 Comments

The question of persisting configuration changes in ESXi and the expected behavior using the ESXi Shell is a question that comes up quite frequently. I think this is primarily due to user's experience with the classic ESX Service Console and the expectation that ESXi will behave exactly the same.

First off, any changes made to ESXi 4.x or 5.x using the vSphere Client, any of the vSphere SDK toolkits (vCLI, PowerCLI, etc) or going through vCenter will preserve the changes across system reboots. The caveat is when going directly into the ESXi Shell (formally TSM) and manually editing or adding files to the default ESXi installation, where the persistence of the files will vary depending on what was updated.

So why does editing files in ESXi filesystem not persisted across system reboots? The reason for this is ESXi uses an in-memory filesystem. For more details, check out The Architecture of VMware ESXi whitepaper by VMware. As I mentioned earlier, the persistence of files will depend on what was updated as there are exceptions to the rules.

There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.

UPDATE (08/14/2014) - Take a look at this article for more details regarding the various times an automatic backup is performed by ESXi

The "automatic" backup process occurs in the following scenarios:

1) /sbin/auto-backup.sh runs via cron the first minute of every hour and performs a "find" operation under /etc looking for files that being with ".#":

~ # cat /var/spool/cron/crontabs/root
#syntax : minute hour day month dayofweek command
01 01 * * * /sbin/tmpwatch.sh
01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)

Here is an example of the files backed up in ESXi 4.x:

~ # find etc -follow -type f -name ".#*"
etc/dropbear/.#dropbear_dss_host_key
etc/dropbear/.#dropbear_rsa_host_key
etc/security/.#access.conf
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.crt
etc/vmware/ssl/.#rui.key
etc/vmware/.#dvsdata.db
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/.#hosts
etc/.#inetd.conf
etc/.#rc.local
etc/.#chkconfig.db
etc/.#random-seed
etc/.#resolv.conf
etc/.#syslog.conf
etc/.#shadow
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx

Here is an example of the files backed up in ESXi 5.x:

~ # find etc -follow -type f -name ".#*"
etc/vmware/vpxa/.#vpxa.cfg
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.key
etc/vmware/ssl/.#rui.crt
etc/vmware/.#config
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/vmware/.#dvsdata.db
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile
etc/sfcb/repository/root/config/.#omc_namespaceconfig.idx
etc/sfcb/repository/root/config/.#omc_namespaceconfig
etc/sfcb/repository/root/config/.#omc_config.idx
etc/sfcb/repository/root/config/.#omc_config
etc/sfcb/.#uuid
etc/sfcb/.#sfcb.cfg
etc/ssh/.#ssh_host_rsa_key
etc/ssh/.#ssh_host_rsa_key.pub
etc/ssh/keys-root/.#authorized_keys
etc/ssh/.#ssh_host_dsa_key
etc/ssh/.#ssh_host_dsa_key.pub
etc/vmsyslog.conf.d/.#vmkernel.conf
etc/vmsyslog.conf.d/.#hostd.conf
etc/vmsyslog.conf.d/.#fdm.conf
etc/vmsyslog.conf.d/.#vpxa.conf
etc/security/.#access.conf
etc/.#chkconfig.db
etc/.#hosts
etc/.#ntp.conf
etc/.#random-seed
etc/.#resolv.conf
etc/.#vmsyslog.conf
etc/.#shadow

2) There is also another "internal" backup process that runs every 10 minutes which is not configurable that backups the ESXi filesystem. In the worse case, you may lose changes made in the last 10 minutes if you had a system crash and it reboots.

With the latest release of ESXi 5, ESXi Shell (previous known as unsupported mode and Tech Support Mode) is officially supported for troubleshooting purposes, modifying or adding files is still not officially supported. Though if you need to make changes or add custom files to either ESXi 4.x or 5.x, there are a few options to persist the changes.

Option #1 - If you edit/modify any of the files listed above, these are automatically backed up by ESXi, it's generally recommend you manually run /sbin/auto-backup.sh to ensure they are backed up after your changes.

Option #2 - Edit/Modify files using /etc/rc.local file which executes at the very end of the bootup process. You can make your file changes in this script and they will automatically execute once the system has ran through all the default start up scripts. There two different methods depending on the complexity of your modifications, you may choose to use something like a here-document to create new/modified files or copy configuration files which have been persisted through a local or remote VMFS/NFS datastore.

Here is an example of setting up SSH keys for ESXi and manually creating the .ssh directory and SSH public keys:

Here is an example of copying a script (ghettoVCB.sh) from persistent local datastore to /bin directory:

Once you have made your changes, you will need to manually run /sbin/auto-backup.sh which ensures the changes made to /etc/rc.local will be persisted upon the next system reboot.

Option #2 provides for the most flexibility and allows you to make various configuration changes or addition of files and can easily be integrated into kickstart installation. The entries would be part of your %firstboot stanza to generate the entries in /etc/rc.local file.

In part 2 of this article, I will go over another method that can be leveraged in both ESXi 4.x and 5.x where custom files can be added and persisted as part of the default installation process using kickstart or through a one time manual configuration. I will provide some examples including persisting custom firewall rules in ESXi 5, so stay tuned!

How Fast is the New vSphere 5 HA/DRS on 64 Node Cluster? FAST!

by // 2 Comments

**** Disclaimer: 32 nodes is still the maximum supported configuration for vSphere 5 from VMware, this has not changed. This is purely a demonstration, use at your own risk ****

Recently while catching up on several episodes of the the weekly VMTN Community Podcast, an interesting comment was made by Tom Stephens (Sr. Technical Marketing for vSphere HA) in episode #150 regarding the size of a vSphere cluster. Tom mentioned that there was no "technical" reason a vSphere cluster could not scale beyond 32 nodes. I decided to find out for myself as this was something I had tried with vSphere 4.x and though the configuration of the cluster completed, only 32 hosts were property configured.

Here is a quick video on enabling the new HA (FDM) and DRS on a vSphere 5 cluster with 64 vESXi hosts, you should watch the entire video as it only took an astonishing 2minutes and 37seconds to complete! Hats off to the VMware HA/DRS engineering teams, you can really see the difference in the speed and performance of the new vSphere HA/DRS architecture in vSphere 5.

vSphere 5 - 64 Node Cluster from lamw on Vimeo.

BTW - If someone from VMware is watching this, what does CSI stand for? I believe this was the codename for what is now known as FDM

New Performance Metrics In vSphere 5

by // 2 Comments

I recently had to look at some performance metrics in my vSphere 5 lab and I was curious if VMware had documented all the new performance metrics. I headed over to the vSphere 5 API reference guide and to my surprise, they were exactly the same as the vSphere 4 API reference guide. Though looking at the vSphere Client, it was obvious there were new performance metrics for features such as Storage DRS that did not exists in vSphere 4.

Using a similar method in a previous post about Power performance metrics, I extracted all the new metrics in vSphere 5 and created the following table that includes the metric name (rollup,units and internal name), collection level and description of the metric. There are a total of 129 new performance metrics that include Storage DRS and HBR (Host Based Replication).

Hopefully this will be fixed in the API documentation when vSphere 5 GA's as I recalled providing the same feedback during the beta program. 

Metric Stat Level Description
cpu
average.MHz.capacity.provisioned 3 Capacity in MHz of the physical CPU cores
average.MHz.capacity.entitlement 1 CPU resources devoted by the ESX scheduler to virtual machines and resource pools
average.MHz.capacity.usage 3 CPU usage in MHz during the interval
average.MHz.capacity.demand 2 The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
average.percent.capacity.contention 2 Percent of time the VMs on this host are unable to run because they are contending for access to the physical CPU(s)
average.number.corecount.provisioned 2 The number of physical cores provisioned to the entity
average.number.corecount.usage 2 The number of virtual processors running on the host
average.percent.corecount.contention 1 Time the VM is ready to run, but is unable to run due to co-scheduling constraints
average.MHz.capacity.demand 2 The amount of CPU resources VMs on this host would use if there were no CPU contention or CPU limit
average.percent.latency 2 Percent of time the VM is unable to run because it is contending for access to the physical CPU(s)
latest.MHz.entitlement 2 CPU resources devoted by the ESX scheduler
average.MHz.demand 2 The amount of CPU resources a VM would use if there were no CPU contention or CPU limit
summation.millisecond.costop 2 Time the VM is ready to run, but is unable to due to co-scheduling constraints
summation.millisecond.maxlimited 2 Time the VM is ready to run, but is not run due to maxing out its CPU limit setting
summation.millisecond.overlap 3 Time the VM was interrupted to perform system services on behalf of that VM or other VMs
summation.millisecond.run 2 Time the VM is scheduled to run
datastore
latest.millisecond.maxTotalLatency 3 Highest latency value across all datastores used by the host
average.KBps.throughput.usage 2 usage
average.millisecond.throughput.contention 2 contention
summation.number.busResets 2 busResets
summation.number.commandsAborted 2 commandsAborted
summation.number.commandsAborted 2 commandsAborted
summation.number.busResets 2 busResets
latest.number.datastoreReadBytes 2 Storage DRS datastore bytes read
latest.number.datastoreWriteBytes 2 Storage DRS datastore bytes written
latest.number.datastoreReadIops 1 Storage DRS datastore read I/O rate
latest.number.datastoreWriteIops 1 Storage DRS datastore write I/O rate
latest.number.datastoreReadOIO 1 Storage DRS datastore outstanding read requests
latest.number.datastoreWriteOIO 1 Storage DRS datastore outstanding write requests
latest.number.datastoreNormalReadLatency 2 Storage DRS datastore normalized read latency
latest.number.datastoreNormalWriteLatency 2 Storage DRS datastore normalized write latency
latest.number.datastoreReadLoadMetric 4 Storage DRS datastore metric for read workload model
latest.number.datastoreWriteLoadMetric 4 Storage DRS datastore metric for write workload model
latest.number.datastoreMaxQueueDepth 1 Storage I/O Control datastore maximum queue depth
disk
average.KBps.throughput.usage 3 Aggregated disk I/O rate
average.millisecond.throughput.contention 3 Average amount of time for an I/O operation to complete
summation.number.scsiReservationConflicts 2 Number of SCSI reservation conflicts for the LUN during the collection interval
average.percent.scsiReservationCnflctsPct 2 Number of SCSI reservation conflicts for the LUN as a percent of total commands during the collection interval
average.kiloBytes.capacity.provisioned 3 provisioned
average.kiloBytes.capacity.usage 2 usage
average.percent.capacity.contention 1 contention
hbr
average.number.hbrNumVms 4 Current Number of Replicated VMs
average.KBps.hbrNetRx 4 Average amount of data received per second
average.KBps.hbrNetTx 4 Average amount of data transmitted per second
managementAgent
average.MHz.cpuUsage 3 Amount of Service Console CPU usage
mem
average.kiloBytes.capacity.provisioned 3 Total amount of memory configured for the VM
average.kiloBytes.capacity.entitlement 1 Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
average.kiloBytes.capacity.usable 2 Amount of physical memory available for use by virtual machines on this host
average.kiloBytes.capacity.usage 1 Amount of physical memory actively used
average.percent.capacity.contention 2 Percentage of time the VM is waiting to access swapped, compressed, or ballooned memory
average.kiloBytes.capacity.usage.vm 2 vm
average.kiloBytes.capacity.usage.vmOvrhd 2 vmOvrhd
average.kiloBytes.capacity.usage.vmkOvrhd 2 vmkOvrhd
average.kiloBytes.capacity.usage.userworld 2 userworld
average.kiloBytes.reservedCapacity.vm 2 vm
average.kiloBytes.reservedCapacity.vmOvhd 2 vmOvhd
average.kiloBytes.reservedCapacity.vmkOvrhd 2 vmkOvrhd
average.kiloBytes.reservedCapacity.userworld 2 userworld
average.percent.reservedCapacityPct 3 Percent of memory that has been reserved either through VMkernel use, by userworlds, or due to VM memory reservations
average.kiloBytes.consumed.vms 2 Amount of physical memory consumed by VMs on this host
average.kiloBytes.consumed.userworlds 2 Amount of physical memory consumed by userworlds on this host
average.percent.latency 2 Percentage of time the VM is waiting to access swapped or compressed memory
average.kiloBytes.entitlement 2 Amount of host physical memory the VM is entitled to, as determined by the ESX scheduler
average.kiloBytes.lowfreethreshold 2 Threshold of free host physical memory below which ESX will begin reclaiming memory from VMs through ballooning and swapping
none.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
average.KBps.llSwapInRate 2 Rate at which memory is being swapped from host cache into active memory
average.KBps.llSwapOutRate 2 Rate at which memory is being swapped from active memory to host cache
average.kiloBytes.overheadTouched 4 Actively touched overhead memory (KB) reserved for use as the virtualization overhead for the VM
average.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
maximum.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
minimum.kiloBytes.llSwapUsed 4 Space used for caching swapped pages in the host cache
none.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
average.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
maximum.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
minimum.kiloBytes.llSwapIn 4 Amount of memory swapped-in from host cache
none.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
average.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
maximum.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
minimum.kiloBytes.llSwapOut 4 Amount of memory swapped-out to host cache
net
average.KBps.throughput.provisioned 2 Provisioned pNic I/O Throughput
average.KBps.throughput.usable 2 Usable pNic I/O Throughput
average.KBps.throughput.usage 3 Average vNic I/O rate
summation.number.throughput.contention 2 Count of vNic packet drops
average.number.throughput.packetsPerSec 2 Average rate of packets received and transmitted per second
average.KBps.throughput.usage.vm 3 Average pNic I/O rate for VMs
average.KBps.throughput.usage.nfs 3 Average pNic I/O rate for NFS
average.KBps.throughput.usage.vmotion 3 Average pNic I/O rate for vMotion
average.KBps.throughput.usage.ft 3 Average pNic I/O rate for FT
average.KBps.throughput.usage.iscsi 3 Average pNic I/O rate for iSCSI
average.KBps.throughput.usage.hbr 3 Average pNic I/O rate for HBR
average.KBps.bytesRx 2 Average amount of data received per second
average.KBps.bytesTx 2 Average amount of data transmitted per second
summation.number.broadcastRx 2 Number of broadcast packets received during the sampling interval
summation.number.broadcastTx 2 Number of broadcast packets transmitted during the sampling interval
summation.number.multicastRx 2 Number of multicast packets received during the sampling interval
summation.number.multicastTx 2 Number of multicast packets transmitted during the sampling interval
summation.number.errorsRx 2 Number of packets with errors received during the sampling interval
summation.number.errorsTx 2 Number of packets with errors transmitted during the sampling interval
summation.number.unknownProtos 2 Number of frames with unknown protocol received during the sampling interval
power
summation.joule.energy 3 Total energy used since last stats reset
average.percent.capacity.usagePct 3 Current power usage as a percentage of maximum allowed power
average.watt.capacity.usable 2 Current maximum allowed power usage
average.watt.capacity.usage 2 Current power usage
storageAdapter
latest.millisecond.maxTotalLatency 3 Highest latency value across all storage adapters used by the host
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
average.percent.OIOsPct 3 The percent of I/Os that have been issued but have not yet completed
average.number.outstandingIOs 2 The number of I/Os that have been issued but have not yet completed
average.number.queued 2 The current number of I/Os that are waiting to be issued
average.number.queueDepth 2 The maximum number of I/Os that can be outstanding at a given time
average.millisecond.queueLatency 2 Average amount of time spent in the VMkernel queue, per SCSI command, during the collection interval
average.KBps.throughput.usage 4 The storage adapter's I/O rate
storagePath
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
latest.millisecond.maxTotalLatency 3 Highest latency value across all storage paths used by the host
summation.number.busResets 2 Number of SCSI-bus reset commands issued during the collection interval
summation.number.commandsAborted 2 Number of SCSI commands aborted during the collection interval
average.KBps.throughput.usage 2 Storage path I/O rate
sys
latest.second.osUptime 4 Total time elapsed, in seconds, since last operating system boot-up
vcResources
average.kiloBytes.buffersz 4 buffersz
average.kiloBytes.cachesz 4 cachesz
average.number.diskreadsectorrate 4 diskreadsectorrate
average.number.diskwritesectorrate 4 diskwritesectorrate
virtualDisk
average.millisecond.throughput.cont 2 Average amount of time for an I/O operation to complete
average.KBps.throughput.usage 2 Virtual disk I/O rate
summation.number.commandsAborted 2 commandsAborted
summation.number.busResets 2 busResets
latest.number.readOIO 2 Average number of outstanding read requests to the virtual disk during the collection interval
latest.number.writeOIO 2 Average number of outstanding write requests to the virtual disk during the collection interval
latest.number.readLoadMetric 2 Storage DRS virtual disk metric for the read workload model
latest.number.writeLoadMetric 2 Storage DRS virtual disk metric for the write workload model