Continuing from part 1 of How to Persist Configuration Changes in ESXi 4.x/5.x Part 1, here is another method which I prefer when trying to persist configuration changes in ESXi. When ESXi boots up, it loads it's filesystem into memory which the modules to be loaded up are determined by the configuration found in /bootbank/boot.cfg and /altbootbank/boot.cfg for the two respective partitions (primary / backup).

UPDATE: You can now persist configuration files such as firewall rules and others using the new VIB Author Fling, please take a look at this article for more details.

Here is an example of ESXi 4.x default boot.cfg:

~ # cat /bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz
build=4.1.0-348481
updated=1
bootstate=0

Here is an example of ESXi 5.x default boot.cfg:

~ # cat /bootbank/boot.cfg
bootstate=0
kernel=tboot.b00
title=Loading VMware ESXi
kernelopt=no-auto-partition
modules=b.b00 --- useropts.gz --- k.b00 --- a.b00 --- ata-pata.v00 --- ata-pata.v01 --- ata-pata.v02 --- ata-pata.v03 --- ata-pata.v04 --- ata-pata.v05 --- ata-pata.v06 --- ata-pata.v07 --- block-cc.v00 --- ehci-ehc.v00 --- s.v00 --- ima-qla4.v00 --- ipmi-ipm.v00 --- ipmi-ipm.v01 --- ipmi-ipm.v02 --- misc-cni.v00 --- misc-dri.v00 --- net-be2n.v00 --- net-bnx2.v00 --- net-bnx2.v01 --- net-cnic.v00 --- net-e100.v00 --- net-e100.v01 --- net-enic.v00 --- net-forc.v00 --- net-igb.v00 --- net-ixgb.v00 --- net-nx-n.v00 --- net-r816.v00 --- net-r816.v01 --- net-s2io.v00 --- net-sky2.v00 --- net-tg3.v00 --- ohci-usb.v00 --- sata-ahc.v00 --- sata-ata.v00 --- sata-sat.v00 --- sata-sat.v01 --- sata-sat.v02 --- sata-sat.v03 --- scsi-aac.v00 --- scsi-adp.v00 --- scsi-aic.v00 --- scsi-bnx.v00 --- scsi-fni.v00 --- scsi-hps.v00 --- scsi-ips.v00 --- scsi-lpf.v00 --- scsi-meg.v00 --- scsi-meg.v01 --- scsi-meg.v02 --- scsi-mpt.v00 --- scsi-mpt.v01 --- scsi-mpt.v02 --- scsi-qla.v00 --- scsi-qla.v01 --- uhci-usb.v00 --- imgdb.tgz --- state.tgz
build=5.0.0-441354
updated=1

As we learned from the previous article, the cron'd /sbin/auto-backup.sh generates a local.tgz which is then converted to state.tgz which contains all files automatically backed up by VMware. This file is loaded up along with other modules as part of the boot process. Understanding this, allows us to take advantage of this feature for persisting our own configuration files.

Here is an example use case for creating .ssh directory for SSH keys and persisting a script
(ghettoVCB.sh) in /bin for an ESXi 4.x host:

Step 1 - Re-create the modified directory structure and files in a temporary local path which will then be tarred and gzip. An example would be the following:

Actual change:
/.ssh/authorized_keys
/bin/ghettoVCB.sh

Temporary local directory structure of change:
/tmp/.ssh/authorized_keys
/tmp/bin/ghettoVCB.sh

Note: It is very important to ensure that the modified files get the stickybit permission set. As noted in the last article that upon a change, the visorFS will automatically create a special file to denote it for backup but also it allows the file to be writable at some later point for custom files being added.

Step 2 - You will use the tar utility to tar/gzip the contents in a file with extension .tgz. One thing to note, the file name including the extension must not exceed 12 characters. In our example, we made two changes and re-created the local structure under /tmp. We will need to change into /tmp directory and tar up the contents by using the following command:

/tmp # tar -czvf ghetto.tgz .ssh/ bin/
.ssh/
.ssh/authorized_keys
bin/
bin/ghettoVCB.sh

Now you should have the contents of .ssh and bin within your *.tgz file. To verify and view the contents, you can run the following command:

/tmp # tar -tzvf ghetto.tgz
drw------- 0/0 0 2011-08-10 04:10:11 .ssh/
-rw------T 0/0 399 2011-08-10 04:10:11 .ssh/authorized_keys
drwxr-xr-x 0/0 0 2011-08-10 04:10:24 bin/
-rwxr-xr-t 0/0 46973 2011-08-10 04:10:24 bin/ghettoVCB.sh

Step 3 - Next we need to copy the *.tgz file into /bootbank and modify the /bootbank/boot.cfg to ensure our new module is included in the boot up process

cp /tmp/ghetto.tgz /bootbank

Here is what the modified ESXi 4.x boot.cfg should look like after:

kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- ghetto.tgz
build=4.1.0-348481
updated=1
bootstate=0

The next time you reboot the system, you will automatically have your .ssh directory containing your SSH keys and the ghettoVCB script under /bin directory.

Now this is great for an inline modification, but what about creating custom configuration files and including that as part of a default kickstart installation? What about something like custom firewall rules in ESXi 5? In the following example, we'll include a custom firewall rule called "virtuallyGhetto.xml" which will be stored in /etc/vmware/firewall when the contents of the module is extracted.

Step 1 - We of course need to create the XML file containing the firewall rule and create the directory structure in which it will be unloaded to.

/tmp/etc/vmware/firewall/virtuallyGhetto.xml

Step 2 - Next we'll tar up the local contents

[root@air tmp]# tar -zcvf ghetto.tgz etc/
etc/
etc/vmware/
etc/vmware/firewall/
etc/vmware/firewall/virtuallyGhetto.xml

Step 3 - This new package will need to be stored on you installation server which will be reachable via http using wget and as part of the %firstboot stanza in your ESXi 5 kickstart. It will download the *.tgz file and append the entry in /bootbank/boot.cfg configuration file. Here are the entries that should go into your kickstart:

# Adding custom files to boot argument (e.g. custom firewall rules,etc)

wget http://air.primp-industries.com/esxi5/ghetto.tgz -O /bootbank/ghetto.tgz
sed -e '/modules=/s/$/ --- ghetto.tgz/' -i /bootbank/boot.cfg

Note: If you add custom files that are located under /etc and you have the stickybit enabled on your file, changes made will persist upon the next reboot by either manually running /sbin/auto-backup.sh or letting it run via cron. If you add custom files that are not located under /etc, any change you make must be periodically updated in your custom *.tgz file else the next reboot, the original file will be loaded.

The question of persisting configuration changes in ESXi and the expected behavior using the ESXi Shell is a question that comes up quite frequently. I think this is primarily due to user's experience with the classic ESX Service Console and the expectation that ESXi will behave exactly the same.

First off, any changes made to ESXi 4.x or 5.x using the vSphere Client, any of the vSphere SDK toolkits (vCLI, PowerCLI, etc) or going through vCenter will preserve the changes across system reboots. The caveat is when going directly into the ESXi Shell (formally TSM) and manually editing or adding files to the default ESXi installation, where the persistence of the files will vary depending on what was updated.

So why does editing files in ESXi filesystem not persisted across system reboots? The reason for this is ESXi uses an in-memory filesystem. For more details, check out The Architecture of VMware ESXi whitepaper by VMware. As I mentioned earlier, the persistence of files will depend on what was updated as there are exceptions to the rules.

There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.

UPDATE (08/14/2014) - Take a look at this article for more details regarding the various times an automatic backup is performed by ESXi

The "automatic" backup process occurs in the following scenarios:

1) /sbin/auto-backup.sh runs via cron the first minute of every hour and performs a "find" operation under /etc looking for files that being with ".#":

~ # cat /var/spool/cron/crontabs/root
#syntax : minute hour day month dayofweek command
01 01 * * * /sbin/tmpwatch.sh
01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)

Here is an example of the files backed up in ESXi 4.x:

~ # find etc -follow -type f -name ".#*"
etc/dropbear/.#dropbear_dss_host_key
etc/dropbear/.#dropbear_rsa_host_key
etc/security/.#access.conf
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.crt
etc/vmware/ssl/.#rui.key
etc/vmware/.#dvsdata.db
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/.#hosts
etc/.#inetd.conf
etc/.#rc.local
etc/.#chkconfig.db
etc/.#random-seed
etc/.#resolv.conf
etc/.#syslog.conf
etc/.#shadow
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx

Here is an example of the files backed up in ESXi 5.x:

~ # find etc -follow -type f -name ".#*"
etc/vmware/vpxa/.#vpxa.cfg
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.key
etc/vmware/ssl/.#rui.crt
etc/vmware/.#config
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/vmware/.#dvsdata.db
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile
etc/sfcb/repository/root/config/.#omc_namespaceconfig.idx
etc/sfcb/repository/root/config/.#omc_namespaceconfig
etc/sfcb/repository/root/config/.#omc_config.idx
etc/sfcb/repository/root/config/.#omc_config
etc/sfcb/.#uuid
etc/sfcb/.#sfcb.cfg
etc/ssh/.#ssh_host_rsa_key
etc/ssh/.#ssh_host_rsa_key.pub
etc/ssh/keys-root/.#authorized_keys
etc/ssh/.#ssh_host_dsa_key
etc/ssh/.#ssh_host_dsa_key.pub
etc/vmsyslog.conf.d/.#vmkernel.conf
etc/vmsyslog.conf.d/.#hostd.conf
etc/vmsyslog.conf.d/.#fdm.conf
etc/vmsyslog.conf.d/.#vpxa.conf
etc/security/.#access.conf
etc/.#chkconfig.db
etc/.#hosts
etc/.#ntp.conf
etc/.#random-seed
etc/.#resolv.conf
etc/.#vmsyslog.conf
etc/.#shadow

2) There is also another "internal" backup process that runs every 10 minutes which is not configurable that backups the ESXi filesystem. In the worse case, you may lose changes made in the last 10 minutes if you had a system crash and it reboots.

With the latest release of ESXi 5, ESXi Shell (previous known as unsupported mode and Tech Support Mode) is officially supported for troubleshooting purposes, modifying or adding files is still not officially supported. Though if you need to make changes or add custom files to either ESXi 4.x or 5.x, there are a few options to persist the changes.

Option #1 - If you edit/modify any of the files listed above, these are automatically backed up by ESXi, it's generally recommend you manually run /sbin/auto-backup.sh to ensure they are backed up after your changes.

Option #2 - Edit/Modify files using /etc/rc.local file which executes at the very end of the bootup process. You can make your file changes in this script and they will automatically execute once the system has ran through all the default start up scripts. There two different methods depending on the complexity of your modifications, you may choose to use something like a here-document to create new/modified files or copy configuration files which have been persisted through a local or remote VMFS/NFS datastore.

Here is an example of setting up SSH keys for ESXi and manually creating the .ssh directory and SSH public keys:

Here is an example of copying a script (ghettoVCB.sh) from persistent local datastore to /bin directory:

Once you have made your changes, you will need to manually run /sbin/auto-backup.sh which ensures the changes made to /etc/rc.local will be persisted upon the next system reboot.

Option #2 provides for the most flexibility and allows you to make various configuration changes or addition of files and can easily be integrated into kickstart installation. The entries would be part of your %firstboot stanza to generate the entries in /etc/rc.local file.

In part 2 of this article, I will go over another method that can be leveraged in both ESXi 4.x and 5.x where custom files can be added and persisted as part of the default installation process using kickstart or through a one time manual configuration. I will provide some examples including persisting custom firewall rules in ESXi 5, so stay tuned!