The question of persisting configuration changes in ESXi and the expected behavior using the ESXi Shell is a question that comes up quite frequently. I think this is primarily due to user's experience with the classic ESX Service Console and the expectation that ESXi will behave exactly the same.
First off, any changes made to ESXi 4.x or 5.x using the vSphere Client, any of the vSphere SDK toolkits (vCLI, PowerCLI, etc) or going through vCenter will preserve the changes across system reboots. The caveat is when going directly into the ESXi Shell (formally TSM) and manually editing or adding files to the default ESXi installation, where the persistence of the files will vary depending on what was updated.
So why does editing files in ESXi filesystem not persisted across system reboots? The reason for this is ESXi uses an in-memory filesystem. For more details, check out The Architecture of VMware ESXi whitepaper by VMware. As I mentioned earlier, the persistence of files will depend on what was updated as there are exceptions to the rules.
There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.
UPDATE (08/14/2014) - Take a look at this article for more details regarding the various times an automatic backup is performed by ESXi
The "automatic" backup process occurs in the following scenarios:
1) /sbin/auto-backup.sh runs via cron the first minute of every hour and performs a "find" operation under /etc looking for files that being with ".#":
~ # cat /var/spool/cron/crontabs/root
#syntax : minute hour day month dayofweek command
01 01 * * * /sbin/tmpwatch.sh
01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)
Here is an example of the files backed up in ESXi 4.x:
~ # find etc -follow -type f -name ".#*"
etc/dropbear/.#dropbear_dss_host_key
etc/dropbear/.#dropbear_rsa_host_key
etc/security/.#access.conf
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.crt
etc/vmware/ssl/.#rui.key
etc/vmware/.#dvsdata.db
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/.#hosts
etc/.#inetd.conf
etc/.#rc.local
etc/.#chkconfig.db
etc/.#random-seed
etc/.#resolv.conf
etc/.#syslog.conf
etc/.#shadow
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
Here is an example of the files backed up in ESXi 5.x:
~ # find etc -follow -type f -name ".#*"
etc/vmware/vpxa/.#vpxa.cfg
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.key
etc/vmware/ssl/.#rui.crt
etc/vmware/.#config
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/vmware/.#dvsdata.db
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile
etc/sfcb/repository/root/config/.#omc_namespaceconfig.idx
etc/sfcb/repository/root/config/.#omc_namespaceconfig
etc/sfcb/repository/root/config/.#omc_config.idx
etc/sfcb/repository/root/config/.#omc_config
etc/sfcb/.#uuid
etc/sfcb/.#sfcb.cfg
etc/ssh/.#ssh_host_rsa_key
etc/ssh/.#ssh_host_rsa_key.pub
etc/ssh/keys-root/.#authorized_keys
etc/ssh/.#ssh_host_dsa_key
etc/ssh/.#ssh_host_dsa_key.pub
etc/vmsyslog.conf.d/.#vmkernel.conf
etc/vmsyslog.conf.d/.#hostd.conf
etc/vmsyslog.conf.d/.#fdm.conf
etc/vmsyslog.conf.d/.#vpxa.conf
etc/security/.#access.conf
etc/.#chkconfig.db
etc/.#hosts
etc/.#ntp.conf
etc/.#random-seed
etc/.#resolv.conf
etc/.#vmsyslog.conf
etc/.#shadow
2) There is also another "internal" backup process that runs every 10 minutes which is not configurable that backups the ESXi filesystem. In the worse case, you may lose changes made in the last 10 minutes if you had a system crash and it reboots.
With the latest release of ESXi 5, ESXi Shell (previous known as unsupported mode and Tech Support Mode) is officially supported for troubleshooting purposes, modifying or adding files is still not officially supported. Though if you need to make changes or add custom files to either ESXi 4.x or 5.x, there are a few options to persist the changes.
Option #1 - If you edit/modify any of the files listed above, these are automatically backed up by ESXi, it's generally recommend you manually run /sbin/auto-backup.sh to ensure they are backed up after your changes.
Option #2 - Edit/Modify files using /etc/rc.local file which executes at the very end of the bootup process. You can make your file changes in this script and they will automatically execute once the system has ran through all the default start up scripts. There two different methods depending on the complexity of your modifications, you may choose to use something like a here-document to create new/modified files or copy configuration files which have been persisted through a local or remote VMFS/NFS datastore.
Here is an example of setting up SSH keys for ESXi and manually creating the .ssh directory and SSH public keys:
Here is an example of copying a script (ghettoVCB.sh) from persistent local datastore to /bin directory:
Once you have made your changes, you will need to manually run /sbin/auto-backup.sh which ensures the changes made to /etc/rc.local will be persisted upon the next system reboot.
Option #2 provides for the most flexibility and allows you to make various configuration changes or addition of files and can easily be integrated into kickstart installation. The entries would be part of your %firstboot stanza to generate the entries in /etc/rc.local file.
In part 2 of this article, I will go over another method that can be leveraged in both ESXi 4.x and 5.x where custom files can be added and persisted as part of the default installation process using kickstart or through a one time manual configuration. I will provide some examples including persisting custom firewall rules in ESXi 5, so stay tuned!
