The question of persisting configuration changes in ESXi and the expected behavior using the ESXi Shell is a question that comes up quite frequently. I think this is primarily due to user's experience with the classic ESX Service Console and the expectation that ESXi will behave exactly the same.
First off, any changes made to ESXi 4.x or 5.x using the vSphere Client, any of the vSphere SDK toolkits (vCLI, PowerCLI, etc) or going through vCenter will preserve the changes across system reboots. The caveat is when going directly into the ESXi Shell (formally TSM) and manually editing or adding files to the default ESXi installation, where the persistence of the files will vary depending on what was updated.
So why does editing files in ESXi filesystem not persisted across system reboots? The reason for this is ESXi uses an in-memory filesystem. For more details, check out The Architecture of VMware ESXi whitepaper by VMware. As I mentioned earlier, the persistence of files will depend on what was updated as there are exceptions to the rules.
There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.
UPDATE (08/14/2014) - Take a look at this article for more details regarding the various times an automatic backup is performed by ESXi
The "automatic" backup process occurs in the following scenarios:
1) /sbin/auto-backup.sh runs via cron the first minute of every hour and performs a "find" operation under /etc looking for files that being with ".#":
~ # cat /var/spool/cron/crontabs/root
#syntax : minute hour day month dayofweek command
01 01 * * * /sbin/tmpwatch.sh
01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)
Here is an example of the files backed up in ESXi 4.x:
~ # find etc -follow -type f -name ".#*"
etc/dropbear/.#dropbear_dss_host_key
etc/dropbear/.#dropbear_rsa_host_key
etc/security/.#access.conf
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.crt
etc/vmware/ssl/.#rui.key
etc/vmware/.#dvsdata.db
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/.#hosts
etc/.#inetd.conf
etc/.#rc.local
etc/.#chkconfig.db
etc/.#random-seed
etc/.#resolv.conf
etc/.#syslog.conf
etc/.#shadow
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
Here is an example of the files backed up in ESXi 5.x:
~ # find etc -follow -type f -name ".#*"
etc/vmware/vpxa/.#vpxa.cfg
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.key
etc/vmware/ssl/.#rui.crt
etc/vmware/.#config
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/vmware/.#dvsdata.db
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile
etc/sfcb/repository/root/config/.#omc_namespaceconfig.idx
etc/sfcb/repository/root/config/.#omc_namespaceconfig
etc/sfcb/repository/root/config/.#omc_config.idx
etc/sfcb/repository/root/config/.#omc_config
etc/sfcb/.#uuid
etc/sfcb/.#sfcb.cfg
etc/ssh/.#ssh_host_rsa_key
etc/ssh/.#ssh_host_rsa_key.pub
etc/ssh/keys-root/.#authorized_keys
etc/ssh/.#ssh_host_dsa_key
etc/ssh/.#ssh_host_dsa_key.pub
etc/vmsyslog.conf.d/.#vmkernel.conf
etc/vmsyslog.conf.d/.#hostd.conf
etc/vmsyslog.conf.d/.#fdm.conf
etc/vmsyslog.conf.d/.#vpxa.conf
etc/security/.#access.conf
etc/.#chkconfig.db
etc/.#hosts
etc/.#ntp.conf
etc/.#random-seed
etc/.#resolv.conf
etc/.#vmsyslog.conf
etc/.#shadow
2) There is also another "internal" backup process that runs every 10 minutes which is not configurable that backups the ESXi filesystem. In the worse case, you may lose changes made in the last 10 minutes if you had a system crash and it reboots.
With the latest release of ESXi 5, ESXi Shell (previous known as unsupported mode and Tech Support Mode) is officially supported for troubleshooting purposes, modifying or adding files is still not officially supported. Though if you need to make changes or add custom files to either ESXi 4.x or 5.x, there are a few options to persist the changes.
Option #1 - If you edit/modify any of the files listed above, these are automatically backed up by ESXi, it's generally recommend you manually run /sbin/auto-backup.sh to ensure they are backed up after your changes.
Option #2 - Edit/Modify files using /etc/rc.local file which executes at the very end of the bootup process. You can make your file changes in this script and they will automatically execute once the system has ran through all the default start up scripts. There two different methods depending on the complexity of your modifications, you may choose to use something like a here-document to create new/modified files or copy configuration files which have been persisted through a local or remote VMFS/NFS datastore.
Here is an example of setting up SSH keys for ESXi and manually creating the .ssh directory and SSH public keys:
Here is an example of copying a script (ghettoVCB.sh) from persistent local datastore to /bin directory:
Once you have made your changes, you will need to manually run /sbin/auto-backup.sh which ensures the changes made to /etc/rc.local will be persisted upon the next system reboot.
Option #2 provides for the most flexibility and allows you to make various configuration changes or addition of files and can easily be integrated into kickstart installation. The entries would be part of your %firstboot stanza to generate the entries in /etc/rc.local file.
In part 2 of this article, I will go over another method that can be leveraged in both ESXi 4.x and 5.x where custom files can be added and persisted as part of the default installation process using kickstart or through a one time manual configuration. I will provide some examples including persisting custom firewall rules in ESXi 5, so stay tuned!
Any chance you could post an update showing how you would go about keeping the ghetto backup script in the /tmp directory for example. Each time I write the script in /tmp, it is removed by the OS a while later.
While your details above are probably good for experienced users, I am not a very highly ESXi experienced user other than using the product from GUI.
Would be most appreciated.
Thank you.
/sbin/auto-backup.sh is not able to persist the change in /etc/security/access.conf in ESXi 5.5 after reboot?
Option #2 works though.
Hello,
I use Esxi 5.5 and the file /etc/rc.local was not backed up by /sbin/auto-backup.sh after I modyfied it.
So I had to use the file /etc/rc.local.d/local.sh to insert some modifications which survived a reboot.
Best regards
Siegfried
Can anyone advise if the crontabs themselves persist through reboot? /var/spool/cron/crontabs/root
no, crontab is not persistent. But you can set /etc/rc.local.d/local.sh to modify the crontab and relaunch cron. So it will be done at each boot.
I needed to make sure that keys for a nagios user were backed up and made persistent. As mentioned you cannot create the required .#authorized_keys file manually, but there is such a file in /etc/ssh/keys-root.
So I put my public key for nagios in the /etc/ssh/keys-root/authorized_keys file.
I then renamed the keys-root directory like this:
mv /etc/ssh/keys-root /etc/ssh/keys-nagios
So I now have a .#authorized_keys file in keys-nagios and that will now get backed up. I don't care about the keys-root directory because I will not be using a root key.
Thanks for the info above that led me to this solution.