Increase Syslog count in ESXi using Busybox

There was an interesting question on the ESXi forum today using syslog and logging to a local VMFS volume and potentially filling up the datastore. By default, the log files are 2MB each before they are rotated out with a total of 9 copies all together. This ensures that the logs will not grow beyond 18MB and potentially fill up your datastore.

Within the unsupported console of ESXi, there is a little tool called "Busybox" which implements a set of familiar command line utilities called applets. Some of these applets are uptime, crond, chroot, md5sum, etc... If you login to the unsupported console and just type "busybox" on the console, you will see the following:

If you look carefully, you will see that one of the applets is "syslogd", which is the syslog daemon running in ESXi. To access a specific applet, you will just type busybox and then the name of the applet along with -h which forces the applet to provide a set of available options. If we do this for syslogd, you will see the following:

The default configurations should ensure that your logs don't go crazy and fill up the volume, but the downside is that your log history will not be kept forever. That is why it is a best practice to setup a remote syslog server to send all your logs for further processing and auditing if necessary.

However, if you wanted to change this defaut, you can. As you can see from the options above, you can configure not only the size (max of 2MB) before rotating out, but also the number of logs to keep (max of 99). While you cannot increase the size of the individual logs, you can specify a larger number of logs to keep. If you want to make the changes live while the system is running, you'll need to perform the following:

1. Find the current pid of syslogd

~ # ps | grep syslog
4313 4313 busybox syslogd

2. Kill the syslog process

~ # kill $(cat /var/run/syslogd.pid)

3. Let's say you want to keep 50 copies instead of 9

~ # busybox syslogd -b 50

Note:

• The input to -s is in bytes with range (0-2097151)
• The input to -b has range (1-99)

It's important to note that this change will not persist through a reboot. I have not been able to figure out where this is set; it could just be hardcoded in the binary by default. A way around this is to re-define the configuration upon boot up by adding an entry to /etc/rc.local which will kill the current running syslogd and then start up with the new parameters as shown above.

Add the following lines to the end of /etc/rc.local:

kill $(cat /var/run/syslogd.pid)
busybox syslogd -b 50

Now for this change to fully persist, you need to do one more thing, you'll need to run /sbin/auto-backup.sh which will force a local backup of the ESXi configuration files which includes /etc/rc.local so that it'll survive through the next reboot. Now you'll be able to store additional ESXi logs for a longer period of time if you choose to log locally.