There was an interesting question on the ESXi forum today using syslog and logging to a local VMFS volume and potentially filling up the datastore. By default, the log files are 2MB each before they are rotated out with a total of 9 copies all together. This ensures that the logs will not grow beyond 18MB and potentially fill up your datastore.
Within the unsupported console of ESXi, there is a little tool called "Busybox" which implements a set of familiar command line utilities called applets. Some of these applets are uptime, crond, chroot, md5sum, etc... If you login to the unsupported console and just type "busybox" on the console, you will see the following:
If you look carefully, you will see that one of the applets is "syslogd", which is the syslog daemon running in ESXi. To access a specific applet, you will just type busybox and then the name of the applet along with -h which forces the applet to provide a set of available options. If we do this for syslogd, you will see the following:
The default configurations should ensure that your logs don't go crazy and fill up the volume, but the downside is that your log history will not be kept forever. That is why it is a best practice to setup a remote syslog server to send all your logs for further processing and auditing if necessary.
However, if you wanted to change this defaut, you can. As you can see from the options above, you can configure not only the size (max of 2MB) before rotating out, but also the number of logs to keep (max of 99). While you cannot increase the size of the individual logs, you can specify a larger number of logs to keep. If you want to make the changes live while the system is running, you'll need to perform the following:
1. Find the current pid of syslogd
~ # ps | grep syslog
4313 4313 busybox syslogd
2. Kill the syslog process
~ # kill $(cat /var/run/syslogd.pid)
3. Let's say you want to keep 50 copies instead of 9
~ # busybox syslogd -b 50
- The input to -s is in bytes with range (0-2097151)
- The input to -b has range (1-99)
It's important to note that this change will not persist through a reboot. I have not been able to figure out where this is set; it could just be hardcoded in the binary by default. A way around this is to re-define the configuration upon boot up by adding an entry to /etc/rc.local which will kill the current running syslogd and then start up with the new parameters as shown above.
Add the following lines to the end of /etc/rc.local:
kill $(cat /var/run/syslogd.pid)
busybox syslogd -b 50
Now for this change to fully persist, you need to do one more thing, you'll need to run /sbin/auto-backup.sh which will force a local backup of the ESXi configuration files which includes /etc/rc.local so that it'll survive through the next reboot. Now you'll be able to store additional ESXi logs for a longer period of time if you choose to log locally.
A couple of corrections and an important caveat to add.
Max log file size is larger than 2MB - I set mine to 10MB, which you CAN change (author states max of 2MB, then states it can't be changed - contradictory).
rc.local command should not specify busybox. syslogd will start in busybox without it being called. If busybox is called, an instance of busybox will startup within busybox, then syslogd within that. Use this command format to use the author's settings:
syslogd -b 50
Be careful that you don't fill up portions of ESXi's file system. I moved my syslogs to shared SAN disk, formated VMFS3, and thought I would be safe to keep the max logs of 99, but that setting also impacts the /var/logs/messages logging. /var is very limited in space and I filled it, which caused stability issues. VMware Support recommended a max of 20 log files. To relocate syslogs to SAN, ref VMware KB1016621.
For the moment I am using this configuration, although it hasn't been in place very long. I have experienced syslogd not running, apparently failing to start on one boot but succeeding on the next, and possibly stopping some period after reboot.
It is my understanding that ESX 5.0 has much improved logging - I look forward to experiencing it.
This is my syslogd startup command in rc.local:
syslogd -b 20 -s 10240 -S