WilliamLam.com

Free Linux & Windows Syslog Alternatives to depercated vi-logger in vMA 5

by // 12 Comments

Those of you who currently use vi-logger in vMA 4.x as a free syslog server for your ESX(i) hosts may notice this functionality has been removed in the latest vMA 5 release. VMware decided to remove the syslog functionality in vMA in favor of combining it with the vCenter Server. If you decide to run vCenter 5 on Windows, you have the option of installing an additional syslog collector on the same or separate Windows system and registering it as a vCenter plugin. If you are using the new VCVA (vCenter Server Virtual Appliance), there is also a syslog collector that is installed by default.

Using vMA's vi-logger was an easy and free solution, but you still have some alternatives without having to use vCenter or install/build a new syslog server. The following will document a free syslog solution for both a Linux or Windows platform.

Linux Syslog server alternative using vMA 5.0
You can actually leverage the existing syslog server on the latest vMA 5 release and with a few customization, get it setup to start collecting logs from your ESX(i) hosts as before with vi-logger.

Step 1 - It is recommend that you configure an additional disk on vMA for your syslogs as the size of vMA is quite tiny for additional use. I will assume that you know how to add and configure an additional disk, if not you can do a simple search on Google. In this example, I have a second disk that is 10GB and it is mounted up under /var/log/remote which is where the ESX(i) logs will be stored in.

Step 2 - You will need to edit the syslog configuration under /etc/syslog-ng/syslog-ng.conf and you will need to add three entries. The first addition is to configure the source for log messages from the network and enabling both udp/tcp on port 514, you may add the following under the default "src" entry.

source network {
udp6( port(514) );
tcp6( port(514) );
};

The next two entries will define the destination and how it'll log. You will add the following at the end of the syslog-ng.conf configuration file.

destination log_remote {
file("/var/log/remote/$HOST_FROM/$YEAR-$MONTH/messages-$YEAR-$MONTH-$DAY"
create_dirs(yes) frac-digits(3)
template("$ISODATE $PROGRAM $MSGONLY\n")
template_escape(no)
);
};
log {
source(network);
destination(log_remote);
};

The "log_remote" destination will send all logs from your ESX(i) hosts into /var/log/remote and will have the following format: $HOST_FROM/$YEAR-$MONTH/messages-$YEAR-$MONTH-$DAY

Step 3 -  Now you will need to restart the syslog server for the changes to take effect. You will need to run the following command: sudo /etc/init.d/syslog restart

If everything went successful, you should now be able to configure your ESX(i) hosts to point to your vMA 5 system and you should see logs appearing under /var/log/remote

Note: You will need to use sudo to view the directory under /var/log/remote and to view the logs

Windows Syslog server alternative using vCenter Syslog Collector
The vCenter Syslog Collector can be installed and used without the use of vCenter, you can easily turn any existing or new Windows system into a syslog server for your ESX(i) hosts for free.

Step 1 -  It is recommend that you configure a seperate disk on the Windows system that you are going to be using for your syslog server. I will assume that you know how to add and configure an additional disk, if not you can do a simple search on Google. In this example, I have a second disk that is 10GB and listed as Syslog (E: drive)

Step 2 - You will need access to the vCenter Server 5.0 installation ISO or executable to install the Syslog Collector utility. Start the installer and select and install VMware Syslog Collector

Step 3 - You have the option of using the local C:\ drive, but I would recommend setting up a separate drive if you can. If you decide to change the default log location, you need to ensure that you specify the following directory structure VMware\VMware Syslog Collector\Data else you will run into issues with the installation. In this example, I have moved my logs into E:\ drive and the path looks like the following: E:\VMware\VMware Syslog Collector\Data. You also have the ability to change the size of the log files before rotation and the number of logs before rotating.

Step 4 - If you are installing the Syslog Collector on the same host as vCenter Server, you should select the integrated installation else you should select a standalone installation.

Step 5 - The next screen will be the default ports to enable for both TCP/UDP and SSL which can be configured or left as the default as recommend.

Step 6 - The screen is how the Syslog Collector will be identified on the network and it should just be the IP Address of the host.

If everything went successful, you should now be able to configure your ESX(i) hosts to point to your Windows Syslog Collector system and you should see logs appearing under E:\VMware\VMware Syslog Collector\Data

As you can see even with vi-logger being removed in the latest version of vMA 5, you can easily still configure a free syslog server with your ESX(i) hosts on either a Linux or Windows platform.

ESXi Lockdown mode does not play nice with vMA

by // 1 Comment

Today on the VMTN community forums, a user identified an interesting side effect when using vMA's vilogger and enabling lockdown mode on an ESXi host. What the user found was the vilogger daemon stopped collecting logs when lockdown mode was enabled for an ESXi host. At first, I thought lockdown mode should have no affect on vilogger, as it only disables the "root" account from accessing ESXi host other than from the DCUI (Directo Console User Interface).

I replicated the setup in my lab using an ESXi host that was being managed by vMA via vi-fastpass (fpauth) and enabled vilogger for this host. I verified log collection was functional before enabling lockdown mode on the ESXi host, right away vilogger stopped collecting logs when lockdown mode was enabled. When using the "vilogger list" command, the status of the ESXi host goes from "collecting" to "No Permission". I found this to be quite odd and verified what the user was describing in his environment.

Next was to take a look at the vilogger logs which is stored in /var/log/vmware/vma/vilogd.log and I found the same "No Permission" error.

I decided to login to the ESXi host and tailed the hostd logs to see what was going on when lockdown mode was being enabled. What I found was pretty surprising to me, there was a task that removed permissions from the vi-adminXX user account, I was pretty sure at this point, the culprit was related to lockdown mode.

I decided to take a look at VMware's documentation to see what the behavior of Lockdown Mode was and the following snippet taken from vSphere's online documentation explains it all:

The text highlighted in red is the key to the issue the user is facing and specifically the very last section where it states:

you cannot run vCLI commands from an administration server, from a script, or from vMA against the host

This meant that not only the root account was locked out, but all other accounts found on the ESXi host whether they are custom from your environment or from auxiliary systems such as VMware vMA, would be completely disabled. What is even more interesting, even read-only accounts would no longer function, they too had to go to vCenter to be re-proxied to specific ESXi host.

This has a few implications for users considering Lockdown Mode:

  1. All scripts including resxtop and user authentication must go through vCenter. If vCenter went down, you have no remote way to access your ESXi host. This also meant that you could not remotely start up vCenter if it was hosted in a virtual machine but rather from DCUI after enabling Local Tech Support Mode
  2. The use of vMA's vilogger is completely useless when Lockdown Mode is enabled for ESXi host. Users may want to consider setting up a traditional syslog server and have the logs forwarded from the ESXi host

IMHO, I don't think Lockdown Mode should crippled the vilogger functionality, the logging is a "Read" operation and I think re-configuring it to "read-only" role should have suffice. I also think that VMware could have done a better job working with the vMA engineers to support this functionality and have some documentation regarding this issue. For now, if you rely on any type of automation that goes directly to an ESXi host and you are thinking about Lockdown mode, you may want to think twice.