WilliamLam.com

Quick Tip - Monitoring ESXi remote syslog forwarding

by // 3 Comments

When an ESXi host is unable to forward its logs to a remote syslog server, a VMkernel Observation (VOB) is automatically raised by the host and it can be used to proactively alert administrators, which has been possible since ESXi 5.0 .... per this blog post from 2012 after some Googling! 😅😂

While I was pretty confident the behavior described above still holds true for our latest ESXi 7.x and 8.x releases, I wanted to be sure before responding back to a colleague. I deployed the latest ESXi 7.0 Update 3q and ESXi 8.0 Update 3b and after configuring syslog forwarding, I disabled the NIC on my Aria Operations for Logs to simulate a network disconnect and I saw the following log entry in /var/log/vobd.log

2024-09-28T21:12:00.298Z: [UserLevelCorrelator] 7452916537us: [esx.problem.vmsyslogd.remote.failure] The host "192.168.30.62:514" has become unreachable. Remote logging to this host has stopped.

By default, ESXi will attempt to retry the remote syslog connection after the configured timeout (default 180 seconds), which is a relatively new configuration option that is available with ESXCLI (esxcli system syslog config set --default-timeout XX).

[Read more...]

Dynamic ESXi firewall rulset for non-standard syslog ports in vSphere 8.0 Update 2b

by // 5 Comments

For most users who configure syslog for their ESXi hosts (hopefully everyone is doing that for audit, compliance and troubleshooting purposes), they typically stick with the default syslog ports 514 for UDP/TCP or 1514 for TLS.

A huge benefit of using the default syslog ports is that the ESXi firewall is already configured with these rulesets configured for outbound access.


If you require to use a non-standard syslog port for ESXi, the current solution was not ideal. While you can open up a custom port using the ESXi firewall, the issue is persisting that customization, which either requires a custom VIB or messing around with local.sh startup script.

A nice enhancement that is included with the recent release of vSphere 8.0 Update 2b is the support for a dynamic ESXi ruleset when non-standard syslog ports is configured.

As you can see in the example below when I configure my ESXi host to use a syslog server with a custom port 12345, the ESXi will automatically create a dynamic firewall ruleset that will open up that port for outbound connectivity. If you change the port or disable the syslog configuration, then the dynamic ruleset will be updated and/or removed.

Enhanced vCenter Server Audit Event & Logging in vSphere 6.7 Update 2

by // 9 Comments

A couple of years back I had published a detailed analysis on vCenter Server's Authentication (AuthN) and Authorization (AuthZ) from an auditing and logging standpoint. This has been the go to reference for many of our customers and the posts also includes a number of log samples which I have documented in the following Github repository.

In addition to serving as a reference for our customers, it has also helped our Product and Engineering teams understand where we still had some gaps and how we could improve the overall user experience. As hinted in the recently announced vSphere 6.7 Update 2 release, which will be available soon, there are number of new auditing enhancements that have been made to both vCenter Server and the vCenter Single Sign-On (SSO) service that I think customers will really appreciate.

"Real" client IP address in Events

When you look at a login or logout Event in vCenter Server today, you may have noticed the user's client IP Address is actually of the vCenter Server rather than the actual remote client's address and the reason for this is explained here. In vSphere 6.7 Update 2, the real client IP Address is now captured and is included in all successful login/logout and failed logins. This information can now enable administrators to easily identify unauthorized access and be able to quickly track down the systems initiating the connections.

[Read more...]