When an ESXi host is unable to forward its logs to a remote syslog server, a VMkernel Observation (VOB) is automatically raised by the host and it can be used to proactively alert administrators, which has been possible since ESXi 5.0 .... per this blog post from 2012 after some Googling! 😅😂
While I was pretty confident the behavior described above still holds true for our latest ESXi 7.x and 8.x releases, I wanted to be sure before responding back to a colleague. I deployed the latest ESXi 7.0 Update 3q and ESXi 8.0 Update 3b and after configuring syslog forwarding, I disabled the NIC on my Aria Operations for Logs to simulate a network disconnect and I saw the following log entry in /var/log/vobd.log
2024-09-28T21:12:00.298Z: [UserLevelCorrelator] 7452916537us: [esx.problem.vmsyslogd.remote.failure] The host "192.168.30.62:514" has become unreachable. Remote logging to this host has stopped.
By default, ESXi will attempt to retry the remote syslog connection after the configured timeout (default 180 seconds), which is a relatively new configuration option that is available with ESXCLI (esxcli system syslog config set --default-timeout XX).
As noted in my 2012 blog post, you can create a vCenter Server Alarm using the VOB IDs (esx.problem.vmsyslogd.remote.failure) and this continues to work with the latest releases of vCenter Server.
If any of your ESXi hosts fails to connect to their configured syslog server, an alarm will now automatically be raised vCenter Server as you can see from the screenshot below.
If you are using Aria Operations for Logs, you can also get proactive notifications by configuring the Content Pack Alerts, which consumes the same VOB information, but this can now be done globally across your entire deployment rather than individual vCenter Servers, which is an added benefit.
Lastly, if you want to verify that a specific ESXi host is properly forwarding its logs to your remote syslog server, you can send a custom syslog message using the ESXCLI "mark" command, which can be useful to designate a specific point for debugging, troubleshooting or verification purposes.
Thanks William for providing such a useful tip !
Nice!
Similarly, the vSphere content pack in Operations for Logs has a built in alert for when hosts stop sending logs. I always recommend that customers enable it.
Do you know such mechanism exists for tanzu or tkgm?