WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple
You are here: Home / Automation / Quick Tip - Monitoring ESXi remote syslog forwarding

Quick Tip - Monitoring ESXi remote syslog forwarding

10.01.2024 by William Lam // 3 Comments

When an ESXi host is unable to forward its logs to a remote syslog server, a VMkernel Observation (VOB) is automatically raised by the host and it can be used to proactively alert administrators, which has been possible since ESXi 5.0 .... per this blog post from 2012 after some Googling! 😅😂

While I was pretty confident the behavior described above still holds true for our latest ESXi 7.x and 8.x releases, I wanted to be sure before responding back to a colleague. I deployed the latest ESXi 7.0 Update 3q and ESXi 8.0 Update 3b and after configuring syslog forwarding, I disabled the NIC on my Aria Operations for Logs to simulate a network disconnect and I saw the following log entry in /var/log/vobd.log

2024-09-28T21:12:00.298Z: [UserLevelCorrelator] 7452916537us: [esx.problem.vmsyslogd.remote.failure] The host "192.168.30.62:514" has become unreachable. Remote logging to this host has stopped.

By default, ESXi will attempt to retry the remote syslog connection after the configured timeout (default 180 seconds), which is a relatively new configuration option that is available with ESXCLI (esxcli system syslog config set --default-timeout XX).

As noted in my 2012 blog post, you can create a vCenter Server Alarm using the VOB IDs (esx.problem.vmsyslogd.remote.failure) and this continues to work with the latest releases of vCenter Server.


If any of your ESXi hosts fails to connect to their configured syslog server, an alarm will now automatically be raised vCenter Server as you can see from the screenshot below.


If you are using Aria Operations for Logs, you can also get proactive notifications by configuring the Content Pack Alerts, which consumes the same VOB information, but this can now be done globally across your entire deployment rather than individual vCenter Servers, which is an added benefit.

Lastly, if you want to verify that a specific ESXi host is properly forwarding its logs to your remote syslog server, you can send a custom syslog message using the ESXCLI "mark" command, which can be useful to designate a specific point for debugging, troubleshooting or verification purposes.

More from my site

  • Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm
  • Programmatically accessing the Broadcom Compatibility Guide (BCG)
  • Supported chipsets for the USB Network Native Driver for ESXi Fling
  • Quick Tip - Auditing ESXi boot firmware type
  • Recovering ESXi 7.x & 8.x host after forgetting or losing root password

Categories // Automation, ESXi, vSphere 7.0, vSphere 8.0 Tags // ESXi, syslog

Comments

  1. *protectedAO says

    10/01/2024 at 11:59 am

    Thanks William for providing such a useful tip !

    Reply
  2. *protectedJason M says

    10/01/2024 at 7:06 pm

    Nice!

    Similarly, the vSphere content pack in Operations for Logs has a built in alert for when hosts stop sending logs. I always recommend that customers enable it.

    Reply
  3. *protectedHarshvardhan Gupta says

    10/01/2024 at 9:30 pm

    Do you know such mechanism exists for tanzu or tkgm?

    Reply

Thanks for the comment!Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • VMUG Connect 2025 - Minimal VMware Cloud Foundation (VCF) 5.x in a Box  05/15/2025
  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...