WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple
You are here: Home / Automation / Heads Up - Potential missing vCenter Server Events due to sequence ID overflow

Heads Up - Potential missing vCenter Server Events due to sequence ID overflow

07.15.2022 by William Lam // 2 Comments

We had a few users who had reported issues with using the VMware Event Broker Appliance (VEBA) solution where VEBA was not receiving any events from the connected vCenter Server. This was really puzzling for the team to debug because the user clearly saw events in both the vSphere UI as well as using vSphere Automation Clients like PowerCLI.

After a bit of debugging with a few of our users (huge thanks to Michael Gasch for driving this), we discovered that in certain environments, the generated sequence number that is used for the vCenter Event ID has overflowed and causes the value to have a negative number. To further complicate the debugging, there are actually two ways of fetching vCenter Server Events using the vSphere API. The first is to just look at the LatestPage property, which will return the most recent events and not care about event ID and the second is to use CreateCollectorForEvents() which is more of an event stream and it does care about the event ID being non-negative. You can probably guess which vSphere API VEBA was using, not only because of our check-pointing feature but LatestPage could lose events from a client request point of view for chatty environments.

With the actual root cause identified, we were happy that this was not an issue with VEBA but it did highlight a potential vSphere issues for certain environments where events may appear to be missing. We have also reported the issue to VMware Engineering and improvements are being worked on, the chances of running into this scenario is believed to be low. While most customers may not run into this problem, it can certainly is not easy to diagnosis, especially if you are not using VEBA. We recently had another user who ran into this exact problem and we were able to quickly point them to VMware Support for remediation.

I started to think, how could a user quickly identify whether they are having this problem, especially if they are not using VEBA? I decided to reproduce the issue locally and here is how you can check if your vCenter Server is affected.

Open a browser to https://[VC_FQDN]/mob/?moid=EventManager&doPath=latestEvent and login with an administrative account. Look for either chainId or key and if the value is negative, then you are affected by this issue. You can also refresh the URL which will pull in other events and they should also have negative values for those properties. If you are affected by the issue, please open a VMware Support request and reference PR #2906239.

More from my site

  • Forwarding vCenter Events into AWS EventBridge using vCenter Event Broker Appliance
  • vCenter Event Broker Appliance Updates - VMworld, Fling, Community & Open Source
  • VMware Event Broker Appliance (VEBA) v0.8.0
  • NSX Alarms in vCenter Server using vSphere Events in vSphere 8
  • vSphere Event-Driven Automation using VMware Event Router on VMware Cloud on AWS with Knative or AWS EventBridge

Categories // Automation, vSphere Tags // event, VMware Event Broker Appliance

Comments

  1. *protected_M_P says

    07/16/2022 at 12:18 am

    Thank you, mr. Lam, for another very interesting post!
    BTW, you'd fix
    https://[VC_FQDN]mob/?moid=EventManager&doPath=latestEvent
    to
    https://[VC_FQDN]/mob/?moid=EventManager&doPath=latestEvent

    Reply
    • William Lam says

      07/16/2022 at 7:08 am

      Thanks for catch! Its been updated

      Reply

Thanks for the comment!Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...