WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / Automation / Heads Up - Potential missing vCenter Server Events due to sequence ID overflow

Heads Up - Potential missing vCenter Server Events due to sequence ID overflow

07.15.2022 by William Lam // 2 Comments

We had a few users who had reported issues with using the VMware Event Broker Appliance (VEBA) solution where VEBA was not receiving any events from the connected vCenter Server. This was really puzzling for the team to debug because the user clearly saw events in both the vSphere UI as well as using vSphere Automation Clients like PowerCLI.

After a bit of debugging with a few of our users (huge thanks to Michael Gasch for driving this), we discovered that in certain environments, the generated sequence number that is used for the vCenter Event ID has overflowed and causes the value to have a negative number. To further complicate the debugging, there are actually two ways of fetching vCenter Server Events using the vSphere API. The first is to just look at the LatestPage property, which will return the most recent events and not care about event ID and the second is to use CreateCollectorForEvents() which is more of an event stream and it does care about the event ID being non-negative. You can probably guess which vSphere API VEBA was using, not only because of our check-pointing feature but LatestPage could lose events from a client request point of view for chatty environments.

With the actual root cause identified, we were happy that this was not an issue with VEBA but it did highlight a potential vSphere issues for certain environments where events may appear to be missing. We have also reported the issue to VMware Engineering and improvements are being worked on, the chances of running into this scenario is believed to be low. While most customers may not run into this problem, it can certainly is not easy to diagnosis, especially if you are not using VEBA. We recently had another user who ran into this exact problem and we were able to quickly point them to VMware Support for remediation.

I started to think, how could a user quickly identify whether they are having this problem, especially if they are not using VEBA? I decided to reproduce the issue locally and here is how you can check if your vCenter Server is affected.

Open a browser to https://[VC_FQDN]/mob/?moid=EventManager&doPath=latestEvent and login with an administrative account. Look for either chainId or key and if the value is negative, then you are affected by this issue. You can also refresh the URL which will pull in other events and they should also have negative values for those properties. If you are affected by the issue, please open a VMware Support request and reference PR #2906239.

More from my site

  • Forwarding vCenter Events into AWS EventBridge using vCenter Event Broker Appliance
  • vCenter Event Broker Appliance Updates - VMworld, Fling, Community & Open Source
  • NSX Alarms in vCenter Server using vSphere Events in vSphere 8
  • vSphere Event-Driven Automation using VMware Event Router on VMware Cloud on AWS with Knative or AWS EventBridge
  • Integrating VMware Event Broker Appliance (VEBA) with Zapier

Categories // Automation, vSphere Tags // event, VMware Event Broker Appliance

Comments

  1. _M_P says

    07/16/2022 at 12:18 am

    Thank you, mr. Lam, for another very interesting post!
    BTW, you'd fix
    https://[VC_FQDN]mob/?moid=EventManager&doPath=latestEvent
    to
    https://[VC_FQDN]/mob/?moid=EventManager&doPath=latestEvent

    Reply
    • William Lam says

      07/16/2022 at 7:08 am

      Thanks for catch! Its been updated

      Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • Self-Contained & Automated VMware Cloud Foundation (VCF) deployment using new VLC Holodeck Toolkit 03/29/2023
  • ESXi configstorecli enhancement in vSphere 8.0 Update 1 03/28/2023
  • ESXi on Intel NUC 13 Pro (Arena Canyon) 03/27/2023
  • Quick Tip - Enabling ESXi Coredumps to be stored on USB 03/26/2023
  • How to disable the Efficiency Cores (E-cores) on an Intel NUC? 03/24/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...