One of the really exciting features that will be included in the upcoming release of the VMware Event Broker Appliance (VEBA) v0.7 release (currently in Tech Preview) is the support for incoming webhooks! This will allow customers to easily build event-driven automation for non-vSphere based events and even non-VMware events while still maintaining a consistent consumption experience. If you are interested in learning more about the upcoming VEBA v0.7 release, Michael Gasch and myself will be doing a LIVE VMworld Session - VEBA Revolutions - Unleashing the Power of Event-Driven Automation #CODE2773 that you should definitely add to your schedule builder!
Webhook support can easily be enabled during the initial VEBA appliance deployment using a few new OVF properties or configured through the VMware Event Router configuration when deploying to an existing Kubernetes cluster using kubectl or Helm. Once the webhook endpoint is running, users can simply publish their custom events as a conformant CloudEvent and VEBA will ensure these custom events are immediately available for consumption by function authors. This means any product and/or service that can construct a custom HTTP payload including headers will be able to take advantage of this new VEBA feature! I also want to mention that this is NOT the only way to produce custom events that VEBA can ingest, but is certainly one simple way.
To help make this concept more concrete, I wanted to see how we could integrate VMware Cloud events into VEBA by using this new webhook mechanism and using the VMware Cloud Notification Gateway. Below is a diagram to help illustrate what is happens when a VMware Cloud event is generated and how it can be consumed by VEBA. The beauty of this type of a solution is the "Event Producer" does not have to know anything about the "Event Consumer" or how they might consume the data. The producer simply pushes events into VEBA and if there is a consumer who cares about a specific event and wishes to do something about it, they can create a function that will listen for a specific event(s) and perform an operation like sending to Slack as an example.
- Event is produced by VMware Cloud and pushed by the VMware Cloud Notification Gateway (NGW)
- A conformant CloudEvent payload is constructed from VMware Cloud event by NGW service
- NGW forwards the custom CloudEvent to VEBA's webhook endpoint (https://[VEBA-FQDN]/webhook)
- VEBA functions can now react to these custom CloudEvents (e.g. SDDC Provisioned Event)
Below are the instructions for configuring NGW webhook subscription, which will be needed before VEBA customers can start consuming VMware Cloud events.
Pre-Reqs:
- PowerCLI 12.0 or newer
- VMware Cloud on AWS scoped Refresh Token
- VMware.VMC.Notification 1.0.6+ Community Module Installed
Step 1 - Connect to the VMC Service API endpoint using Connect-VMC cmdlet:
Connect-VMC -RefreshToken $RefreshToken
Step 2 - Connect to the VMC Notification Gateway API endpoint using Connect-VmcNotification function:
Import-Module VMware.VMC.Notification
Connect-VmcNotification -OrgName $OrgName -RefreshToken $RefreshToken
Step 3 - Next, we need to create our NGW webhook subscription using the NGW API. In the example below, we are subscribing to all events, but you can certainly subscribe to a specific set of events. The benefit of subscribing to all events is that you do not have to think about what events you may want in the future, which would require additional webhook subscription or updating the existing. VEBA will simply receive all events BUT it will not be processed unless you have create a function to reach to a specific event, which makes this model very scalable as new events are added from VMware Cloud or new use cases arise.
As mentioned earlier, we also need NGW to construct a valid CloudEvent payload when it sends the VMware Cloud Event and we can do this by leveraging the templating functionality that is part of NGW, which will allow us to control the payload format but also take advantage of dynamic variables such as the resource names and IDs. These template variables are denoted by use of {} (brackets). In addition, the CloudEvent payload must also an Id, Type, Source, Specversion, Datacontenttype and Data for it to be a conformant CloudEvent which VEBA will be able to process. It is also important to understand the type value that we are defining which is in the form of vmware.vmc.<ngw-event-id>.v0, the reason is that when we create a function in VEBA, if we wish to filter on a specific event like SDDC provision, we need to know the full key which will look like vmware.vmc.SDDC-PROVISION.v0 in the case of a new SDDC that is created. Lastly for my VEBA setup, I have also enabled authentication and hence an additional Authorization header must also be added to WebhookHeaders parameter.
$vmcVebaWebhookNotificationParams = @{
ClientId = "vmc-veba-webhook-notification";
WebhookURL = "https://veba.vmware.corp/webhook";
WebhookHeaders = @{
"Content-Type" = 'application/cloudevents+json'
"Authorization" = "Basic XXX"
};
WebhookBody = @{
"id" = "{message.notification_id}"
"type" = "vmware.vmc.{event_id}.v0"
"source" = "https://vmc.vmware.com/console/sddcs/{org_id}"
"specversion" = "1.0"
"datacontenttype" = "application/json"
"data" = [ordered]@{
"org_id" = "{org_id}"
"org_name" = "{org_name}"
"resource_id" = "{message.resource_id}"
"resource" = "{message.resource_type}"
"resource_name" = "{message.resource_name}"
"message_username" = "{message.user_name}"
"message" = "{message.text}"
}
};
NotificationEvents = @("*");
}
Step 4 - Finally, we create the NGW webhook subscription by using the New-VmcNotificationWebhook function and passing in our param from the previous step. I have enabled the -Troubleshoot option as this is useful to see what the underlying REST API call looks like for the NGW request. The key here with the template property is that it must be a JSON encoded string, so this was useful to see as I was converting the PowerShell object into a JSON string.
New-VmcNotificationWebhook @vmcVebaWebhookNotificationParams -Troubleshoot
Here is an example output if the NGW webhook subscription was successfully created
To verify that the NGW webhook subscription was configured correctly, you can exercise any VMware Cloud activity that would generate an event such as an SDDC provision.
Now, to confirm that we can subscribe to these new VMware Cloud events from VEBA, we first need to confirm that we are receiving the webhook. To do so, we check the logs for the VMware Event Router for Webhook Provider. You can run the following command to tail the logs and you should see an entry on successfully receiving the webhook from NGW and then forwarding that the broker.
# k -n vmware-system logs deployment.apps/vmware-event-router-webhook
2021-09-12T20:48:05.449Z INFO [WEBHOOK] webhook/webhook.go:100 enabling endpoint authentication with basic auth
2021-09-12T20:48:05.449Z INFO [MAIN] router/main.go:122 starting webhook listener {"commit": "UNKNOWN", "version": "UNKNOWN", "address": "[::]:8080"}
2021-09-12T20:48:05.449Z INFO [KNATIVE] injection/injection.go:61 Starting informers...
2021-09-12T20:48:05.559Z INFO [MAIN] router/main.go:169 created Knative processor {"commit": "UNKNOWN", "version": "UNKNOWN", "sink": "http://default-broker-ingress.vmware-functions.svc.cluster.local"}
2021-09-12T20:48:05.559Z WARN [METRICS] metrics/server.go:59 disabling basic auth: no authentication data provided
2021-09-12T20:48:05.559Z INFO [WEBHOOK] webhook/webhook.go:226 starting webhook server
2021-09-12T20:48:05.560Z INFO [METRICS] metrics/server.go:98 starting metrics server {"address": "http://0.0.0.0:8082/stats"}
2021-09-13T15:40:38.684Z INFO [KNATIVE] knative/knative.go:181 sending event {"eventID": "b3ba3ee9-67fe-4f58-9ac2-f4f0959bbf12", "subject": ""}
2021-09-13T15:40:38.687Z INFO [KNATIVE] knative/knative.go:193 successfully sent event {"eventID": "b3ba3ee9-67fe-4f58-9ac2-f4f0959bbf12"}
Next, we can confirm the content of the payload by looking at the broker logs. I will say, depending on your environment, this can be extremely verbose, so you may need to search through the logs for the keyword "vmware.vmc."
# k -n vmware-functions logs deployment.apps/default-broker-ingress -f
2021/09/13 15:20:32 received: Validation: valid
Context Attributes,
specversion: 1.0
type: vmware.vmc.SDDC-PROVISION.v0
source: https://vmc.vmware.com/console/sddcs/XXX
id: 709290b2-784a-4836-8aff-521f83fa2913
datacontenttype: application/json
Data,
{
"org_id": "b12345e8-48f1-4517-99fe-0bddc753e999",
"org_name": "Production",
"resource_id": "315a4b5a-25d2-476f-879e-4c6b27224e1b",
"resource": "SDDC",
"resource_name": "VEBA-SDDC",
"message_username": "*protected email*",
"message": "Deployment of SDDC"
}
Once the NGW webhook subscription has been configured, you no longer have to touch the settings. As new events are added, you will automatically get them forwarded to VEBA, where they can be consumed. To demonstrate how easy it is to consume these new VMware Cloud events, I have created two examples that will allow customer to easily get either a Slack or Microsoft Teams notification when an SDDC is provisioned.
Here is an example of what the Slack and Microsoft Teams notification would look like simply by subscribing to the custom event type: vmware.vmc.SDDC-PROVISION.v0
I am currently in the process of submitting both of these example functions to theVEBA repository and will share the links once the examples are available. Not only are these custom events easy to consume, but it will also opne up the door for more interesting use cases that can now be solved by customers without having to have direct access to VMware Cloud or NGW, which makes this a very scalable and secure solution.
Thanks for the comment!