I recently found out that my vCenter Server Appliance (VCSA) scheduled backups had been failing for some time but I was not aware. While there is a default vCenter Server alarm for failed VCSA backups, however it is only visible when you have selected the vCenter Server inventory object in left hand navigation, which is not a typical item that I select as shown in the screenshot below.
Furthermore, I was actually logged into the VCSA's VMware Appliance Management Interface (VAMI) not too long ago while updating to a newer patch release and I had checked to see if backups were successful, which I thought they were when looking at the "Status" column.
It was not until I took a closer look, did I realize that the schedule backup activity log is ordered from oldest to newest, meaning what I saw was a backup that has happend a long time ago and not necessary the latest backup attempt. I found this to be a really strange UX since as you would need to page to very end to check whether the latest backups were succssful or not.
Note: I have already filed an internal bug to ask for the VCSA schedule backup activity to be ordered from newest to oldest, so users can quickly see if the latest backups are successful or not.
In addition to going to vSphere UI to see the current VCSA backup status, which is pulled from VAMI, you can also use the VAMI API for retrieving this information. Using the VAMI API, you will need to use the Appliance Backup Job Details REST API, which will return all VCSA backup job activity and you can then look at the last entry to determine if the latest backup was successful or not.
As many of our users typically prefer PowerShell or PowerCLI examples, I wanted to try out the auto-generated vSphere REST API PowerCLI cmdlet to access this function, which is called Invoke-ListBackupJobDetails. The other nice thing about the auto-generated cmdlet to use the VAMI API, it is just a single command and simply piping the output to selecting the last item as shown in example below:
(Invoke-ListBackupJobDetails | Get-Member -MemberType NoteProperty | Select -Last 1).Definition
As you can see from the output, we can see that the status field for this backup has failed which is the same information shown in the vSphere and VAMI UI respectively.
A lesson for everyone, make sure your backups are occurring and that they are successful. You do not want a surprise when you need to recover and realize you have no recent backups. This is also true for testing your backups periodically to ensure everything is working as expected.
Ideally, when the first VCSA backup failed, we would get notified immediately, which vCenter Server can enable when using email alarm. However, if you wish to subscribe to this event and do something more advanced like event-driven automation using VMware Event Broker Appliance (VEBA), then you can subscribe to this vCenter Server event called com.vmware.applmgmt.backup.job.failed.event and if you want to also get notified when a backup is successful, then you can subscribe to com.vmware.applmgmt.backup.job.finished.event
“ I have already filed an internal bug to ask for the VCSA schedule backup activity to be ordered from newest to oldest”
Ordering items in a list was never VMware’s strength. I recall unordered lists from year 2012 when I have started working for VMware. We write 2024 now and unordered lists still haunt me in my day job working with NSX. They are not just presenting you with unordered lists in the age of AI but ordering is not possible at all. Sorry, I had to vent
It's been a bug 'at least to me' for a long time. Every time you do an upgrade where an new appliance gets stood up, the backup configuration isn't copied over and no notification is given. I've been told that's as designed. If you know someone, it would be nice to see this behavior fixed.
It's insane. Why aren't vCenter roles sorted when you are assigning them to a user/group?
Note that the API call requires the user to be in the local admin group... which is annoying for monitoring, where you don't want your user to be able to alter anything.
Not true if you’re using event driven automation in vCenter, just RO role is needed … especially powerful when combined w/VEBA
Indeed, but not very practical for service providers, as it would mean installing and maintaining a VEBA for each vCenter.
Also, despite a good solution, it's not really a drop-in replacement 🙂
VEBA can be deployed as an appliance or k8s app. The latter would be better fit for large scale env or multi-VC env which we’ve got users doing
Does no one look at the date on the backup? It's a simple click on the column to reorder to newest first. It would be nice, and proper, to have it default to newest, but it really isn't more than a minor annoyance.
Think bigger: worldwide enterprises.
Cool, thanks for sharing. This will speed up my checks.
This is a great article! However, I'm a bit confused as I have a specific question. I'm looking to monitor WAMI backup jobs using a vCenter alarm, vRealize Operations, or a PowerCLI script. Could you suggest the simplest method or point me to an article that covers this?
I brought this up to a product manager before and they were like, it's not there? Simplest thing would be to have a vCenter alarm automatically fire if no backups have been made in the last 24 hours. User can adjust threshold or disable alarm, but on by default. Simple.
Why doesn't Aria Operations have an alert for this? (or am I just blind and cannot find it?)