Certificate lifecycle management is not something anyone looks forward to, it is time consuming and usually not automated. However, it is a necessity for many of our customers. The process gets even more challenging when needing replace certificates across multiple VMware products, not only careful orchestration but also properly reestablishing trust between product just adds another layer of operational complexity. Within the Integrated System Business Unit (ISBU) at VMware, which produces both the VMware Validated Design (VVD) and VMware Cloud Foundation (VCF), the team has been working on a way to simplify certificate management, not only for individual products (working with product teams) but also holistically at the VMware SDDC level.
This initially started with the development of a tool called Certificate Generation Utility (CertGen), which helps customers generate new certificates for various products within the VMware SDDC. Although it was developed for the VVD, any VMware customer who consumed products within the VVD, could also leverage this tool. We all know certificate generation can be a pain, but it is not as challenging or as complex as the actual certificate replacement process itself which is also fully documented by the VVD team here.
This is where the new Fling comes in, the SDDC Certificate Tool, which automates the manual steps outlined by the VVD and helps customers easily replace certificates that they have created (CertGen or another process) and automatically orchestrates this across the different products within the SDDC. The tool is command-line driven and uses a JSON configuration file which can contain all or a subset of the VMware SDDC products, which is great for supporting different environments and allows for easy source control. Extensive pre-checks are also built into the tool to validate the certificates themselves (e.g. expiry, chain validation, etc) also also preventing miss-match of information (e.g. SAN entries, number of nodes, etc) which then get compared against your actual environment before any changes are applied. The JSON also contains a section referred to as Service Accounts, which is merely other VMware product accounts that the tool supports to reestablish trust after replacing the certificate for given product.
One thing to be aware of for VCF customers, is that tool is already integrated into SDDC Manager as it sits outside of all the VMware products and is the key infrastructure for providing certificate lifecycle management for the VCF solution.
Here is a quick video of how the certificate replacement tool works:
The Fling currently supports the following products:
| Product | Minimum Version | Maximum Version |
|---|---|---|
| Platform Services Controller | 6.0 U2 | 6.7 |
| vCenter Server | 6.0 U2 | 6.7 |
| NSX-V | 6.2.4 | 6.4.1 |
| vRealize Log Insight | 3.6 | 4.6 |
| vRealize Operations Manager | 6.3 | 6.7 |
| vRealize Automation | 7.4 | 7.4 |
| vRealize Business for Cloud | 7.1 | 7.4 |
Note: One thing you may notice above is that ESXi is currently not supported, the reason for this is primarily due to the duration it takes to lifecycle certificates for an ESXi host as it requires evacuating VMs off the host and this can take a significant amount of time compared to replacing the certificates for all other products. If this is something you would like to see, feel free to leave a comment and the type of workflow you would like to see.
If you have any feedback or enhancements you would like to suggest, please leave a comment on the Fling page which is monitored by the developers.
This most welcomed. But its hardly a solution to VMware chalkenges in supporting certs. Isssuing certs is some what easy and trivial ONCE you know how it works.
THE problem is how cert support changes and is different for every VMware product and/or version of a product.
As an example, vSphere 6.5 certs support some feature, which 6.7 does not support. The upgrade path is to remove the certs on 6.5 and then upgrade to 6.7. Now certs can again be implemented, but with the lesser feature set supported by vSphere 6.7.
Michael,
Yup, completely agree that having different cert format as well as different processes between product is a huge problem. All I can say is there's been some great progress made on this front, things like this takes time (trust me, I've been pushing on this for some time) but I think you'll start to see all fo this come together real soon. Part of the challenge is also not having proper API surfaces which then forces backwards compat, some of this is via CLI or worse UI-only, so that's also another area that we've been focusing on AND ensuring a good UX. In the short term, this tool is to help address some of the individual product limitations/challenges while a longer term solution is being worked on. I believe there'll be some sessions at VMworld talking about this in greater detail.
Regarding 6.5 to 6.7 cert differences, do you happen to know what functionality was removed or wasn't supported? How did you generate the certs, any details you can share, I can forward to the right folks to make sure things like this don't happen again.
William i'm very much looking forward to this 😊
You cannot specify multiple aliases in vSphere 6.7 (CNAMES if a recall correctly). So you have to choose between vc shortname, fqdn and ip. In vSphere 6.5 and earliere you could provide multiple with some tinkering.
Too many Certificate Management tools from VMware, why can't VMware just integrate all these tools into PSC VMCA, and have a decent broker between VMCA & VECS. Then Store all certs in VMDIR for replication and pushed them out to VECS and end points NSX, ESXi.
Something like, Netflix Lumar, https://medium.com/netflix-techblog/introducing-lemur-ceae8830f621 can management certificate request CSR & private keys.
Surely, that would be the way to go. A very good broker on PSC.
Please for the love of God, NO, NO NO, stop this madness with certificate management. WE DO NOT NEED ANOTHER CLI TOOL.
Thanks for the feedback. As I've said to Michael, we've been making some good progress in this area (yes, its taken us much longer than I had even hoped as this is something I and many others have raised). Having a common framework will be key to support multiple products (current and even future) across the VMware SDDC. Regarding the use of VMCA, there's a few challenges there. Not all customers are or want to use VMCA, we don't want to force that as a requirement. Not all products need to talk to VC directly, so this would then add another level of dependency on VC/PSC which isn't great if VC is unavailable. Workflows, especially re-trust at multiple levels would get quite complicated. Like Lumar, an external solution would be the ideal approach that way if the individual products you're replacing have issues, you're not affected. This is why SDDC Manager (which runs as a separate VM) has this fully integrated and handles not only CSR/Private keys but also the replacement. In addition to the broker, we also need to ensure each product exposes the correct primitives (e.g. APIs) to perform the various certificate management operations, we've only been talking about CSR/private key and replacement, but there's plenty more to do such as logging/auditing, RBAC, health checks, expiry notification, etc. I'll be sure to point our PM/Engr teams to the feedback you've shared, but look at the Fling as a way for you to provide us feedback on the type of experience you would like to see (remember, Flings may or may not ever get released as a product and it may also evolve based on customer feedback). Most customers prefer Automation for Certificate management, some may want a UI, again, give the tool a try and let us know what you think.
Certificate are not hard, cert, private key, and chain. If the issues comes with install, and going management. Java SSL Keystore is not pleasant to work with.
I can definitely understand where are you coming from with API and automation, this would make a huge difference. My concern with a central vm for certificate generation is ENTROPY, and be able to monitor this.
The PSC/VC is critical and the heart of all operations, if it is down, certificate expiry notification and renewal is the last thing on my mind.
Hi William,
Will there be support for vRealize Suite, Horizon View and SRM? Is that planned?
Thanks
Dameer
Hi willian, I get this following error while generating the CSR file using SDDC certificate tool fling
root@vcsa [ / ]# java -jar lib/certreplace-*.jar -config config/config.json -createlocalcsr -passwordEntry
Error: Unable to access jarfile lib/certreplace-*.jar
Hi Willam. What about vRealize Orchestrator, vRealize Buisenes?
Hi, when will there be an update including support for vSphere 7 and the new versions of the other product?
For feedback on Flings, please leave a message on the specific Fling site