Like many of our users, I was excited to hear that VMware Private AI Services (PAIS) will now be included as part of VMware Cloud Foundation (VCF), I already had some ideas brewing in my head and I definitely needed to get some hands on!
While setting up some of the requirements for PAIS, I ran into a couple of issues that revolved around the use of self-signed TLS certificates, which is probably common for many of you, especially in a lab/proof of concept environment.
I spent good chunk of the day debugging the issues, which was not even the worse part, but it was the error messages that we saw. The error messages was not from the product code, but rather the underlying libraries that it relies upon and if you try to interpret the message as-is, you could go down a rabbit hole.
PAIS Engineering is already aware of the issues I ran into, so these will be enhanced in future updates but I did want to share the scenarios in case you run into them while deploying PAIS in your environment.
Issue #1 - After deploying the PAISConfiguration, the PAIS API container has the following in the logs:
Message: api: MinimumReplicasUnavailable: Deployment does not have minimum availability.: ReplicaSet "pais-api-cfc35286-46e6-43ac-bd67-239961c03189-6c4bc5f774" is progressing. pais-api-cfc35286-46e6-43ac-bd67-239961c03189-56cc59f46c-v6dk2/main terminated with {ExitCode:1 Signal:0 Reason:Error Message:Unexpected error in program (<class 'httpx.ConnectError'>): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'auth.vcf.lab'. (_ssl.c:1006)
While a hostname mismatch in a self-signed TLS certificate is a real possibility, in my case, they matched up correctly. We eventually discovered, there were differences in how Golang and Python was validating a certificate. It looks like Python "certifi" expects the TLS certificate to include the subjectAltName property and because that was not included, the underlying library threw error but it was difficult to debug as there were several layers to check.
Issue #2 - When I attempted to push a model that I had downloaded into Harbor, which is the model store, it threw the following error:
Error: pushing: Head "https://registry.vcf.lab/v2/models/tinyllama/tinyllama-1.1b-chat-v1.0/manifests/sha256:1ebd5838920fdccc02d1778bf4228a7e1551b795a3a55bfad777ed5fb4aeddcc": tls: failed to verify certificate: x509: “registry.vcf.lab” certificate is not standards compliant
I was running a standalone Harbor instance with and self-signed TLS certificate and I had already been successful in using it as a container registry. To workaround the insecure registry issue that you would face when using Docker to login, I had added my registry to the Docker Engine configuration file override "insecure-registries": ["registry.vcf.lab"] via Docker Desktop. While this was sufficient for use with Docker, when pushing a model, you need to use the VCF command-line tool and the PAIS plugin. As part of this flow, a trust must be setup between the CLI and registry and Docker workaround would not apply. Luckily, the fix is easy which is simply adding the Harbor self-signed TLS certificate into macOS keychain, similiar to trusting a self-signed TLS certificate within the browser. Again, the workaround is simple and expected, but its the error message that threw me off as it mentions the certificate was not compliant. If you are using Windows or Linux, you will need to apply a simliar workaround by adding the certificate to the local trust store.
My take away here is that even if you have good error messages, the underlying libraries that you use and depend on may still not and while I have seen many variations of self-signed TLS trust issues, I think some of the errors have actually gotten worse, not better ... just my 2cents!
Thanks for the comment!