With the rising cost of consuming AI services in the public cloud, the availability of powerful open-weight models, and growing concerns around security and data sovereignty, there was certainly a lot of interest from attendees at last week’s VMware Explore in running their own AI models within their VCF Private Cloud environments.
The timing could not have been better with the release of VMware Cloud Foundation (VCF) Private AI Services (PAIS) 3.0, which is part of the broader VCF Private AI Cloud offering and perfectly complements the latest VCF 9.1.1 release. VCF 9.1.1 also introduces a number of new AI capabilities for infrastructure administrators, including the new AI Assistant for VCF, which can leverage the same PAIS infrastructure to consume AI models running directly within your VCF Private Cloud environment.
Some key highlights in PAIS 3.0 include the new Model-as-a-Service capability, which can now be shared across different vSphere Namespaces to better utilize precious GPU resources, the ability to consume PAIS services directly from vCenter Server in addition to VCF Automation (VCFA), and support for built-in local identity for streamlined proof-of-concepts (POC)/lab deployments.
While not new, the Private AI Services (PAIS) Artifact Mirroring Tool (AMT) provides an easy way to download all the required container images for non-internet-connected or air-gapped environments. A typical workflow would involve inspecting the various deployment YAML manifests, using the imgpkg utility to export the required container images, and then manually updating the deployment manifests to reference your local container registry, hoping that everything works as expected. Since this was my first time using PAIS AMT, I thought it would be useful to document the workflow for those interested in trying out PAIS in their lab environment.
Requirements:
- Access to a local Container Registry (you can use VCF Infrastructure Services (VIS) Fling Appliance for PoC/Testing purposes) with at least 60GB of free space
- ~40-50GB storage available on your local desktop to store downloaded container images
- Docker Client installed on your local desktop
- vSphere Supervisor configured with VPC Networking using either Centralized or Distributed Transit Gateway
Step 1 - Log in to the Broadcom Support Portal (BSP) and download the following files (links included below) to your local workstation with internet connectivity:
- VCF Consumption CLI (9.1.1)
- VCF Consumption CLI Plugins (9.1.1)
- VMware Private AI Services (PAIS) (3.0)
Step 2 - Extract the VCF Consumption CLI package, ensure the binary has execute permissions, and place it in your desired directory (e.g. /usr/local/bin).
Step 3 - Extract the VCF Consumption CLI Plugins and then use the VCF Consumption CLI to install all plugins.
vcf plugin install all --local-source ./VCF-Consumption-CLI-PluginBundle-Darwin_AMD64-9.1.1.0.25665404
Step 4 - Log in to the BSP and navigate to Downloads. Click the Registry Tokens button on the right to generate a new registry token.
Step 5 - Log in to both the Broadcom Container Registry and your local container registry. You can obtain the Broadcom Container Registry base URL from the contents of the pais-3.0.0-external.yml file. In the example below, the Broadcom registry is pais-docker.packages.broadcom.com. For my local registry, I am using Harbor provided by the VIS Fling Appliance with a project named pais, resulting in a local registry URI of vis.vcf.lab:9443/pais.
Use the email address associated with your BSP account as the username and the registry token generated in Step 4 as the password.
docker login pais-docker.packages.broadcom.com
docker login vis.vcf.lab:9443/pais -u admin
Ensure that you have successfully logged in to both container registries before proceeding to the next step.
Step 6 - Run the pull command and specify the PAIS deployment manifest to download all required container images and store them locally in a tar file, which we will import into our local container registry in the next step. In this example, I will be using the NVIDIA OSS GPU Operator. If you plan to use NVIDIA AI Enterprise (NVAIE), you will also need to log in to the NVIDIA Container Registry before running the pull command so that the required NVAIE container images can be downloaded.
vcf pais amt pull pais-3.0.0-external.yml --gpu-operator-variant=oss
Depending on the speed of your connection, this can take up to 10-15 minutes and by default, all exported container images will be stored in a folder called pais-store in the current working directory.
Step 7 - Finally, transfer the pais-store directory to a system that has access to your local container registry. In my example, the system already has access to the local registry. After successfully logging in to the registry using Docker, run the push command and specify the path to the pais-store directory to begin uploading the PAIS container images to your local container registry.
vcf pais amt push vis.vcf.lab:9443/pais --local-store ./pais-store
After the upload has completed, you can view all the PAIS container images that have been pushed to your local container registry.
What is really cool about using the PAIS AMT is that the workflow automatically generates a new PAIS deployment manifest that includes the URLs for your local container registry, so you do not have to manually update or figure out the required registry URLs.
Within the pais-store directory, you will find a new deployment manifest called pais.yml, which you can use to deploy the PAIS Supervisor Service. The generated manifest will automatically reference the container images that were uploaded to your local container registry. In addition, you will find a yaml-svc-cfg.yaml file that contains the required configuration overrides, which you will need to specify when enabling the PAIS Supervisor Service.
To register PAIS as a new Supervisor Service, login to the vSphere UI and in left hand navigation select Supervisor Management > Services > Add New Service and provide the pais.yml.
Note: If you are using self-signed TLS certificate for your container registry, you will need to first import the trusted CA certificate into your vSphere Supervisor before you can enable the service. In vSphere UI, in left hand navigation select Supervisor Management > Supervisor Cluster > Configure > Container Registries to add a new TLS certificate.
To enable thee PAIS Supervisor Service, navigate to Supervisor Management > Services > Private AI Services > Actions > Manage Service and select the vSphere Supervisor Cluster to deploy the service. When prompted for the optional override, make sure to copy the contents from yaml-svc-cfg.yaml as shown in the screenshot below.
If everything was setup correctly, you should see the configured status with green checkmark next to the PAIS Supervisor Service!
Thanks for the comment!