VCF 9.1.1 - Deploying Your Own AI Models with VCF Private AI Services (PAIS) 3.0

My recent blog post on the new AI Assistant for VCF in VCF Operations 9.1.1 received a lot of positive feedback from both internal and external readers, especially from those who were not aware that this capability was available and were excited to try it out. The excitement is understandable, as this opens up many interesting possibilities for infrastructure administrators to operate and manage their environments more efficiently.

In that article, I focused on how to enable AI Assistant for VCF and quickly get started using Google Gemini as a cloud-based LLM provider. While that is certainly the fastest way to start exploring the capability, AI Assistant for VCF also supports local AI models deployed using VCF Private AI Services (PAIS) 3.0, enabling customers to take advantage of powerful AI models running securely and more cost-effectively within their own on-premises environment.

In this blog post, we will now focus on deploying and consuming local AI models with PAIS. This builds on another recent article where I walked through the setup of VCF Private AI Services (PAIS) using the Artifact Mirroring Tool (AMT) for air-gapped environments. 

Requirements:

• vSphere Supervisor configured with VPC Networking using either Centralized Transit Gateway (CTGW) or Distributed Transit Gateway (DTGW)
• VCF Private AI Services (PAIS) Supervisor Service enabled (see HERE more details)
• VCF Consumption CLI installed on your local desktop (see HERE more details)
• Docker Client installed on your local desktop
• Hugging Face (HF) CLI installed on your local desktop
• NVIDIA GPU for use with ESX DirectPath I/O or vGPU
• Access to a local Harbor registry (you can use VCF Infrastructure Services (VIS) Fling Appliance for PoC/Testing purposes) with sufficient storage for your desired models

Step 1 - Download your desired AI models from Hugging Face using the HF CLI. In the example below, I am downloading two GGUF models for use with llama.cpp engine: Gemma 4 26B GGUF and Qwen3.8 27B GGUF.

mkdir -p gemma4-26b-gguf
hf download bartowski/google_gemma-4-26B-A4B-it-GGUF --include "*Q4_K_M*.gguf" --local-dir gemma4-26b-gguf
mkdir -p qwen3.8-27b
hf download unsloth/Qwen3.8-27B-GGUF --local-dir qwen3.8-27b

Step 2 - Upload your AI model to your Harbor registry by changing into the directory containing the downloaded model files. Use the VCF Consumption CLI to push the model, specifying the desired model name, local registry endpoint (e.g. vis.vcf.lab:9443/models), and model version.

cd gemma4-26b-gguf
vcf pais models push --modelName bartowski/gemma4-26b-gguf --modelStore vis.vcf.lab:9443/models -t v1.0
cd qwen3.8-27b
vcf pais models push --modelName unsloth/qwen3.8-27b-gguf --modelStore vis.vcf.lab:9443/models -t v1.0

In my example, I have a project in Harbor called models, which will host all of my AI models. As shown below, the model name becomes part of the registry path. For example, to deploy the Qwen3.8 27B model, the full model URI is vis.vcf.lab:9443/models/unsloth/qwen3.8-27b-gguf:v1.0. Keep this URI format handy, as we will need it when deploying the model runtime.

Step 3 - Enable DirectPath I/O on the ESXi host that contains the physical NVIDIA GPU that will be used to deploy your AI model

Step 4 - Create a new vSphere Namespace that will be used to configure the PAIS service and run your AI models. Log in to the vSphere UI and, from the left-hand navigation, select Supervisor Management > Namespaces and create a new namespace (e.g. pais).

In addition to configuring the desired vSphere Storage Policy for the namespace, create a new VM Class that includes the desired vCPU, memory, and DirectPath I/O device that was configured in the previous step. In my example, I created a VM Class called nvidia-a2000-rtx with 8 vCPUs, 32GB of reserved memory, which is required when using DirectPath I/O, and the NVIDIA RTX A2000 installed in my Minisforum MS-A2. For the VKS Control Plane nodes that is provisioned by PAIS service, you can use best-effort-medium VM Class or larger based on your intended workload.

Step 5 - We are now ready to deploy the PAIS service, which will allow us to deploy our AI models and expose them as OpenAI-compatible model endpoints. In the vSphere UI, navigate to your vSphere Namespace under Supervisor Management > Namespaces > Namespace > Resources. From here, we will use the Local Consumption UI built into vCenter Server to deploy both the PAIS service and our AI models.

Select the Model Runtime Only option and then choose the default Storage Class and VM Class to use for the VKS Control Plane for running the PAIS service.

For users to access the PAIS service, you can either connect to an OIDC provider that supports the Authorization Code grant flow with PKCE (Proof Key for Code Exchange), which I have previously demonstrated using Keycloak, or use the built-in local user capability introduced in PAIS 3.0. For this example, we will use the new local user option and there is an additional step required today to enable API token generation, see the next step for more details.

Toggle Local User Accounts and add your desired user accounts and credentials.

Note: The Issuer Base URL field must be populated with a valid HTTPS URL due to a known UI quirk. When deploying PAIS using the YAML manifest instead, this field can be left as an empty string.

Under GPU Configuration, select the passthrough option. If you intend to deploy both CPU- and GPU-based models, make sure to select one of the GPU options, as the GPU Operator will not be deployed when only CPU support is selected.

Note: While not in scope for this blog post, you can override the desired GPU Operator version as well use your own container registry for air-gapped deployments.

A summary of your configuration will be provided at the end of the PAIS configuration wizard. Once the deployment begins, it can take a few minutes to complete. You can monitor the Status field, and once it shows Ready, the PAIS service has been successfully deployed.

Step 6 - After the PAIS service has been deployed, we can retrieve the ingress IP address assigned from our configured VPC networking and create a DNS entry to provide a friendly FQDN for accessing the PAIS OpenAI-compatible model endpoint. This endpoint is how we will interact with the different AI models that we deploy.

Currently, configuring an FQDN for the PAIS Model Endpoint requires patching the PAISConfiguration resource with an annotation containing the desired FQDN. This requires using kubectl to connect to the vSphere Supervisor and update the PAISConfiguration resource. If you have selected the local user option, we also need to update the resource to add support for API Token generation.

Create a new kubernetes context for connecting to our vSphere Supervisor by running the following command:

vcf context create sv01 --endpoint=sv01.vcf.lab --username administrator[at]vsphere[dot]local --insecure-skip-tls-verify --auth-type basic
vcf context use sv01

Next, we will edit our PAISConfiguration resource by running the following command:

kubectl -n pais edit paisconfiguration default

Add the paisconfigurations.pais.vmware.com/cname annotation with value of the FQDN that you had created in DNS and the apiTokens section for local user account.

apiTokens:
   expirationDefault: 720h
   expirationMax: 8760h

The update is pretty quick and if you refresh the PAIS service in the vSphere UI, we should now see the FQDN has been populated with your configured value.

Step 7 - To verify that the PAIS Model Endpoint is accessible and that we can generate an API token using our local user account, open an incognito browser window and navigate to the configured FQDN. You should see a login screen where you can enter the username and password for the local user account created in Step 5.

Once logged in, click your username in the upper-right corner and select API Tokens. You should see a screen similar to the one shown below, where you can generate a new API token. Generate a token and save it, as we will use it to verify connectivity to the PAIS Model Endpoint.

Step 8 - To deploy an AI model endpoint, navigate to the Model Runtime tab and click New Model Endpoint to begin the deployment wizard.

For the Model URL, specify the model registry URI from the previous step (e.g. vis.vcf.lab:9443/models/unsloth/qwen3.8-27b-gguf.0). For the registry credentials, select the pais-pais-reg-creds* secret, which will be used to authenticate with the Harbor registry and pull the model.

The Routing Name is the model name that will be returned when listing the available models from the PAIS Model Endpoint, while the Display Name is simply a human-friendly label for the model.

Next, select the desired Model Type and Model Engine. It is important to understand that out of the box, PAIS 3.0 ships with three engines with the following configuration:

If you are using a model that requires LlamaCPP and need GPU acceleration, then you will need to override the model engine by toggling Advanced Configuration, which will allow you to specify a custom container image and additional CLI arguments. If the container image is hosted in your own registry, you can also specify the appropriate registry secret, as shown in the example below.

In my setup, I am deploying a llama.cpp-based model, so I need to override the default llama model engine image with a CUDA-enabled image such as ghcr.io/ggml-org/llama.cpp to ensure GPU acceleration is used when serving the model.

The default NVIDIA GPU Operator version deployed by PAIS is 25.10.1, with support for overriding the version up to 26.3.1. Currently, this override is not exposed through the PAIS service configuration UI and must be configured by editing the PAISConfiguration.spec.nvidiaConfig.driver.version property.

Finally, configure the VM Class, which can either be a general CPU-based VM Class or the custom GPU-based VM Class that you configured in Step 4. In my example, I will select the GPU-based VM Class that includes my NVIDIA GPU configured for DirectPath I/O.

Since the VKS Control Plane is already running from the initial PAIS service deployment in Step 5, deploying the model is relatively quick. PAIS simply needs to deploy a new VKS Worker Node, attach the desired GPU, and then automatically deploy and configure the appropriate GPU Operator depending on whether you are using NVIDIA AI Enterprise (NVAIE) or NVIDIA OSS.

A large portion of the deployment time will be spent downloading the AI model from your registry to the VKS Worker Node, especially for larger models. The best way to monitor the deployment progress is to obtain the Kubernetes context for the underlying VKS Cluster. From there, you can monitor the GPU Operator deployment, model download, and the PAIS Model Endpoint as it is being deployed.

Refresh your vSphere Supervisor kubernetes context in case the token has expired (see Step 6 if you need to create the context)

vcf context refresh sv01
vcf context use sv01

We need to retrieve the VKS Cluster label by running the following command:

kubectl -n pais get cluster

Create a kubernetes context to the VKS Cluster by running the following command and specify your vSphere Namespace and VKS Cluster label from previous command:

vcf context create vks --endpoint=sv01.vcf.lab --username administrator[at]vsphere[dot]local --insecure-skip-tls-verify --auth-type basic --type k8s --workload-cluster-namespace pais --workload-cluster-name pais-74dba7db
vcf context use vks:pais-74dba7db

Once you have the context to VKS Cluster, we can run the following command and see where things are at whether it is the GPU operator or PAIS Model Endpoint being deployed:

kubectl get pods -A

The PAIS Model Endpoint is ready when all pods are in a running or completed state. You can also look at the logs for GPU Operator driver or model endpoint pod with the following:

# kubectl -n gpu-operator logs nvidia-gpu-driver-ubuntu24.04-864f5dfc8-l6qtp

DRIVER_ARCH is x86_64

========== NVIDIA Software Installer ==========

Starting installation of NVIDIA driver version 580.105.08 for Linux kernel version 6.8.0-117-generic

Unloading NVIDIA driver kernel modules...
Unmounting NVIDIA driver rootfs...
Updating the package cache...
Resolving Linux kernel version...
Proceeding with Linux kernel version 6.8.0-117-generic
Installing Linux kernel headers...
Installing Linux kernel module files...
Generating Linux kernel version string...
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 580.105.08....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: Unable to determine the default X library path. The path /tmp/null/lib will be used, but this path was not detected in the ldconfig(8) cache, and no directory exists at this path, so it is likely that libraries installed there will not be found by the loader.


WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.


WARNING: Unable to find a suitable destination to install 32-bit compatibility libraries. Your system may not be set up for 32-bit compatibility. 32-bit compatibility files will not be installed; if you wish to install them, re-run the installation and set a valid directory with the --compat32-libdir option.

Parsing kernel module parameters...
Configuring the following firmware search path in '/sys/module/firmware_class/parameters/path': /run/nvidia/driver/lib/firmware
Loading ipmi and i2c_core kernel modules...
Loading NVIDIA driver kernel modules...
+ modprobe nvidia NVreg_CoherentGPUMemoryMode=driver
+ modprobe nvidia-uvm
+ modprobe nvidia-modeset
+ set +o xtrace -o nounset
Starting NVIDIA persistence daemon...
Mounting NVIDIA driver rootfs...
Done, now waiting for signal

# kubectl -n pais logs pais-modelendpoint-7bf7d68c-1314-461b-ab48-4a2c37a95f98-cdnx6r4
0.00.206.775 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.207.248 W srv  llama_server: -----------------
0.00.207.254 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.207.254 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.207.255 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.207.255 W srv  llama_server: -----------------
0.00.208.630 I srv    load_model: loading model '/data/model/Qwen3.8-27B-UD-Q4_K_S.gguf'
0.01.387.888 W model has unused tensor blk.64.attn_norm.weight (size = 20480 bytes) -- ignoring
0.01.387.899 W model has unused tensor blk.64.post_attention_norm.weight (size = 20480 bytes) -- ignoring
0.01.387.907 W model has unused tensor blk.64.attn_q.weight (size = 51609600 bytes) -- ignoring
0.01.387.909 W model has unused tensor blk.64.attn_k.weight (size = 5570560 bytes) -- ignoring
0.01.387.912 W model has unused tensor blk.64.attn_v.weight (size = 5570560 bytes) -- ignoring
0.01.387.921 W model has unused tensor blk.64.attn_output.weight (size = 25804800 bytes) -- ignoring
0.01.387.924 W model has unused tensor blk.64.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.01.387.926 W model has unused tensor blk.64.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.01.387.929 W model has unused tensor blk.64.ffn_gate.weight (size = 73113600 bytes) -- ignoring
0.01.387.932 W model has unused tensor blk.64.ffn_down.weight (size = 73113600 bytes) -- ignoring
0.01.387.935 W model has unused tensor blk.64.ffn_up.weight (size = 73113600 bytes) -- ignoring
0.01.387.939 W model has unused tensor blk.64.nextn.eh_proj.weight (size = 43008000 bytes) -- ignoring
0.01.387.942 W model has unused tensor blk.64.nextn.enorm.weight (size = 20480 bytes) -- ignoring
0.01.387.945 W model has unused tensor blk.64.nextn.hnorm.weight (size = 20480 bytes) -- ignoring
0.01.387.953 W model has unused tensor blk.64.nextn.shared_head_norm.weight (size = 20480 bytes) -- ignoring
0.05.276.127 I cmn          init: llama threadpool init, n_threads = 8
0.05.532.822 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 60160, kv_unified = 'true'
0.05.538.933 W srv          init: chat template supports preserving reasoning, it is enabled by default (may use more tokens, disable via --no-reasoning-preserve)
0.05.538.994 I srv  llama_server: model loaded
0.05.539.004 I srv  llama_server: listening on http://0.0.0.0:3080

Once our PAIS Model Endpoint is ready, we can head back into vSphere UI and select the deployed model and we will see a summary of our model deployment along with a sample API request to our OpenAI endpoint which will be: https://model.vcf.lab/api/v1/compatibility/openai/v1

Use the API token that you had generated from Step 7 and set it to PAIS_API_KEY environment variable and copy the curl command (you will need to add -k for self-signed TLS) and if everything was setup correctly, you should get back a response from the AI model!

We can also list the available models being served by our PAIS Model Endpoint by performing a GET on the following URL:

curl -k -X GET "https://model.vcf.lab/api/v1/compatibility/openai/v1/models" \
  -H "Authorization: Bearer $PAIS_API_KEY" \
  -H "Content-Type: application/json"

If you only have a single model being served, you will get a single response but if you deployed additional model endpoints, then this single PAIS Model Endpoint URL will return all models that an OpenAI client can consume.

For those interested in non-UI workflwos, I have also published working examples of both the PAIS service configuration and model endpoint deployment manifests which means once the basic infrastructure (VM Classes, Registries, etc) have been setup, you can get everything up and running with just two YAML manifests!

• paisconfiguration-local-user.yaml
• llamacpp-qwen3.8-27b-gguf-modelendpoint.yml
• llamacpp-gemma4-27b-gguf-modelendpoint.yml
• vllm-qwen2.5-14b-modelendpoint.yml