WilliamLam.com

Modding an NVIDIA RTX 4000 Ada (20GB VRAM) into Minisforum MS-A2

by // 2 Comments

The Minisforum MS-A2 has been my go to platform for running an optimized and fully functional VMware Cloud Foundation (VCF) 9.0 lab environment. From vSAN ESA to NSX VPC to vSphere Kubernetes Service to VCF Automation to Data Services Manager, it runs it all!

Over the past couple of weeks, I had also been experimenting with running VMware Private AI Services (PAIS) using the MS-A2. For those with an eye for detail, may have noticed that the NVIDIA GPU requirement was actually being satisfied by an ASUS NUC 14 Performance, which includes an NVIDIA GeForce RTX 4070 (8GB VRAM). The ASUS NUC was purely for validation and prototyping purposes to ensure that I could actually run PAIS before investing in a more capable and certainly more expensive NVIDIA GPU šŸ¤‘

One of the lesser known capabilities of the MS-A2 is the ability to add a half-height PCIe device and from the very beginning when I had first heard about the MS-A2, my plan and hope was to be able to add an NVIDIA GPU to the system!

[Read more...]

MS-A2 VCF 9.0 Lab: Deploying Model Endpoint with DirectPath I/O using VMware for Private AI Services (PAIS)

by // Leave a Comment

In this final blog post, we will now deploy several AI model endpoints (downloaded from Hugging Face), configure our private data source which can be a shared location (Google Drive, Confluence, Microsoft Sharepoint, or S3-compatible endpoint) or using local filesĀ and then consuming them using an AI Agent that is built using VMware for Private AI Services (PAIS).

As mentioned in the very first blog post of this mini-series, my goal was to get hands experience with PAIS but without the need to have anĀ NVIDIA GPU capable of vGPU, which would also require an NVIDIA AI for Enterprise (NVAIE) license.

Luckily, we can use an NVIDIA GPU via DirectPath I/O, thanks to the backend plumbing the PAIS Engineering team have built and had shared with me šŸ˜Š

For my proof of concept, I am using an ASUS NUC 14 Performance, which has an NVIDIA GeForce RTX 4070 mobile GPU (8GB VRAM). The ASUS NUC 14 is running alongside my Minisforum MS-A2 setup, is only used to deploy the completions model endpoint. The use of the ASUS NUC 14 is purely for prototyping and experimentation purposes to demonstrate that anyone can play with PAIS within their lab environment. I plan to use a more powerful NVIDIA GPU setup, which I will share more details at a later point for those interested.

References:

Requirements:

[Read more...]

Side-loading VCF binaries into VCF Installer & SDDC Manager for Air-Gapped EnvironmentsĀ 

by // Leave a Comment

The VMware Cloud Foundation (VCF) Installer (Day 0) and SDDC Manager (Day N) supports two common methods for downloading VCF software into a users environment.

  1. Connect to Broadcom's online depot (supports network proxy) and directly download the software into VCF Installer/SDDC Manager, which I will describe as Online Depot with/Automated Transfer
  2. Mirror Broadcom's online portal and create an offline depot used by VCF Installer/SDDC Manager to download the software, which I will describe as Offline Depot with/Automated Transfer

To help illustrate the two options, I have create the following visual:


Option 1: VCF Installer/SDDC Manager connects directly to Broadcom.com to download VCF software

Option 2: VCF Download Tool (VCFDT) is used to connect to Broadcom.com and download a mirror copy of the online depot and then transfer that to an offline depot (web server) that is then used by VCF Installer/SDDC Manager

Option 2 is the preferred choice by most organizations as it only requires a single online download of the VCF software and then it can be efficiently hosted and replicated within their datacenter(s). Many organizations also have an established process in distributing other IT/Enterprise software (Windows updates/patches, Linux updates/patches, etc), which are typically hosted on an internal web server and Option 2 simply adheres to their existing operating model.

In the off chance, your organization does not have to distribute other IT/Enterprise software or the ability to request or setup a web server, VCF Installer/SDDC Manager actually supports a 3rd option which I will describe as Local Depot w/Manual Transfer

[Read more...]