In this blog post, we will setup an AI Model Store using Harbor, which will allow us to download Hugging Face models and upload that to our private Harbor registry for use with VMware for Private AI Services (PAIS).
Requirements:
- VM to install Harbor or an existing Harbor instance
- Docker Client installed on your local desktop
- Hugging Face CLI installed on your local desktop (Refer to the official documentation for installation steps)
- VCF CLI installed on your local desktop (Refer to the official documentation for installation steps)
Step 1 - Install Harbor using HTTPS, which you can refer to the official documentation for installation steps. Depending on how many AI models you plan to download, you may want to increase the filesystem to ensure you have enough storage capacity as some AI models can get quite large.
Note: While you can use the Harbor vSphere Supervisor Service, I opted for an external Harbor instance as I can easily backup the VM by exporting it as an OVA, useful for when needing to rebuild my environment and not having to setup Harbor and re-upload content.
Step 2 - Login to Harbor Admin UI and create a new project that will be used to host your AI models. In my example, I have named my project models and is configured as a public repository, meaning I will not require authentication to pull the AI models. If you decide to add authentication, then you will need to ensure a pull secret is configured when deploying your model endpoint.
When pushing or pulling your AI model, the URI will be Harbor FQDN and then followed by the project name, model name and model tag(e.g. registry.vcf.lab/models/MODEL_NAME:MODEL_TAG)
Step 3 - Login to your Habor registry using the Docker Client and the credentials that have upload permission.
docker login registry.vcf.lab/models -u admin -p VMware1!
Step 4 - Download the root CA certificate (setup from Step 1) from your Harbor registry and add that to your desktop certificate trust store. This is required as VCF CLI will need to establish a trust between the local system that will perform the upload of the AI model with Harbor.
- For macOS, refer to these instructions to add CA certificate to macOS keychain
- For Linux, refer to these instructions to add CA certificate to Linux certificate store
- For Windows, refer to these instructions to add CA certificate to Windows certificate store
Note: If this step was not performed correctly, you will run into Issue #2 as outlined in this blog post when attempting to push an AI model to Harbor
Step 5 - Identify the Hugging Face AI models that you wish to download and copy the name ID next to the model
In my setup, I am using the smaller models due to resources constraints:
- https://huggingface.co/BAAI/bge-small-en-v1.5 (will be used for Embeddings model endpoint)
- https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0 (will be used for Completions model endpoint)
Using the Hugging Face CLI, we can download these two models and place them into a directory by running the following commands:
mkdir bge-small-en-v1.5 huggingface-cli download BAAI/bge-small-en-v1.5 --local-dir bge-small-en-v1.5 mkdir tinyllama-1.1b-chat-v1.0 huggingface-cli download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir tinyllama-1.1b-chat-v1.0
Step 6 - Lastly, we can now push our AI models that we have downloaded into our private Harbor registry. You will need to first change into the directory of the AI model folder BEFORE you push! I made the mistake of forgetting to do that and ran into issues later on when deploying the model endpoint. I am hoping we can enhance the VCF CLI as there are no errors thrown and the output made it seem like the AI model was uploaded successfully.
When you provide a name for the AI model to be uploaded, make sure it follows DNS naming convention (e.g. lowercase, no spaces, etc) and you will need to also provide a tag for the model.
cd bge-small-en-v1.5 vcf pais models push --modelName baai/bge-small-en-v1.5 --modelStore registry.vcf.lab/models -t v5 cd tinyllama-1.1b-chat-v1.0 vcf pais models push --modelName tinyllama/tinyllama-1.1b-chat-v1.0 --modelStore registry.vcf.lab/models -t v1
If everything was setup correctly, after a few minutes, you should find the new AI models that you have downloaded is now hosted within your Harbor instance, ready to be served and consumed privately within your infrastructure!
Thanks for the comment!