If you recall a few weeks back, I had shared a custom vGhetto Content Library that I had created that contains a collection of my Nested ESXi and VSAN OVF Templates that anyone with a vSphere 6.0 or vCloud Director 5.x environment could subscribed to. This custom Content Library is also known as a 3rd Party Content Library and is one of the capabilities that I am most excited about with the new Content Library feature in vSphere 6.0. The reason I think this is such a cool capability is that I believe it opens up a wide range of use cases not only for our customers but also for our partner eco-systems enabling ease of content management and content distribution for VMware customers.
Just imagine, a vendor who might be providing Virtual Appliances (OVF/OVA), ISOs, VIBs, patches, workflows, blueprints, scripts, etc. can host a 3rd Party Content Library and as a customer you can easily subscribe to a particular repository. Once subscribed, you can either have the content automatically downloaded or synchronized on-demand. In fact, customers can also create their own 3rd party Content Library and distribute their own content within their own organization (local and remote sites) and even be able to share it with others in the VMware community.
Before I jump into the process of creating your own 3rd Party Content Library, I wanted to quickly go over the different options for publishing and subscribing to a Content Library. Below is a diagram to help you visualize the different options you have for publisher and subscriber sources.
Here is the same information as shown in the diagram but in a table format:
| Publisher | Subscriber |
|---|---|
| vCenter Server | vCenter Server |
| vCloud Director | vCenter Server |
| 3rd Party | vCenter Server |
| 3rd Party | vCloud Director |
Note1: Although the diagram technically shows a 5th scenario where a vCenter Server is the publisher and vCloud Director is the Subscriber, this particular flow is not entirely functional at the moment. Though it will work, it will only allow you to sync non-VM Templates and a different version of the script would be needed. This is a known issue and hopefully it will be resolved in a future update within vCloud Director.
Note2: For 2nd scenario where vCloud Director is a Publisher and vCenter Server is a Subscriber, On-Demand synchronization and overriding an existing OVF Library is currently not supported.
Here is the high level workflow for creating your own 3rd Party Content Library:
- Create directory structure layout for your content
- Create the JSON metadata index
- Publish the 3rd Party Content Library using HTTP(s)
- (Optional) Periodically update the JSON metadata index on new or updated content
Step 1 - Here is an example of what your directory structure layout should look like:
Step 2 - To assist with the JSON metadata indexing, Eric Cao, one of the Content Library Engineers was kind enough to provide the following Python script called make_vcsp_2018.py which you will need to download onto a system which has the Python interpreter running
UPDATE (07/26/18) - You can also create 3rd party content library directly on Amazon S3 or Microsoft Azure Blob using the updated versions of the script make_vcsap_2018.py and make_vcsap_2022.py respectively
UPDATE (05/29/17) - Thanks to Eric, the script now supports both Python2/3 as well as a new 4th optional command-line argument for performing md5check (Default: true)
Step 3 - The make_vcsp_2018.py script accepts two arguments: the first is the display name of the Content Library and the second argument is the path to the root directory of your Content Library. Here is an example:
python make_vcsp_2018.py vghetto-content-library /Users/lamw/desktop/vghetto-content-library
Once the script has finished building the metadata index, we can now take a look at our content and you will notice some additional files that have been added to your Content Library directory as seen in the screenshot below:
You will notice two new files at the root directory of your 3rd Party Content Library. The first is the items.json which provides reference to all the files within the library and you will see that each subsequent directory will also contain an individual item.json file describing the content within the local directories. The lib.json is the 3rd Party Content Library endpoint and is what you will be specify when subscribing to a 3rd Party Content Catalog and by providing the absolute URL to this file when setting up a new Content Library.
Step 4 - To publish the 3rd Party Content Library, you just need to host the content somewhere that supports HTTP(s), this can be as simple as using a regular Web Server to something like an Object Store such as Amazon S3.
Note: If you plan to use a non-traditional Web Server for hosting your 3rd Party Content, ensure that the root URL is the same for accessing all files and can not change. For example, you will not be able to host a 3rd Party Content Library using Dropbox as the root URL changes for each and every file.
Step 5 (Optional)- Periodically you will most likely want to add new content and files to your 3rd Party Content Library and as part of that process you will also need to update the metadata index. You can easily do so by re-running the make_vcsp_2018.py script against your existing 3rd Party Content Library. The script will only generate new metadata index for files that have been newly added and for any changes to existing files, the appropriate metadata files will only updated.
As you can see the process in creating and publishing a 3rd Party Content Library is pretty straight forward and simple. I hope to see more of our partners start to publish their own 3rd Party Content Libraries, I think it would be extremely beneficial for VMware customers and I look forward to hearing about them :). If you have created your own 3rd Party Content Library and would like to share the details, feel free to leave a comment and I will be aggregating a list below so that you can easily find them all in one place.
Published 3rd Party Content Libraries
| Author | Library Content | Link |
|---|---|---|
| William Lam | Nested ESXi & VSAN OVF Templates | Subscribe to vGhetto Nested ESXi Template Content Library in vSphere 6.0 |
| Alan Renouf | Some sample OVF/OVAs | Subscribe to Alan's Content Library in vSphere 6.0 |
Hi William,
Any idea if there is a way for multiple users on the same vCenter to share a content library so i don't have to store all the templates and iso files multiple times as each user creates their own Content Library? Even better would be the ability to share a Content Library with all vCetners in a site and only storing the files once.
Yes, you can setup permissions so that a group of users can all share/contribute to a common Content Library. In terms of your second question, the boundary for Content Library is at the VC level, this means that you would need to setup the other VC's as subscribers to your main VC if you want to share the content and it would be replicated.
thanks for that, any idea where the permissions are I cannot find where they are set?
They're under Roles in the vSphere Web/C# Client just like all the other permissions. There's a new "Content Library Administrator" sample role which you can see all the different privileges that can be assigned. You can then add/remove privileges either using an existing role or creating a new one.
The problem I have is that I create a library as *protected email* then login with my domain user account that has full admin privileges on the whole vCenter and when I go to Content Libraries it says there are none. I then log back in as SSO admin and the library I created is there and works fine.
This should work as long as your AD Domain account has full Administrator privileges. Can you check to see if the AD Domain User/Group is also part of the SSO Administrator group? You can check by going to Administration->Users and Groups and under "Group" there should be one called Administrator. If not, can you try adding the AD User/Group and see if that helps?
Yes if I apply the permission at the Global level to all users for the role defined for Content Library access then it works.
Sorry to make that clear: in the web client I go to: Administration -> Global Permissions -> Add then select the AD group for my users then select the Role customized for Content Library access or the sample one and apply it all those users then get those permissions.
I couldn't find any other way to get permissions to apply to the Content Library. This is ok if you don't need to apply global permissions to the same group for any other reason unless you create a custom role specifically for use in the Global permissions.
I'm not entire clear on the "publish" step. Could you flesh out some details on how exactly you publish the content library from a web host? Specifically if I was using Amazon S3 ?
Hi William, Thanks for the article and script. Can we use the script for creating catalog for VCD 8.20 too?
Note* This script does not work with Python 3+. Stick with 2.7x if you don't want encoding errors 🙂
Hi,
Trying to run my own content library with your script but didn't find how to deploy vm from it ?
Got my ovf correctly published but cannot use it for "new vm from Content Library" operation.
nevermind ... just found out that the manual synchronization of the item is required.
In fact got inconsistent result where vm template type stay Unknown and cannot be used.
Any idea why ?
Are you sync'ing OVF or OVA? Only the former is a "deployable" type today in CL. This is the same workflow if you were to use the CL UI to upload an OVF, if you upload OVA it'll be treated as a regular file
I'm using ovf but when i sync an item im can't use it.
i have to delete it and resync
Hey, So I have more of a flat structure of my files and script does generate item.json and there I have everything like this:
{
"files": [
{
"hrefs": [
"vm.vmdk"
],
"etag": "907d7fd341c17a78205844683de051e9",
"name": "item.vmdk",
"size": xxx
},
"hrefs": [
"vm.vmdk"
],
"etag": "907d7fd341c17a78205844683de051e9",
"name": "item.ovf",
"size": xxx
},
...
When I create a lib out of this I get some general system errors, with NULL pointers reported by vCenter that I am using.
My question is ... Is this suppose to work, or I need to have separated folders in the root folder for each vmdk, ovf ... file?
Thx!
I managed to fix this, but now once I import all items are marked as UNKNOWN, anyone had something similar?
How did you fix null pointer error?
I setup third party content library on a windows server and used IIS for web server but getting null pointer error while trying to sync it.
Hello William!
How to host a VMTX template in a third party content library? What files from the template will be needed there?
VMTX can’t be hosted as it’s vCenter specific feature and is associated with VC