I recently spent some time exploring VMware Cloud Foundation (VCF) using VMware Cloud Builder, which is a tool that automates the entire VCF deployment and configuration when provided a set of ESXi hosts that meet all the pre-requisites. To begin a VCF deployment, users will typically provide a VCF deployment parameter workbook (XLS) which contains the configuration for your deployment which is then uploaded to Cloud Builder (UI or API).
I personally prefer the JSON option which Cloud Builder also supports, which for me is much easier to read and edit. For those interested, after submitting your XLS document, you can retrieve the generated JSON configuration file in the Cloud Builder filesystem located in /tmp/sddcspec-[UUID].json.
While working on some automation, I needed to separate out the ESXi and NSX-T networking, which by default is configured using a single Distributed Virtual Switch (VDS). This is a common configuration pattern for environments that only have two network adapters. However, if you have up to four network adapters, VCF can also support another deployment configuration where two VDS are used. I came to learn about this while talking to one of the Cloud Builder Tech Leads and shared the required JSON configuration changes to support this model.
Below is the JSON snippet for the required dvsSpecs section that demonstrates the use of a dual VDS configuration where NSX-T networking makes use of the additional vmnics.
"dvsSpecs": [{ "dvsName": "vcf-m01-cl01-vds01", "vcenterId": "vcenter-1", "vmnics": [ "vmnic0", "vmnic1" ], "mtu": 9000, "networks": [ "MANAGEMENT", "VMOTION", "VSAN" ], "niocSpecs": [ { "trafficType": "VSAN", "value": "HIGH" }, { "trafficType": "VMOTION", "value": "LOW" }, { "trafficType": "VDP", "value": "LOW" }, { "trafficType": "VIRTUALMACHINE", "value": "HIGH" }, { "trafficType": "MANAGEMENT", "value": "NORMAL" }, { "trafficType": "NFS", "value": "LOW" }, { "trafficType": "HBR", "value": "LOW" }, { "trafficType": "FAULTTOLERANCE", "value": "LOW" }, { "trafficType": "ISCSI", "value": "LOW" } ], "isUsedByNsxt": false }, { "dvsName": "vcf-m01-nsx-vds01", "vcenterId": "vcenter-1", "vmnics": [ "vmnic2", "vmnic3" ], "mtu": 9000, "networks": [], "isUsedByNsxt": true } ]
With the above configuration, here is what the ESXi networking configuration will look like after VCF is deployed and configured. At the time of writing this, I was using the latest VCF 4.2 release and the respective Cloud Builder version that is supported.
While sharing the news with a fellow colleague, I also learned that in the latest VCF 4.1/4.2 release, excel workbook also now contains the ability to configure a dual VDS network profile as shown in the screenshot below. This definitely was not there in 4.0 release and most likely was introduced in either 4.1 or 4.2 release.
nosbigys says
William - great post! I recently went through the process of building automation to deploy VCF Cloud Builder for Digital Rebar Platform (DRP). We have customers with 10s of Thousands of ESXi hosts that also consume VCF.
I take in the JSON format for bootstrapping the cluster via Cloud Builder.
However ... as you briefly mentioned ...
"a set of ESXi hosts that meet all the pre-requisites"
That is one of the biggest challenges. Ensuring all ESXi hosts are pre-configured and in the same configuration for VCF is critical. Especially if you need to meet VCF spec / HCL compliance. We do all of the BIOS/Firmware/RAID, and ESXi configuration to meet HCL spec compliance ...
THEN ... we can get to the VCF Cloud Builder bootstrap and be relatively confident it'll be successful.
Again - a GREAT article. Thanks!
William Lam says
Indeed and from a learning/exploration standpoint, it can also be easily meet all these pre-reqs. Stay tuned, I've got some more things to share in this space 🙂
jstreit2017 says
William,
Your article is very timely. It sounds like NSX-t will use all the interfaces of the VDS. In your situation where a host has 4 physical links, and you don't want two different VDS's, is it possible to control which interfaces NSX is using through the JSON file? (ex. interfaces vmnic2 & 3) Thanks.
William Lam says
not possible afaik. You have to map vmnic to specific VDS