As promised, here is the complimentary solution to my existing Automated vSphere with Tanzu Lab Deployment Script, which will automatically deploy and configure the required infrastructure (vCenter Server Appliance, ESXi, vSAN and HAProxy VMs) so that you can quickly jump to enabling Workload Management on your vSphere Cluster.
FYI: Ben Corrie, one of the Engineers on the vSphere with Tanzu team recently published a vSphere with Tanzu 4-Part Deep Dive video series where he walks you through in deploying everything from scratch along with the concepts that should help you better understand how vSphere with Tanzu works. He is actually doing this in his own personal homelab and thought this might be useful to share with others. Kudos Ben and highly recommend folks check out his video if you new to vSphere with Tanzu and Kubernetes.
Enabling Workload Management is a manual step after the automated deployment script and as you know, I prefer to automate as much as I can. I have updated my existing PowerCLI Workload Management Module to now also support the new vSphere with Tanzu capability using HAProxy for networking instead of NSX-T. The module can be downloaded from PowerShell Gallery by simply running
Install-Module VMware.WorkloadManagement
Note: I should also mention that as of PowerCLI 12.1, there are now official PowerCLI cmdlets for configuring Workload Management which did not exists when vSphere 7.0 initially GA'ed and reason for this module. I have not personally taken a look at the cmdlets, but expect simliar set of user input will be required to enable.
Given the difference of input between an NSX-T and HAProxy based vSphere with Tanzu Cluster, I decided to create a new function called New-WorkloadManagement2 (very creative) and it accepts the following parameters:
- ClusterName - Name of vSphere Cluster to enable Workload Management
- TanzuvCenterServer - Hostname/IP of the new Tanzu vCenter Server that was deployed
- TanzuvCenterServerUsername - Username to connect to new Tanzu vCenter Server
- TanzuvCenterServerPassword - Password to connect to new Tanzu vCenter Server
- TanzuContentLibrary - Name of the Tanzu Kubernetes Grid subscribed Content Library
- ControlPlaneSize - Size of Control Plane VMs (TINY, SMALL, MEDIUM, LARGE)
- HAProxyVMName - The display name of the HAProxy VM
- HAProxyRootPassword - Root password for HAProxy VM
- HAProxyUsername - HAProxy Control Plane Username (default: wcp)
- HAProxyPassword - HAProxy Control Plane Password
-
HAProxyIPAddress - HAProxy Control Plane IP Address
- HAProxyPort - HAProxy Control Plane port (default: 5556)
- MgmtNetwork - Supervisor Management Network for Control Plane VMs (default: DVPG-Supervisor-Management-Network)
- MgmtNetworkStartIP - Starting IP Address for Control Plane VMs (5 consecutive free addresses)
- MgmtNetworkSubnet - Netmask for Management Network
- MgmtNetworkGateway - Gateway for Management Network
- MgmtNetworkDNS - DNS Server(s) to use for Management Network
- MgmtNetworkDNSDomain - DNS Domain(s)
- MgmtNetworkNTP - NTP Server(s)
- WorkloadNetworkLabel - Workload Network label defined in vSphere with Tanzu (default: network-1)
- WorkloadNetwork - Workload Network (default: DVPG-Workload-Network)
- WorkloadNetworkStartIP - Starting IP Address for Workload VMs
- WorkloadNetworkIPCount -Number of IP Addresses to allocate from starting from WorkloadNetworkStartIP
- WorkloadNetworkSubnet - Subnet for Workload Network
- WorkloadNetworkGateway - Gateway for Workload Network
- WorkloadNetworkDNS - DNS Server(s) to use for Workloads
- WorkloadNetworkServiceCIDR - K8S Service CIDR (default: 10.96.0.0/24)
- StoragePolicyName - Name of VM Storage Policy to use for Control Plane VMs, Ephemeral Disks & Image Cache
- HAProxyVMvCenterServer- Hostname/IP of the vCenter Server managing HAProxy VM to automatically retrieve CA certificate
- HAProxyVMvCenterUsername - Username to connect to vCenter Server managing HAProxy VM to automatically retrieve CA certificate
- HAProxyVMvCenterPassword - Password to connect to vCenter Server managing HAProxy VM to automatically retrieve CA certificate
- LoadBalancerLabel - Load Balancer label defined in vSphere with Tanzu (default: tanzu-haproy-1)
- LoadBalancerStartIP - Starting IP Address for HAProxy Load Balancer
- LoadBalancerIPCount - Number of IP Addresses to allocate from starting from LoadBalancerStartIP
- LoginBanner - Login message to show during kubectl login
At first glance, this may seem like a large amount of input and it is but that is what is needed when using the vSphere UI to enable Workload Management. I have streamlined some of the parameters and used defaults (assuming you used my deployment script). If not, you can always override the default values if you deployed vSphere with Tanzu using HAProxy the manual method as this ultimately just uses the vCenter Server REST API for Workload Management.
Here is an example of using the New-WorkloadManagement2 function:
$vSphereWithTanzuParams = @{ ClusterName = "Workload-Cluster"; TanzuvCenterServer = "tanzu-vcsa-1.cpbu.corp"; TanzuvCenterServerUsername = "*protected email*"; TanzuvCenterServerPassword = "VMware1!"; TanzuContentLibrary = "TKG-Content-Library"; ControlPlaneSize = "TINY"; MgmtNetworkStartIP = "172.17.31.120"; MgmtNetworkSubnet = "255.255.255.0"; MgmtNetworkGateway = "172.17.31.1"; MgmtNetworkDNS = @("172.17.31.5"); MgmtNetworkDNSDomain = "cpbu.corp"; MgmtNetworkNTP = @("5.199.135.170"); WorkloadNetworkStartIP = "172.17.36.130"; WorkloadNetworkIPCount = 20; WorkloadNetworkSubnet = "255.255.255.0"; WorkloadNetworkGateway = "172.17.36.1"; WorkloadNetworkDNS = @("172.17.31.5"); WorkloadNetworkServiceCIDR = "10.96.0.0/24"; StoragePolicyName = "tanzu-gold-storage-policy"; HAProxyVMvCenterServer = "mgmt-vcsa-01.cpbu.corp"; HAProxyVMvCenterUsername = "*protected email*"; HAProxyVMvCenterPassword = "VMware1!"; HAProxyVMName = "tanzu-haproxy-1"; HAProxyIPAddress = "172.17.31.116"; HAProxyRootPassword = "VMware1!"; HAProxyUsername = "wcp"; HAProxyPassword = "VMware1!"; LoadBalancerStartIP = "172.17.36.2"; LoadBalancerIPCount = 125 } New-WorkloadManagement2 @vSphereWithTanzuParams
The function will first connect to the vCenter Server which was used to deploy the HAProxy VM and then use the Guest Operations API (Invoke-VMScript) to automatically extract the HAProxy certificate which is one of the required parameters. It will then disconnect from that vCenter Server and connect to the new deployed vCenter Server from the automation script to provide the rest of the required paraemters. If the enablement request was accepted, you should see a message that asks you to use the vSphere UI to monitor the progress and when you navigate to the Workload Management section of the vSphere UI, you should see the enablement process begin. If all information was provided correctly, this can take up to 30-40min to complete.
HI William Lam,
Your article helpful for setup my lab. But i have some question.
From screenshot below link, After I tried deploy guestbook application on deployments menu at compute tab on namespce not show anything. Is this correct?
https://ibb.co/hMjtv7M
William, thank you for the very good and so often helpfull bolg! I tried to use the new cmdlet Enable-WMCluster to enable Workload Management, but since it is very new i can find some esamples. Would be VERY great if you could do an script also with the new PowerCLi commands. Thank you so much. Paul
Hi there William, after all congrats for the amazing work!!
I'd like ask you for help to be able to fix an issue while deploying my Tanzu home lab.
The whole infrastructure is well deployed ... nested esx, networking, and Workload Management
the problem is I can't access to the Control Plane IP Address (192.168.20.11) I've tried to re-deploy many times the whole infra and all the time getting success but I never get access to control plane IP
The 192.168.20.x network is accessible by from the any VM in the same range in Nested Infraestructure and the parent one ... all IP's 10.24.168.50-52 (Supervisor Cluster) is reachable as well.
But the HA Proxy Control Plane Address (192.168.20.11) isn't reachable.
Does anyone can help me with this ?? I'm totally stuck!!
Thanks in advance.
William, I tried your WorkloadManagement2 Skript, thank you very much. But I have this problem: I do not use subscribed content-library, but a local (following the procedure on the vmware site). I do this for performance reasons. If I use the GU everything goes fine. but your skript seem to accept only subscribed Content Libraries. Maybe you could fix this also to support local content libraries? would be great thank you so much!
The error I get is this:
...
Connecting to Tanzu vCenter Server to enable Workload Management ...
TKG Content Library has not fully sync'ed, please try again later
Great post!
How can I specify Workload and Magement network portgroups in this script?
Thank you.
Viky
Did you even read the blog post? Details are there 🙂
Hi...Team.
Lam congratuation for the post.!
I have a question.....could you help me.
implement tanzu, but it does not complete, when seeing events on the host I see error:
"An error occurred while customizing VM SupervisorControlPlaneVM (2) . For details reference the log file in the guest OS."
I also see warnings:
Guest operation authentication failed for operation Start Program on Virtual machine SupervisorControlPlaneVM (2) .
When viewing in kubernetes state:
Error: "Resource Type Deployment, Identifier vmware-system-netop/vmware-system-netop-controller-manager is not found."
Warning: "Node Kubernetes health status is Unknown"
Info: "Kubernetes cluster health endpoint problem at . Details: Waiting for API Master IP assignment"
"Customization operations of the guest OS for Master node VM with identifier vm-3166 is pending."
followed by that I see that it enters a loop, where it eliminates the vm "SupervisorControlPlaneVM" and recreates them, and thus it remains creating and eliminating