I know many of you have been asking me about my vSphere with Kubernetes automation script which I had been sharing snippets of on Twitter. For the past couple of weeks, I have been hard at work making the required changes between the vSphere 7 Beta and GA workflows, some additional testing and of course documentation. Hopefully the wait was worth it (I think it is) and if you enjoy the script or have benefited, please consider adding 🌟to the Github repo to show your support! Thanks and enjoy
Had to make some updates to one of my vGhetto Automated Lab Deployment Scripts
💥44min to automate all required #vSphere7 infrastructure! 🤛🎤🥳
1 x VCSA 7.0
3 x ESXi + vSAN 7.0
1 x NSX-T 3.0 UA
1 x NSX-T EdgeNeed to clean up #ProjectPacific wording but its working great! pic.twitter.com/ZInPgVgbGS
— William Lam (@lamw) April 4, 2020
The Github repository:
Before getting started, please carefully read through the requirements section along with the complete sample end-to-end execution if you are new to vSphere with Kubernetes. You will need to have a VMware Cloud Foundation (VCF) 4.0 license before you can get started and specifically an NSX-T Advance license which is one of the required parameters within the script. If you do not have access to a VCF 4 license, I strongly recommend taking part in the recent VMUG Advantage Homelab Group Buy effort which I had started to easily get access to the latest VMware releases along with a nice 15% discount!
The script supports deploying both a standard vSphere 7 environment with just VCSA, ESXi and vSAN as well as the complete solution which includes NSX-T to support vSphere with Kubernetes. For more details, please refer to the FAQ.
Perfect job Wiliam!
Thanks William!
I'm getting this error during deployment, any ideas?
https://imgur.com/a/2UbuIZk
@Ray I received a similar message regarding the storage policy part, today.
For lines 1035 & 1036 add " -Server $vc " to the New-TagAssignment command and to the New-SpbmStoragePolicy.
This should resolve the issue as the policy is being applied to the main VCSA instance instead of the nested one. I am in the process of redeploying and will confirm.
Hope that helps.
Regarding the NSX-T Manager error make sure you can resolve the FQDN for the NSX-T manager from where your executing the script from.
https://imgur.com/vecSEe3
The deployment was successful adding the -Server $vc mentioned above.
https://imgur.com/vtsryUx
Thank you VirtualizeStuff. I've just pushed the VM Storage Policy scope fix and glad to hear everything deployed successfully for you
Thank you VirtualizeStuff!
I realized the issue with NSX-T Manager and was able to get that one resolved. I tried deploying again with your fix and it appears to work, but I'm still getting an unauthenticated error for the Get-Tag command:
https://imgur.com/a/IPSIK43
Not sure if this is really a problem. The rest of the script seems to complete successfully.
I am getting below error. Everything else is working perfectly.
1036 | New-SpbmStoragePolicy -Name $StoragePolicyName -AnyOfRuleSets …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 4/17/2020 4:12:39 PM New-SpbmStoragePolicy Index was outside the bounds of the array.
Thanks Heng. I believe this was also reported on the Github repo and I think I know why. Not had time to look at it yet, but that should go away in a future update. Glad to hear everything else is working for you
Thanks William for your quick reply! I am new to this. Do I need to manually apply any change for this failure?
No, it's just a warning. You should see a new Pacific Gold VM Storage Policy in the deployed vCenter Server Appliance. If you see that, then you're good
Thanks for creating this! I'm following your scripts and readme on github, but I am getting tripped up on the Ingress/Egress both nested, and in my real lab that was built manually. I can successfully ping the IP that gets assigned to the control plane but whenever I try to launch the webpage, it times out either from VCSA or just typing in the URL. From reading the main docs on VMware site, your edge vlan and Ingress/Egress must be on the same VLAN, which I see is there, and can't overlap IP's, which they aren't. I'm not getting why I can ping it but can't connect, this is after multiple deployments.
Thanks for the script! Do you have plans or suggestions for a (non-automated) guide for deployment in a non-nested environment? I'd love to get this working on my NUC vSAN cluster 🙂
Great work William,
deployment works like a charm.
Unfortunately I run into a weird situation when configuring the workload management as its stuck after deploying the supervisorcontrolplanes. The first one starts, but then it doens't continue (I waited about 3 hours) showing the following status:
Configure operation for the Master node VM with identifier vm-1012 failed.
The log shows that it loops: master is not fully configured
Anybody running into the same issue?
Yep, I'm seeing the same problem as well.
Yes, I am experiencing the exact same issue. 1 of the 3 initially deployed Master nodes starts being configured and it freezes there.
I have exact same issue there. The problem is that the 1st SupervisorControlPlaneVM is not reachable via network. I'm struggling with network topology there. My management network at 300th VLAN. Should I specify VLAN in my nested network portgroups as well?
https://imgur.com/a/v7f3aYz
Ok, my issue was with configuration of "underlay" ESX network with promiscuous and forget transmit off. After switching it on the Control VM become available via network. But the error remained: vmware vsphere Configure operation for the Master node VM with identifier "identifier" failed.
found and fixed the issues - unfortunately, I don't know what exactly caused the issues, but I changed the following
1) allow promiscuous and forget transmit on the dvswitch the supervisor-VMs (thanks for the hint skalugin2014!)
2) changed the NSX T0 connected vSwitch to MTU 9000 (was 1600 before)
Then the workload deployment completed successfully (it can take an hour and you need to ignore the errors)
But I couldn't add the tanzu content library and image pull for new pods failed as well - the fetcher wasn't able to download the images
3) changed the routeros (not sure what the issue was) to a pfsense as my default gateway
4) enabled mtu 9000 for the port towards the NSX T0
5) enabled sNAT for the Kubernetes ip ranges
Now it runs fine and so far all seems to work.
I shared my findings here after deploying the nested setup using William's script: https://itnext.io/nested-vsphere-7-and-kubernetes-lab-deployment-explained-f9bfca0112f5
Dennis, thanks for the write up! Can you configure the Pfsense with MTU 9000 and not have to touch the TOR switches? Trying to get this to work without having to change the TOR switches.
TIA, Mike
Mike, I always changed the physical switches as well. I don't think that it will work without changing the switch ports pointing towards your LAN.
I have the same error: stuck at Master VM deployment.
Checked MTUs on both "host" esxi and nested, forged and promiscues, nothing helped.
Strange is that i had enabled Workload once on this cluster, then I removed everything and now i have this error. Nothing changed (as fas as I know).
Using distributed switch (not NSX-T)
I hit the same. only one SupervisorControlPlaneVM is up, other 2 even complaint "There is no network assigned to this virtual machine". MTU, forged and promiscuous helped nothing.
I think that I solved this with "DNS suffix" input during enablement of Workload. It's optional input but still.
Now I have similar problems/error at deploying Guest clusters.....
Awesome work, looking forward to getting this going!
Anybody had luck getting this running on AMD non-Epyc? The NSX-T Edge won't come up. It reports lack of AES-NI but seems like it may actually be lack of DPDK support per Mike and Ben's discussion here:
https://vswitchzero.com/2019/02/21/nsx-t-pcpu-requirements-for-edges/
Hoping I can find out where config.py lives in the Edge OVA so that I can try the "hack" and get the script to complete. Not sure if not having DPDK support will cause further issues later though.
Any other suggestions?
Just built a shiny new Ryzen lab to try this out. Hopefully Ryzen isn't kicked out of the vSphere with Kubernetes homelab club! 🙂
Always carefully read through new product release notes, you'll never know when you find some goodies in there 🙂
In NSX-T 3.0, AMD is indeed supported but for specific models (snippet from https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.0/rn/VMware-NSX-T-Data-Center-30-Release-Notes.html)
AMD EPYC support: Edge Nodes, VM and Bare Metal can now be deployed on AMD EPYC series CPU:
AMD EPYC 7xx1 Series (Naples)
AMD EPYC 3000 Embedded Family and newer
AMD EPYC 7xx2 Series (Rome)
It's worth noting that I was getting failures on the Import-VApp commands e.g. when importing the NSX-T manager appliance - due to the vCenter Server certificate not being trusted, I assume, as the md5 sum checked out.
I found that adding "-Force" to all of the Import-VApp statements fixed it for me.
Thanks, I had the same issue and it really helped me!
Hi,
after spending a complete day on getting this deploy to work using "VMware Workstation 15" on Windows 2019 I mananged to get the scripted components deployed. BUT at enabling the workload management candys i get a incompatible cluster when i press the (i) i get message "Compatible clusters must have a minimum of two ESXi hosts configured with a validate license for Workload Management, Fully Automated DRS, vSphere HA, a vSphere Distributed Switch 7.0 along with enough capacity to store the requisite infrastructure componenters you'll be configuring."
But i don't have a glue what i have misconfigured. I used these settings here : https://www.williamlam.com/2020/04/deploying-a-minimal-vsphere-with-kubernetes-environment.html
Is there any log file were i can see what did not passed the test ??
Thanks,
Michel
i found some issues in the log file. looks like this could be the issue ???
[05-14-2020_07:26:18] Powering On pacific-nsx-edge-3a ...
[05-14-2020_07:26:18] Creating vApp vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI ...
New-VApp: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:766
Line |
766 | $VApp = New-VApp -Name $VAppName -Server $viConnection -Location …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 5/14/2020 7:26:20 PM New-VApp The operation is not supported on the object.
[05-14-2020_07:26:20] Creating VM Folder Project-Pacific ...
[05-14-2020_07:26:20] Moving Nested ESXi VMs into vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI vApp ...
Move-VM: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:777
Line |
777 | … Move-VM -VM $vm -Server $viConnection -Destination $VApp -Confir …
| ~~~~~
| Cannot validate argument on parameter 'Destination'. The argument is null or empty. Provide an
| argument that is not null or empty, and then try the command again.
Move-VM: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:777
Line |
777 | … Move-VM -VM $vm -Server $viConnection -Destination $VApp -Confir …
| ~~~~~
| Cannot validate argument on parameter 'Destination'. The argument is null or empty. Provide an
| argument that is not null or empty, and then try the command again.
[05-14-2020_07:26:20] Moving pacific-vcsa-3 into vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI vApp ...
Move-VM: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:784
Line |
784 | … Move-VM -VM $vcsaVM -Server $viConnection -Destination $VApp -Confir …
| ~~~~~
| Cannot validate argument on parameter 'Destination'. The argument is null or empty. Provide an
| argument that is not null or empty, and then try the command again.
[05-14-2020_07:26:21] Moving pacific-nsx-3 into vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI vApp ...
Move-VM: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:790
Line |
790 | … ove-VM -VM $nsxMgrVM -Server $viConnection -Destination $VApp -Confir …
| ~~~~~
| Cannot validate argument on parameter 'Destination'. The argument is null or empty. Provide an
| argument that is not null or empty, and then try the command again.
[05-14-2020_07:26:21] Moving NSX Edge VMs into vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI vApp ...
Move-VM: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:797
Line |
797 | … ve-VM -VM $nsxEdgeVM -Server $viConnection -Destination $VApp -Confir …
| ~~~~~
| Cannot validate argument on parameter 'Destination'. The argument is null or empty. Provide an
| argument that is not null or empty, and then try the command again.
[05-14-2020_07:26:21] Moving vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI to VM Folder Project-Pacific ...
Move-VApp: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:802
Line |
802 | Move-VApp -Server $viConnection $VAppName -Destination (Get-Folde …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 5/14/2020 7:26:21 PM Move-VApp Could not find VApp with name
| 'vGhetto-Nested-Project-Pacific-NSX-T-External-Lab-pSZrDVlI'.
Move-VApp: D:\powershell\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:802
Line |
802 | Move-VApp -Server $viConnection $VAppName -Destination (Get-Folde …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 5/14/2020 7:26:21 PM Move-VApp Value cannot be found for the mandatory parameter VApp
[05-14-2020_07:26:21] Disconnecting from 192.168.236.140 ...
Do you have DRS enabled on your physical vSphere Cluster? This is required to create the vApp construct ... but even then, it shouldn't impact your enablement of Workload Management. As suggested in the troubleshooting blog post, take a look at the various compat checks and tail the wcp.log to see whats happening, this is probably environmental
hi william,
thanks for replying
I fixed the Vapp and moved all the hosts into this.
but still is get a incompatible cluster
i got this message below from the wcp log.
Any idea how to fix?
2020-05-15T12:47:52.0052 debug ucp [opID=uapi] Sending response with output {"output”:{"STRUCTURE”:{ "com.umuare.cis.data.prouider.resource_mode1.resu1t_set”:{"items":[{"STRUCTURE":{"com.umuare.cis.dat a.prouider.resource_mode1.resource_itern":{"property_ualues":[{"OPTIONAL":"domain-cl006:90e22647-9a24 -4e34-949e-04d3fdf50cl0">,{"OPTIONAL":"ClusterComputeResource">,{"OPTIONAL":[{"STRUCTURE":{"com.umwa re.uapi.std.localizable_message":{"args":["90e22647-9a24-4e34-949e-04d3fdf50cl0"I,"default.message": "Failed to list all distributed switches in uCenter 90e22647-9a24-4e34-949e-04d3fdf50cl0.","id":"uce nter. wcp. nsx. 1 ist. dus. error" /'localized": {"OPTIONAL" : nu 11 >, "params" : {"OPT IONAL" : nu 11 »», {"STRUCTURE ":{"com.umware.uapi.std.loca1izable.message":{"args":["domain-cl006"],"defau 1t.message":"Cluster dom ain-cl006 is missing compatible NSX-T UDS.","id":"ucenter.ucp.ncp.cluster.incompatible","localized": {"OPTIONAL":null>,"params":{"OPTIONAL":nul1>>>>]>]>>>],"properties":["OmodelKey’V’Otype", "errors"]," tota l.count" : {"OPT IONAL" : nu 11 »»>
2020-05-15T12:47:52.2262 debug ucp [opID=5ebe77e81 No notifications. seqNum: 1, Current seqNum: 0
As you can see from the error, it stating "Cluster domain-cl006 is missing compatible NSX-T UDS"
Not sure if this is a localization issue but UDS == VDS. Since you're using the Automation, which defaults creates a vSphere 7 VDS (not NSX-T N-VDS), its highly possible VC can't talk to NSX-T due to timeskew which is something I've seen customers run into. You can verify this by looking at the NSX-T Manager API logs which I reference https://www.williamlam.com/2020/05/troubleshooting-tips-for-configuring-vsphere-with-kubernetes.html You should see some token issues if it is what I suspect
Thanks William, During the copy and paste some chars went corrupted , was forgotten to correct it. So yes UDV is VDS. Thanks for the blog link i will check it.
I also did a complete new deploy but at the VCSA deployment i got a IO network copy error , i changed the script to use import-vapp but thats giving me a trust error. Didn't try out the repack tool from vmware to fix this trust error. So i went to deploy it manually. Currently running.
Let's hope it is going to deploy correctly ....
michel
Great job, William! Is it possible to use this same script to deploy into a standalone ESXi host instead of deploying into a vCenter?
No
i have a setup with VMware Workstation on a 16 core , 128 GB mem 1 TB storage server . It is almost working ... Don't think the almost is vmware workstation related . I am sure William could get this to work 😉
What to say thanks for the script but when I start to deploy I get this error "Unable to locate Pacific-VTEP portgroup, please create this network before continuing ..." Thanks for any help you can give
Indeed it would be better to also add this port group creation in a pre instal setup script. Did not wrote the script but maybe my experiences help: You have to create this port group on the VC you are using to run the script against . Also be sure to change all IP numbers in the script to a range you can access from the server where you run the PS on. (Only the once’s listed in the git page) The script creates a couple of VMs on the VC you connect it to. It also creates a new VC . This new VC you can use at the end to enable the kubernetes on. Also be sure to have ALL VM dns names resolvable !! This rollout depends really on DNS. I used CoreDNS for this.
Especially the creation of the edge pre setup was a challenge due to failing REST calls to the NSX , at the end I created it manually.
Nevertheless never got it completely to finish , at enabling the kubernetes cluster I get a vm master setup error.(could be due to my manual adds pointed above )
I will continue to get it working and will also PUSH some changes to the script to make it Re-runable in case of a stopped deploy .
Oh one more point , I run it on VM workstation as platform .
Michel
Thanks for the info
Ok I got it to start deploying and everything seem to be going good but then i get this error
OperationStopped: E:\Utility\ALPHAIT_Home_Lab\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1034
Line |
1034 | $ipPool = $ipPoolService.create($ipPoolSpec)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| A server error occurred: 'com.vmware.vapi.std.errors.invalid_request': . Check
| $Error[0].Exception.ServerError for more details.
[05-21-2020_01:42:39] Creating Overlay & VLAN Transport Zones ...
[05-21-2020_01:42:40] Creating ESXi Uplink Profile ...
[05-21-2020_01:42:40] Creating Edge Uplink Profile ...
[05-21-2020_01:42:43] Creating Transport Node Profile Pacific-Host-Transport-Node-Profile ...
Write-Error: Error in creating NSX-T Transport Node Profile
Write-Error:
({
"details" : "Field level validation errors: {required property host_switch_spec.host_switches[0].ip_assignment_spec.ip_pool_id is missing}",
"httpStatus" : "BAD_REQUEST",
"error_code" : 255,
"module_name" : "common-services",
"error_message" : "Field level validation errors: {required property host_switch_spec.host_switches[0].ip_assignment_spec.ip_pool_id is missing}"
}.Exception.Message)
I had exact same issue . So I created it manually in the NSX portal.
Trying this deployment - got a strange error (below).
It looks as if it is failing when trying to create the disk group on each host.
I'm only doing a vsphere and vsan deployment here - followed instructions in FAQ (set other options to zero)
And everything looks to be there ... I just have to log in after and through the UI cretae disk groups ...
The code looks fine though ... I checked all of the variables (jn-line, after the script completes) ..
.....................................................................
Querying ESXi host disks to create VSAN Diskgroups ...
[05-22-2020_04:06:19] Creating VSAN DiskGroup for tdclab-esx5.tdclab.deployed.af.mil ...
DriveLetter FriendlyName FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining Size
----------- ------------ -------------- --------- ------------ ----------------- ------------- ----
D VMware VCSA 0 5 0 {2}
New-VsanDiskGroup: C:\Users-Source\Scripts\dev\vtdclab7.ps1:879:13
Line |
879 | New-VsanDiskGroup -Server $vc -VMHost $vmhost -SsdCanonic …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 05/22/20 4:06:20 PM New-VsanDiskGroup VSAN runtime fault on server
| '/VIServer=vsphere.local\[email protected]:443/': Unknown server error: '
| Required property deviceName is missing from data object of type HostScsiDisk
while
| parsing serialized DataObject of type vim.host.ScsiDisk
at line 1, column 360
while
| parsing property "cacheDisks" of static type ArrayOfHostScsiDisk
while parsing
| serialized DataObject of type vim.vsan.host.DiskMappingCreationSpec
at line 1, column
| 298
while parsing call information for method InitializeDiskMappings
at line 1, column
| 171
while parsing SOAP body
at line 1, column 64
while parsing SOAP envelope
at line
| 1, column 0
while parsing HTTP request for method initializeDiskMappings
on object of
| type vim.cluster.VsanVcDiskManagementSystem
at line 1, column 0'. See the event log for
| details..
Any specific configuration for "Pacific-VTEP" port group?
Nope, its just a dummy portgroup and you can use the defaults when creating it.
Thanks William, I had another question regarding VCF 4 license, based on my understanding it is a suite of products and each product has its own license isn't it? and its managed via SDDC manager.
Any reference document where VCF4 licensing is explained?
Hello
I don't understand why, but when I try to enable "workload management", I configure all options, process is beginning, but during configuration, I don't know why but I have nested ESXi host which reboot without, but no warning or error during process
Anyway I have sufficient ressource 🙁
Do someone have the same issue ?
Thank you
ps : sorry for my bad english
After some another test, I know exactly when nested esxi host shutdown
The deploy of "SupervisorControlPlaneVM" is OK, and just after, it try to power ON.
And just at this time, ESXi host shutdown immediatly. VM's doesn't have time to boot.
So "SupervisorControlPlaneVM" VM go to another ESXi
Try to power ON
and ... shutdown ESXi host again 🙁
Someone do you know why ?
My hardware lab is 3 chinese barebone with 10170U proc and 64Gb each with vSAN
so... I think problem is on nested ESXi, because regardless of any VMs I make, when I start the VMs, ESXi stop immediatly 🙁
I use ova "Nested_ESXi7.0_Appliance_Template_v1.ova" anyway ...
snifff
David,
The symptoms you're describing is definitely not related to vSphere w/K8s but sounds like its due to https://www.williamlam.com/2020/04/heads-up-nested-esxi-crashes-in-esxi-7-0-running-on-older-cpus.html
The fix should be out later today with the release of ESXi 7.0 Patch 01 which will resolve Nested ESXi crash
by the way, thank you for your answer 🙂
Il will avoid me to search for long time unnecessarily.
yes, you're right...
so, I will waiting up to the patch 01 for ESXi 7.0... snifff
Hi, i'm running into an issue when running the script. I've updated each editable variable a few times to verify everything was correct but cant get past this error.
553 | … n $ovfconfig -Name $VMName -Location $cluster -VMHost $vmhost -Datast …
| ~~~~~~~
| Cannot validate argument on parameter 'VMHost'. The argument is null or empty. Provide an argument that is not
| null or empty, and then try the command again.
Added my physical host to the cluster and it resolved the issue.
Hello
Deployment is OK
Workload management is OK
I create namespace
but ...
I have a issue to access at my namespace.
when I click to 'open'
http://lufia.konyxia.com/NSXT/0.click_OK.png
first access is ok
http://lufia.konyxia.com/NSXT/first-access.png
but if I try to refresh or download CLI tools, I have a "timed out"
http://lufia.konyxia.com/NSXT/second-access(with_F5).png
I always ping anyway
I have to close and re-open, but same issue.
promiscuous and forged transmits are ok, I put 1600 mtu everywhere,
but always same issue, I think for a mtu error.
Dennis Zimmer has write
2) changed the NSX T0 connected vSwitch to MTU 9000 (was 1600 before)
4) enabled mtu 9000 for the port towards the NSX T0
can you tell me what vswitch please ?
thanks
Hello
I tried lot of solutions, but none was successful.
best I can do after change lots of parameters (mtu, promiscuous, change vlan for a full level 2 transports) is to be able to have each time the home page, but after a big time, but it's all, I can't have more 🙁
Nobody would have an idea ?
Thanks
Hi
I have similar problem. Did you find a solution ?
Thanx
William,
I want to commend you for all your hardwork in putting this series of scripts together. I can only imagine the amount of effort you put in to get this to a working state. Please keep up the good job!
Regards,
Thanks for the note Michael. Yes, this has definitely been a ton of work which most folks don't see other than the final results. Its also challenging to keep things up to date with so many other scripts I've written, so finding spare time is always hard as well.
Hello everybody,
I think I know why I can't access to workload page
http://lufia.konyxia.com/NSXT/0.click_OK.png
All is done up to the namespace, but unable to continue.
I use usb network with flings drivers, and I noticed that mtu is limited to 4000
https://flings.vmware.com/usb-network-native-driver-for-esxi/bugs/816
I use vsphere 7 with only usb network because realtek is not more supported with vsphere 7 (only vusbX, not vmnicX).
I would like to know if somebody has successfully used this lab with only fings usb network ?
or if someone have an idea for have it run? (specific conf nsx or nested esxi ?)
Thanks.
Hello,
I have resolved my issue, it was a MTU error beetwen 2 switch, it's ok now
However I have a last issue, even if ostensibly all seem good.
the output of "kubectl get machine" give me :
error: the server doesn't have a resource type "machine"
show picture : http://lufia.konyxia.com/NSXT/error_get_machine.png
anyway, no error up to Step 6 - Verify the TKG Cluster is ready before use by running the following command:
Did I forget something ?
Thanks
Hi, do you hav any idea about no resources found?
I get error whenever i tried to create pod in namespacs it alwas pending state with no resources found when i do kubectl describe pod, and whenever i tried to create TKC and do "kubectl get tkr" "kubectl get machine....." the result always no resources found:(
First of all - thanks a lot!
Just wondering - how long should take creating of TKG cluster?
I mean:
./kubectl apply -f tkg-cluster.yaml
I did it 30 min ago, but I got nothing (status is still "creating" for "tkg get cluster"):
https://i.gyazo.com/a8dfdff25d3ebd65f32fcd93d37d8cb9.png
Found out that the problem is in:
NSX IP pool exhausted
FailedRealizeNSXResource
Aug 7, 2020, 5:26:45 PM
vm is not yet created: vmware-system-capw-controller-manager/WCPMachine/infrastructure.cluster.vmware.com/v1alpha2/im-tkg-nmsps-1/im-tkg-cluster-1/im-tkg-cluster-1-control-plane-bm97l
ReconcileFailure
Aug 7, 2020, 5:26:14 PM
Hmm, how could it be? I deployed only Supervisor cluster and started to deploy TKG cluster (no demo applications etc.).
William,
Thanks for the work, script, attention to detail, and follow up on these questions. I really appreciate it.
I have a quick licensing questions, though. I have VMUG, Paid, and I have VCF 4.0. Reading the setup part of the script, it only shows where to plug in an NSX-T license. I happen to have an NSX-T key, so put it in, and ran the script. The script completed with no errors. Now, though, when I try to add a workload, I get an error saying I'm not licensed for Kubernetes and that I need an Esxi Enterprise with Kubernetes add on license key. I don't have one, nor do I see where to get the license from VMUG.
Where did I go wrong?
For vSphere w/K8s to be enabled from vCenter Server standpoint, you need to the Enterprise w/K8s key which is NOT included from VMUG afaik, BUT you can simply use an eval deployment (which is what the script assumes) and hence you'll get 60 days to satisfy that requirement. The VCF 4.0 license includes all of these details but for that to be successful, you need to follow the VCF deployment which uses SDDC Manager, the underlying products are then license on its behalf. Hope that makes sense
It kinda makes sense, but I must be missing something somewhere...When I try to add a workload, it fails claiming I don't have the needed license, even within the 60 days. So what kind of trial is it? I'm sorry if I'm being block-headed.
Just make sure your VCSA/ESXi hosts are eval and you’ve got proper NSX-T License
So there really is no way to properly license so that it runs for more than 60 days using the VMUG membership at this time. Just about no-one has the hardware in a home lab to run VCF, and until they make the ESXi with Kubernetes licenses available, you'll only have 60 days to try this out. Unless I'm missing something. Don't really have the will to re-create the entire environment every 60 days, even if it's scripted.
I find VMware's crazy requirements exceptionally limiting. Rancher is really the way to go if you want to use Kubernetes and get K8S up and running without jumping through insane hoops.
Take a look at the TKG Demo Appliance if you just want basic K8s https://www.williamlam.com/2020/08/tanzu-kubernetes-grid-tkg-demo-appliance-1-1-3.html
I tried deploying the standard deployment for 6.7 but Im having an issues adding the ESXi hosts to VCSA 6.7.. The script seems to hang during the adding ESXi host to Vcenter Cluster phase until the session times out. No hosts are added . I tried manually adding the hosts but "Im having this error "Authenticity of the host's SSL certificate is not verified ". Anyone had this issue before ? Im using ESXi 6.7 and VCSA U3.
hello,
can we use a existing vcenter and deploy only esx vsan nsx-t.....
Just downloaded VCF from VMUG Advantage, and it is a 20 GB OVA file (VMware-Cloud-Builder-4.0.0.0-16008466_OVF10.ova), and not four ISOs, as it seems needed to be able to follow the script. Any ideas or comments?
Thanks in advance.
Keep running into this error
Line |
685 | if(!(Connect-NsxtServer -Server $NSXTMgrHostname -Username $NSXAd …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 10/12/2020 12:35:10 PM Connect-NsxtServer The SSL connection could not be established, see inner exception..
Unable to connect to NSX-T Manager, please check the deployment
The fix for the error above was to set Powershell to ignore the SSL certificate.
Be sure to "Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore" before you run the script. perhaps add it to the script and once finished set back to default.
Hi William, thanks again for your great work! Could you please tell me if it is a requirement that the VCA connects to the Internet in order to deploy the supervisor cluster? It seems that first it tries to download some stuff, then is failing and then it deploys some control VMs OVAs but not able to configure them. I have enabled a proxy in the VCSA admin, but still not working. Any ideas will be appreciated. Thank you!
Hi William, thank you for all scripts provided, great work.
Question about DNS: all hostnames used need to be in our DNS server, everything resolvable as FQDN or are you using IPs for setting up appliances etc?
Either can be supported but for ease of use, recommend setting up DNS
Folks - any ideas on these errors.
[01-12-2021_11:12:51] Creating External T0 Gateway Interface ...
OperationStopped: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1396
Line |
1396 | $t0GatewayInterface = $t0GatewayInterfacePolicyService.update …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| A server error occurred: 'com.vmware.vapi.std.errors.invalid_request': . Check
| $Error[0].Exception.ServerError for more details.
[01-12-2021_11:12:53] Adding Static Route on T0 Gateway Interface from 0.0.0.0/0 to 192.168.2.1 ...
OperationStopped: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1417
Line |
1417 | $staticRoute = $staticRoutePolicyService.patch($T0GatewayName …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| A server error occurred: 'com.vmware.vapi.std.errors.invalid_request': . Check
| $Error[0].Exception.ServerError for more details.
PS C:\Users\win10\Desktop\vm> $Error[0].Exception.ServerError.data
httpStatus error_code module_name error_message
---------- ---------- ----------- -------------
BAD_REQUEST 500012 Policy The path=[] is invalid
I also had this issue. I tried to mannually do these steps in the UI, but don't know if I did them right because I still have problems configuring the workload management feature. What NSX T version are you using? Mine is 3.1 limited export (maybe the limited export part has something to do with it?)
Interesting...im on same - nsx-unified-appliance-3.1.0.0.0.17107212-le
I also have nsx 3.0.1 i could try as well.
I just tried with nsx-unified-appliance-3.0.1.0.0.16404476.ova and i get a bunch of errors now:
[01-15-2021_07:07:02] Creating External T0 Gateway Interface ...
InvalidOperation: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1390
Line |
1390 | $t0GatewayInterfaceSpec.segment_path = "/infra/segments/$Netw …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The property 'segment_path' cannot be found on this object. Verify that the property exists and can be
| set.
InvalidOperation: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1391
Line |
1391 | $t0GatewayInterfaceAddResult = $t0GatewayInterfaceSpec.subnet …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
InvalidOperation: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1392 Line | 1392 | $t0GatewayInterfaceSpec.type = "EXTERNAL" | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | The property 'type' cannot be found on this object. Verify that the property exists and can be set.
InvalidOperation: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1393
Line |
1393 | $t0GatewayInterfaceSpec.edge_path = $edgeClusterNodePath
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The property 'edge_path' cannot be found on this object. Verify that the property exists and can be
| set.
InvalidOperation: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1394
Line |
1394 | $t0GatewayInterfaceSpec.resource_type = "Tier0Interface"
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The property 'resource_type' cannot be found on this object. Verify that the property exists and can
| be set.
OperationStopped: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1395
Line |
1395 | $t0GatewayInterface = $t0GatewayInterfacePolicyService.update …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Unable to get field 'tier0_interface', no field of that name found.
[01-15-2021_07:07:05] Adding Static Route on T0 Gateway Interface from 0.0.0.0/0 to 192.168.2.1 ...
OperationStopped: C:\Users\win10\Desktop\vm\vghetto-vsphere-with-kubernetes-external-nsxt-lab-deployment.ps1:1416
Line |
1416 | $staticRoute = $staticRoutePolicyService.patch($T0GatewayName …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| A server error occurred: 'com.vmware.vapi.std.errors.invalid_request': . Check
| $Error[0].Exception.ServerError for more details.
Did you use that version of both nsx t manager AND nsx t edge? Maybe they don't like being on different versions...
Actually I think I just got it working using the 3.1.0 le version. It says the workload management cluster is running but can't test it right now, will come back to that later...
What I did was set MTU to 1600 on everything I could (all VDS's, T0interface...) like described in this writeup: https://itnext.io/nested-vsphere-7-and-kubernetes-lab-deployment-explained-f9bfca0112f5
And also did the accept promiscuous mode and forged transmits on everything I could find in both the nested and native vsphere instance.
To manually configure the part of the t0 gateway that the script failed on (not the ones you just posted but the ones before that) I did this in the NSX T manager gui:
click on networking -> T0 gateway (there should already be one)
Edit the gateway
Go to interfaces
add
name: default
type: external
ipaddress/mask: what you set in the variables (192.168.0.14/24 in my case)
connected to segment: pacific-segment
edge node: the edge node
MTU: 1600
Then save and close the interface editing part
Go to routing -> static routes
name default network 0.0.0.0/0 next hop 192.168.0.1(your internet gateway)
Will update on if it actually functions tonight, but this got me further in the deployment than before 🙂
Thanks ThomasD
I deployed with full version of NSX 3.0.2 and I got same errors as the last ones i posted.
When you say you got it working, i take it you did the manual fixes from your last post. The script still displayed the same error messages, correct?
I will follow the manual steps you outlined and hopefully will get it going as well.
Yep, so I ignored the 2 errors (the one from creating T0 gateway & adding static route) and then continued with the steps I said above. I can now confirm it works, got workload management enabled by following the rest of the guide on the github repo and got the demo app running.
Found on vmware docs that these are the limitations of the limited export version of nsx t
no IPSec VPN
no HTTPS-based Load Balancer
But HTTP load balancer does seem to work, so that's enough for lab environments I guess
Great news with no errors ...retried with NSX-T - nsx-unified-appliance-3.0.2.0.0.16887203.ova & nsx-edge-3.0.2.0.0.16887208.ova
Issue as you pointed out is likely with LE version.
Anyone knows what this error means?
Everything else seems to run successfully
Creating Project Pacific Storage Policies and attaching to vsanDatastore ...
New-SpbmRule: C:\Users\win10\Desktop\vm\new-updated -vghetto-deployment.ps1:923
Line |
923 | … AllOfRules (New-SpbmRule -AnyOfTags (Get-Tag $StoragePolicyTagName))) …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | All the tags should be from same connection.
I must be doing something completely wrong. I changed the script for my network specifics and some of the names etc and ensured I met all of the requirements, but the scripts fails almost immediately with various errors like:
New-NetworkAdapter: C:\ESX\vghetto-crs-vsphere-lab-deployment.ps1:554
Line |
554 | New-NetworkAdapter -VM $vm -Type Vmxnet3 -NetworkName $NSXVTE …
| ~~~
| Cannot process argument transformation on parameter 'VM'. Object reference not set to an instance of
| an object.
New-NetworkAdapter: C:\ESX\vghetto-crs-vsphere-lab-deployment.ps1:555
Line |
and then finally fails and stops with:
[02-19-2021_10:19:24] Creating VCSA JSON Configuration file for deployment ...
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.
[02-19-2021_10:19:24] Deploying the VCSA ...
Template structure validation failed for template
C:\Users\crsav\AppData\Local\Temp\jsontemplate.json.
The key 'datacenter' in section 'new_vcsa' subsection 'vc' is required. Its
value cannot be null or empty.
The key 'datastore' in section 'new_vcsa' subsection 'vc' is required. Its value
cannot be null or empty.
Error message: Template structure validation failed for template
C:\Users\crsav\AppData\Local\Temp\jsontemplate.json.
[02-19-2021_10:19:26] Setting up NSX-T Edge to join NSX-T Management Plane ...
Connect-NsxtServer: C:\ESX\vghetto-crs-vsphere-lab-deployment.ps1:683
Line |
683 | if(!(Connect-NsxtServer -Server $NSXTMgrHostname -Username $NSXAd …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 2/19/2021 10:19:26 AM Connect-NsxtServer Unable to connect to the remote server
Unable to connect to NSX-T Manager, please check the deployment
This is my first time trying one of the scripts, I have the kubernetes portion marked out (per github FAQ) as I just want the environment with NSX-T. Any help? This is a C240 running 6.7
You can try the Brad's solution.
"The fix for the error above was to set Powershell to ignore the SSL certificate.
Be sure to "Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore" before you run the script. perhaps add it to the script and once finished set back to default."
So reading through some of the other comments,I downloaded a newer version of the NSX-T Manager and Edge to ensure the LE was not tripping it up. I also cannot seem to change the JSON depth error that keeps coming up although not sure that is actually stopping it. I continue to get the error:
[02-19-2021_02:55:49] Creating VCSA JSON Configuration file for deployment ...
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.
[02-19-2021_02:55:49] Deploying the VCSA ...
Template structure validation failed for template
C:\Users\crsav\AppData\Local\Temp\jsontemplate.json.
The key 'datacenter' in section 'new_vcsa' subsection 'vc' is required. Its
value cannot be null or empty.
The key 'datastore' in section 'new_vcsa' subsection 'vc' is required. Its value
cannot be null or empty.
Error message: Template structure validation failed for template
C:\Users\crsav\AppData\Local\Temp\jsontemplate.json.
[02-19-2021_02:55:52] Setting up NSX-T Edge to join NSX-T Management Plane ...
Connect-NsxtServer: C:\ESX\crstest.ps1:684
Line |
684 | if(!(Connect-NsxtServer -Server $NSXTMgrHostname -Username $NSXAd …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 2/19/2021 2:55:52 PM Connect-NsxtServer Unable to connect to the remote server
Unable to connect to NSX-T Manager, please check the deployment
PS C:\ESX>
And it kicks back out. I have confirmed the execution policy, updated PowerShell etc etc. I am POSITIVE I am doing something stupid, but for the life of me cannot see where I am going wrong here. Any help appreciated.
With 7.0u1 vCLS vm fails to start due to insufficient resources. Are you aware of any workaorunds for vsan datastore. I guess I can workaround using a different iscsi datastore.
I added a dedicated vmk interface(same vlan as management interface) enabled with vsan and the issue seems to have dissappeared.
Hello,
I just try the last version (Mar 13, 2021), I get a little error about "TKGContentLibraryName " variable, It seems to be not filled in.
I guess It's related with the library tanzu, surely "https://wp-content.vmware.com/v2/latest/lib.json"
not really a issue because we do it after, but I just notice it.
1007 | … New-ContentLibrary -Server $vc -Name $TKGContentLibraryName -Descri …
| ~~~~~~~~~~~~~~~~~~~~~~
| Cannot bind argument to parameter 'Name' because it is null.
I answer myself, just add the following variable :
$TKGContentLibraryName = "TanzuLib"
$TKGContentLibraryURL = "https://wp-content.vmware.com/v2/latest/lib.json"
I had several runs but whatever I do I keep getting an error when adding vCenter as a compute manager: A server error occurred: 'com.vmware.vapi.std.errors.invalid_request'. When I try it afterwards in the GUI I get a similar error: cannot register compute manager server vcsa.cpbu.corp, credentials are incorrect or invalid compute manager. (Error code: 7061).
I tried it with several versions of vCenter (7.0.0, 7.0.1 and 7.0.2) and two versions of NSX: nsx-unified-appliance-3.1.1.0.0.17483186.ova and nsx-unified-appliance-3.1.0.0.0.17107212-le.ova.
What am I missing? Could the issue be in the underlying ESXi host? Or vSwitch?
I think my issues were related to time... After changing the time server the vCenter can be added as Compute Manager.
Hi there!
I am using Workstation 16 for my Homelab (with 1 nested ESX, 1 vCenter, 1 NSX-T Manager and 1 NSX-T Edge). Everything works (NSX Overlay, Edge Uplink, etc) and is prepared for Tanzu (Content-Library, Storage policies, ...)
When I try to enable Workload Management, the Supervisor Control Plane VMs get deployed (I use 2 master VMs), they get IPs and are pingable. But then the setup runs forever (wcp log: Supervisor node bootstrap configure operation is still in progress...)
Has anybody experienced this issue?
BR Johannes
Hi
I had similar issue but with HAproxy and VDS. Anyway, after I input DNS suffix (it should be optional) while enabling Workload it finished OK.
When you refer to "DNS Suffix", do you mean the "DNS Search Domain" when enabling Workload Management?
Yes, I meant “DNS Search Domain”.
Did you ever find a fix for this? Running into same problem when enabling Workload Management. I do have a DNS entry for the 3 Supervisor Control Plane VMs and I did specify the search domain when setting up Workload Management. Thank you!
Yes, the solution is to use a DNS server on your PC for name resolution. The integrated DNS server in Workstation does not work correctly.
@Johannes - I know this is old, but I am running into the same issue and not really sure what you mean by this statement - I have the DNS search domain filled out, but it is still hanging on this. What exactly do you mean by this comment that you said fixes it?
My issue was related to VMware Workstation and DNS resolution. I solved this problem by using a dedicated DNS server on my PC for proper name resolution. VMware Workstation provides an internal DNS server, but which is not working correctly - this was the reason why the supervisor vms were not deploying correctly.
After enabling Workload Management successfully and creating a namespace (and assigning it storage), I try to log in via kubectl (through powershell) but getting this error. Also should be noted that I have the kubectl in my environment variables already.
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below.
Has anyone gotten this?
This tool is awesome. One thing I was wondering, can you provide a sample of how to edit the Default Layer3 Rule from ALLOW to DROP? Looking to deploy this as a ZTN.
william please help i am coming up with a lot of errors for example
Import-VApp: C:\Users\Administrator\Downloads\TANZU\william.ps1:632
Line |
632 | … $vm = Import-VApp -Source $NestedESXiApplianceOVA -OvfConfigura …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| 28/09/2021 12:10:57 Import-VApp Network 'SJC-CORP-MGMT' is not accessible from host with id
| 'HostSystem-host-14'
can you give me some pointers of what is going wrong please?
Hi William,
Thank you so much for all the good work,
i made issue #30 github just for that
additional code that is required to make use of the 2nd edge
at end of line 1450 after member_index
add [0]
member_index[0]
that's it
Cheers
Abbed strivevirtually.net